Docs On-premise Clustered Migrating to clustered
bugsnag.com Start Free Trial

Migration guide

Migrating from Single Machine to Clustered BugSnag On-premise

Prerequisites

To get started with your migration from Single Machine to Clustered BugSnag On-premise, you’ll need the following things prepared in advance:

  • Kubernetes cluster running at least version 1.19 and at most version 1.24
  • Docker registry
  • Single Machine BugSnag On-premise instance with the latest version installed
  • Ruby 2.7 or higher on the Single Machine Bugsnag On-premise instance

Migration Overview

The migration process has the following steps:

  1. Migrating the configuration - Converts the configuration on the Single Machine instance to the Clustered version.

  2. Configuring the instances to run in migration mode - Configures Single Machine and Clustered instances to connect to each other to migrate the data.

  3. Migrating the databases - Moves data from the Single Machine instance to the Clustered instance. Any new events processed in the Single Machine instance are migrated over. The Clustered instance will not process any events sent to it.

  4. Configuring the instances to disable migration mode - Configures Single Machine and Clustered instances to disable migration mode. Once done the instances are not connected to each other and will process events separately. After this point you cannot go back to migration mode.

It’s highly recommended to run Clustered On-premise in non-migration first before attempting a migration to ensure that the cluster is setup correctly. You can then clear the data prior to a migration by running kubectl delete namespace bugsnag.

When configuring the Clustered instance make sure the cluster is using appropriately sized nodes and the storage sizes for the databases are set correctly as these cannot be changed very easily after the migration is complete. Please contact us for recommendations according to your usage.

Once the migration has been completed and both instances are running in non-migration mode, you can safely change the DNS to route traffic to the new instance or reconfiguring the notify / session end-points in the application.

Running the migration

The migration can take some time and is dependent on the number of events you have, however you can continue to use your current Single Machine instance whilst the data is migrating. We estimate that it takes around 1 hour to migrate 8 million events of data. There will be some downtime after the data has been migrated and Clustered BugSnag On-premise starts, this may take around 10-15 mins.

Migrating the configuration

Download migrate-settings.rb on the Single Machine instance. Run the following command: ruby migrate-settings.rb. This will generate a config required for the migration: config.yaml.

Copy config.yaml to where you will be starting the migration.

Configuring the instances to run in migration mode

Install Replicated KOTS kubectl plugin

Run the install script to download and install the Replicated KOTS kubectl plugin.

curl https://kots.io/install | bash

Install Replicated KOTS admin console

kubectl kots install bugsnag-clustered-kots/stable \
  --name Bugsnag \
  --license-file ./license.yaml \
  --config-values ./config.yaml \
  --namespace bugsnag

The admin console will be available at http://localhost:8800

Migrate databases

On the clustered instance, in the configuration tool available at http://localhost:8800:

  1. In the config section under Migration: Enable migration mode by ticking the checkbox: img
  2. (Optional) Configure the “MongoDB node port” and “Uploads storage node port” settings under the Migration section.
  3. Configure BugSnag to the settings you want to be using after the migration has completed in particular make the sure the following configuration is set correctly:
    • Enable high availability features under the Kubernetes section if you are going to be running BugSnag in high availability mode after migration has completed. img
    • Ensure the number of availability zones is set correctly under the Kubernetes section: img
    • Ensure that the storage sizes for the stateful services under Storage are set correctly as editing these values after initial configuration will require recreation of the stateful services: img
  4. Click “Save config” to deploy BugSnag. Make a note of the values in the migration config section which will be required as configuration on your Single Machine Bugsnag instance.

On the Single Machine instance:

  1. Configure via Replicated settings to enable migration mode: img
  2. Configure this section using the values output earlier:
    • “Kubernetes MongoDB instance address”: This should be configured to the address of the MongoDB instance running on Kubernetes, usually this can be in the form of “Kubernetes Node IP:MongoDB node port”, this address should be accessible from the Single Machine BugSnag instance.
    • “Current MongoDB instance address”: This should be configured to the address of the MongoDB instance running on the Single Machine BugSnag instance, usually this can be in the form of “Single Machine instance IP:57017”, this address should be accessible from the Kubernetes cluster.
    • “Address of uploads storage”: This should be configured to the address of the uploads storage instance running on Kubernetes, usually this can be in the form of “Kubernetes Node IP:Uploads storage node port”
    • “Access key for uploads storage”: Use the value from the earlier output.
    • “Secret key for uploads storage”: Use the value from the earlier output.
  3. Save and restart BugSnag when prompted, the migration will start once BugSnag has restarted. You can check the progress of the migration using the instructions below:

Monitoring uploads migration progress

You can check the progress of the uploads migration by running the following on the Single Machine BugSnag instance:

docker logs bugsnag_migrate-uploads

Once this states “Migration of uploads completed.”, this migration is complete.

Monitoring MongoDB migration progress

You can check the progress of the MongoDB migration by checking the status of the replica set:

kubectl -n <NAMESPACE> exec -it mongo-0 -- mongo-local --eval "rs.status()"

If the MongoDB instance in Kubernetes is in STARTUP2 mode you can check the progress of the collection copying using:

kubectl -n <NAMESPACE> logs -f mongo-0 | grep "repl writer worker"

Once the instance is in SECONDARY mode you can use the following to check the replication lag:

kubectl -n <NAMESPACE> exec -it mongo-0 -- mongo-local --eval "rs.printSlaveReplicationInfo()"

Once this states that the MongoDB instance is 0 secs behind master, this migration is complete.

Migrate redis

Once MongoDB has finished migrating run migrate-redis.sh on the Single Machine BugSnag instance and select “Backup redis data” option to begin backing up the redis data from the instance. Once that has completed you can use the “Restore redis data to kubernetes” option to begin the restore. If you do not have access to the Kubernetes cluster from that instance you should copy the output archive to an instance which does along with the script and run migrate-redis.sh using the restore option and specifying the archive location to begin migrating the redis data to the Clustered instance.

Configuring the instances to disable migration mode

Note that once migration mode has been disabled on both instances any new events sent to the Single Machine instance will not be migrated and there will be a period of downtime until the Clustered installation is fully running and the error traffic has been routed over to the new instance. Also migration mode cannot be re-enabled once it has been disabled. You will have to start the migration from scratch.

Once all the migrations are complete, to disable migration mode and run the two instances separately:

On the Single Machine instance configure the Single Machine BugSnag instance via Replicated settings to disable migration mode and restart: img

Run the following to allow access to the Replicated admin console for the clustered installation:

kubectl kots admin-console -n bugsnag

Disable migration mode and deploy the config change.

img

Once BugSnag is running, Elasticsearch will require a reindex, which you can monitor using “Events in Elasticsearch/Mongo” graph under the “Support” dashboard on Grafana. There will be a period where historical events will not be available in the dashboard until the reindex has completed, any new events will show up on the dashboard immediately. We estimate that it takes around 1 hour to reindex 1 million events of data.