Migrating from Single Machine to Clustered Bugsnag On-premise
To get started with your migration from Single Machine to Clustered Bugsnag On-premise, you’ll need the following things prepared in advance:
The migration process has the following steps:
Migrating the configuration - Converts the configuration on the Single Machine instance to the Clustered version.
Configuring the instances to run in migration mode - Configures Single Machine and Clustered instances to connect to each other to migrate the data.
Migrating the databases - Moves data from the Single Machine instance to the Clustered instance. Any new events processed in the Single Machine instance are migrated over. The Clustered instance will not process any events sent to it.
Configuring the instances to disable migration mode - Configures Single Machine and Clustered instances to disable migration mode. Once done the instances are not connected to each other and will process events separately. After this point you cannot go back to migration mode.
It’s highly recommended to run Clustered On-premise in non-migration first before attempting a migration to ensure that the cluster is setup correctly. You can then clear the data prior to a migration by running
kubectl delete -f bugsnag-kubernetes.yaml.
When configuring the Clustered instance make sure the cluster is using appropriately sized nodes and the storage sizes for the databases are set correctly as these cannot be changed very easily after the migration is complete. Please contact us for recommendations according to your usage.
Once the migration has been completed and both instances are running in non-migration mode, you can safely change the DNS to route traffic to the new instance or reconfiguring the notify / session end-points in the application.
The migration can take some time and is dependent on the number of events you have, however you can continue to use your current Single Machine instance whilst the data is migrating. We estimate that it takes around 1 hour to migrate 8 million events of data. There will be some downtime after the data has been migrated and Clustered Bugsnag On-premise starts, this may take around 10-15 mins.
Run migrate-settings.sh on the Single Machine instance. This will generate a config required for the migration:
.ship folder from where you will be starting the migration. The instance this file is copied to will require access to the Kubernetes cluster.
On the instance that has access to the Kubernetes cluster run the following to run the configuration tool from the folder containing the
docker pull replicated/ship && \ docker run -it -p 8800:8800 --rm -v `pwd`:/out \ -v /var/run/docker.sock:/var/run/docker.sock \ -e HTTP_PROXY -e HTTPS_PROXY -e NO_PROXY \ replicated/ship init --preserve-state \ "replicated.app/bugsnag-clustered"
You will be asked to enter your Clustered Bugsnag On-premise license. Enter the license you received from Bugsnag to continue to the next step.
The configuration tool will be available at
On the clustered instance, in the configuration tool available at
On the Single Machine instance:
You can check the progress of the uploads migration by running the following on the Single Machine Bugsnag instance:
docker logs bugsnag_migrate-uploads
Once this states “Migration of uploads completed.”, this migration is complete.
You can check the progress of the MongoDB migration by checking the status of the replica set:
kubectl -n <NAMESPACE> exec -it mongo-0 -- mongo-local --eval "rs.status()"
If the MongoDB instance in Kubernetes is in
STARTUP2 mode you can check the progress of the collection copying using:
kubectl -n <NAMESPACE> logs -f mongo-0 | grep "repl writer worker"
Once the instance is in
SECONDARY mode you can use the following to check the replication lag:
kubectl -n <NAMESPACE> exec -it mongo-0 -- mongo-local --eval "rs.printSlaveReplicationInfo()"
Once this states that the MongoDB instance is 0 secs behind master, this migration is complete.
Once MongoDB has finished migrating run migrate-redis.sh on the Single Machine Bugsnag instance and select “Backup redis data” option to begin backing up the redis data from the instance. Once that has completed you can use the “Restore redis data to kubernetes” option to begin the restore. If you do not have access to the Kubernetes cluster from that instance you should copy the output archive to an instance which does along with the script and run migrate-redis.sh using the restore option and specifying the archive location to begin migrating the redis data to the Clustered instance.
Note that once migration mode has been disabled on both instances any new events sent to the Single Machine instance will not be migrated and there will be a period of downtime until the Clustered installation is fully running and the error traffic has been routed over to the new instance. Also migration mode cannot be re-enabled once it has been disabled. You will have to start the migration from scratch.
Once all the migrations are complete, to disable migration mode and run the two instances separately:
On the Single Machine instance configure the Single Machine Bugsnag instance via Replicated settings to disable migration mode and restart:
On the instance with access to the Clustered configuration and Kubernetes cluster, run the following to reconfigure Bugsnag:
docker pull replicated/ship && \ docker run -it -p 8800:8800 --rm -v `pwd`:/out \ -v /var/run/docker.sock:/var/run/docker.sock \ -e HTTP_PROXY -e HTTPS_PROXY -e NO_PROXY \ replicated/ship edit
Disable migration mode and apply the generated Kubernetes YAML.
Once Bugsnag is running, Elasticsearch will require a reindex, which you can monitor using “Events in Elasticsearch/Mongo” graph under the “Support” dashboard on Grafana. There will be a period where historical events will not be available in the dashboard until the reindex has completed, any new events will show up on the dashboard immediately. We estimate that it takes around 1 hour to reindex 1 million events of data.