Cluster management

How to use clustering to scale your Bugsnag On-premise installation.

Clustering

If you are on a Bugsnag On-premise plan that supports clustering you can distribute and scale the application containers across multiple hosts. This will allow you to get the best performance out of Bugsnag if you have a lot of traffic or store a lot of data.

Nodes

Upon installation of Bugsnag on-premise, the host you install on will be designated the ‘master’ node in the cluster. It will host any containers with ports that should be exposed to the world and should always have the 'master’ tag. Only one node in the cluster should ever have the 'master’ tag to ensure ports are always available on the expected host.

img

Add node

To scale beyond your initial node, first create a physical host with Docker on it (see installation instructions for which Docker version to install). The specifications of the machine will depend on what you intend to use it for. The firewalls of the new machine and the 'master’ machine should be configured such that they can both communicate with each other on all ports - all nodes in the cluster must be able to talk to each other on any port.

Next go to the Cluster settings on the management site and click “Add node”. Ensure that you add the node via the “Installation Script” and not via “Docker Run”. At this point don’t select any tags, just copy the command and run it on the new machine.

img

When running the script, if prompted ensure the private IP address is bound to the interface that is on the same network as the 'master’ node.

At this point, the node should appear in the Cluster settings of the management site, indicating it is both “connected” and “initialized”. If the node shows an initialization warning, click the warning icon to review the issues and decide whether or not to proceed or update the host.

img

Assign tags

In order to assign containers to your new node you will need to tag it appropriately from the Cluster settings. Here we will discuss all the possible tags and how they should be used.

Tag: master

The master tag should only ever be applied to the initial node that you install Bugsnag on-premise on. This will ensure externally facing ports are always available on that host.

Tags for stateless services

Stateless services can be easily scaled and redistributed around the cluster without having to move data around.

Tag: event-server

The event-server tag can be used on any number of nodes, each of which will have a configurable number of “event servers” running on it. Increasing the number of event servers in this way should give greater capacity and reliability when the event notify traffic is very high. A cluster restart is required for this to take effect.

Tag: session-server

The session-server tag can be used on any number of nodes, each of which will have a configurable number of “session servers” running on it. Increasing the number of session servers in this way should give greater capacity and reliability when the session notify traffic is very high. A cluster restart is required for this to take effect.

Tag: event-processing

The event-processing tag can be used on any number of nodes, each of which will have a configurable number of “event workers” running on it along with other apps involved in processing events. Increasing the number of event workers should allow the system to process events faster when the event notify traffic is very high. This will take effect immediately without needing to restart the cluster.

Tags for data services

Services with data currently can not be moved from the master node after initial installation. For other nodes they may be moved post installation, but the original node must be removed from the cluster and the data manually migrated to the new node. Please contact our support team to discuss data migration if you think you may require it.

Tag: elasticsearch

The elasticsearch tag should be used on only one node. At initial installation, a container running Elasticsearch will be tied to that node and the data saved to <Bugsnag data directory>/elasticsearch on that host. Once Elasticsearch is established on a node it currently can not be moved unless that node has been removed from the cluster (you can not remove the master node). If you do relocate Elasticsearch, it is advisable that you manually copy the data directory to the same location on the new host otherwise it will need to rebuild itself which may take some time.

Tag: mongodb

The mongodb tag should be used on only one node. At initial installation, containers running MongoDB will be tied to that node and the data saved to <Bugsnag data directory>/mongodb on that host. Once MongoDB is established on a node it currently can not be moved unless that node has been removed from the cluster (you can not remove the master node). If you do relocate MongoDB, it is essential that you manually copy the data directory to the same location on the new host to avoid data loss.

Tag: rabbitmq

The rabbitmq tag should be used on only one node. At initial installation, a container running RabbitMQ will be tied to that node and the data saved to <Bugsnag data directory>/rabbitmq on that host. Once RabbitMQ is established on a node it currently can not be moved unless that node has been removed from the cluster (you can not remove the master node). If you do relocate RabbitMQ, it is advisable that you manually copy the data directory to the same location on the new host.

Tag: redis

The redis tag should be used on only one node. At initial installation, all containers running Redis will be tied to that node and the data saved to <Bugsnag data directory>/redis-* (one for each Redis) on that host. Once Redis is established on a node it currently can not be moved unless that node has been removed from the cluster (you can not remove the master node). If you do relocate the Redises, it is essential that you manually copy all the Redis data directories to the same location on the new host to avoid data loss.

Remove node

Before removing a node from the cluster, check whether it is tagged with either elasticsearch, mongodb, rabbitmq or redis. If it is you will lose data if you do not copy it to another node and tag that new node appropriately.

It should also be noted that the 'master’ node - the primary node in the cluster on which the core app containers are run - can not be removed.

To remove a node do the following:

  • Stop Bugsnag On-premise from the management console.
  • Wait for the containers to stop.
  • Run sudo service replicated-operator stop on the node you wish to remove (Note: the command for stopping a service may differ depending on your Linux distribution).
  • If the node is tagged with either elasticsearch, mongodb, rabbitmq or redis, ensure you copy the data over to another node which will be given the same tag to avoid data loss. See the tag definitions above for information about where the data is stored.
  • On the Cluster settings page on the management console, you will now be able to delete the node.
  • Ensure any required tags are set on one of the remaining nodes.
  • Start Bugsnag On-premise from the management console.

Change IP address of node

If you change the private IP address of a node, you must update it in the node’s configuration or services may not be able to talk to each other.

To update the IP address, connect to the host and modify /etc/default/replicated-operator and update PRIVATE_ADDRESS with the new IP address. Once you have done this, restart the Replicated operator:

sudo service replicated-operator restart