High Availability Cluster (E20)¶

Initial Setup Guide for High Availability¶

To ensure your instance and configuration are compatible with high availability, please review the Configuration and Compatibility section.

1. Upgrade Mattermost Server to version 4.0 or later. See Upgrading Mattermost Server.
2. Set up a new Mattermost server with version 4.0 or later by following one of our Install Guides. This server must use an identical copy of the configuration file, config.json. Verify the servers are functioning by hitting each independent server through its private IP address.
3. Modify the config.json files on both servers to add ClusterSettings as described in High Availability.
4. Verify the configuration files are identical on both servers then restart each machine in the cluster.
5. Modify your NGINX setup so that it proxies to both servers. For more information about this, see Proxy Server Configuration.
6. Open System Console > Advanced > High Availability to verify that each machine in the cluster is communicating as expected with green status indicators. If not, investigate the log files for any extra information.

Adding a Server to the Cluster¶

1. Backup your Mattermost database and the file storage location. For more information about backing up, see Backup & Disaster Recovery.
2. Set up a new Mattermost server. This server must use an identical copy of the configuration file, config.json. Verify the server is functioning by hitting the private IP address.
3. Modify your NGINX setup to add the new server. For information about this, see Proxy Server Configuration.
4. Open System Console > Advanced > High Availability to verify that all the machines in the cluster are communicating as expected with green status indicators. If not, investigate the log files for any extra information.

Removing a Server from the Cluster¶

1. Backup your Mattermost database and the file storage location. For more information about backing up, see Backup & Disaster Recovery.
2. Modify your NGINX setup to remove the server. For information about this, see Proxy Server Configuration.
3. Open System Console > Advanced > High Availability to verify that all the machines remaining in the cluster are communicating as expected with green status indicators. If not, investigate the log files for any extra information.

Mattermost Server Configuration¶

upstream backend {
        server 10.10.10.2:8065;
        server 10.10.10.4:8065;
  }

  server {
      server_name mattermost.example.com;

      location / {
            client_max_body_size 50M;
            proxy_set_header Upgrade $http_upgrade;
            proxy_set_header Connection "upgrade";
            proxy_set_header Host $http_host;
            proxy_set_header X-Real-IP $remote_addr;
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
            proxy_set_header X-Forwarded-Proto $scheme;
            proxy_set_header X-Frame-Options SAMEORIGIN;
            proxy_pass http://backend;
      }
}

Updating Configuration Changes While Operating Continuously¶

A service interruption is not required for most configuration updates. See Server Upgrades Requiring Service Interruption for a list of configuration updates that require a service interruption.

You can apply updates during a period of low load, but if your HA cluster is sized correctly, you can do it at any time. The system downtime is brief, and depends on the number of Mattermost servers in your cluster. Note that you are not restarting the machines, only the Mattermost server applications. A Mattermost server restart generally takes about 5 seconds.

1. Make a backup of your existing config.json file.
2. For one of the Mattermost servers, make the configuration changes to config.json and save the file. Do not reload the file yet.
3. Copy the config.json file to the other servers.
4. Shut down Mattermost on all but one server.
5. Reload the configuration file on the server that is still running. Go to System Console > Configuration and click Reload Configuration from Disk
6. Start the other servers.

Updating Server Version While Operating Continuously¶

A service interruption is not required for security patch dot releases of the Mattermost server.

You can apply updates during a period when the anticipated load is small enough that one server can carry the full load of the system during the update.

Note that you are not restarting the machines, only the Mattermost server applications. A Mattermost server restart generally takes about 5 seconds.

1. Review the upgrade procedure in the Upgrade Enterprise Edition section of Upgrading Mattermost Server.
2. Make a backup of your existing config.json file.
3. Set your proxy to move all new requests to a single server. If you are using NGINX and it’s configured with an upstream backend section in /etc/nginx/sites-available/mattermost then comment out all but the one server that you intend to update first, and reload NGINX.
4. Shut down Mattermost on each server except the one that you are updating first.
5. Update each Mattermost instance that is shut down.
6. On each server, replace the new config.json file with your backed-up copy.
7. Start the Mattermost servers.
8. Repeat the update procedure for the server that was left running.

Server Upgrades Requiring Service Interruption¶

A service interruption is required when the upgrade includes a change to the database schema or when a change to config.json requires a server restart, such as when making the following changes:

• Default Server Language
• Rate Limiting
• Webserver Mode
• Database
• High Availability

If the upgrade includes a change to the database schema, the database is upgraded by the first server that starts.

Apply upgrades during a period of low load. The system downtime is brief, and depends on the number of Mattermost servers in your cluster. Note that you are not restarting the machines, only the Mattermost server applications.

1. Review the upgrade procedure in the Upgrade Enterprise Edition section of Upgrading Mattermost Server.
2. Make a backup of your existing config.json file.
3. Stop NGINX.
4. Upgrade each Mattermost instance.
5. On each server, replace the new config.json file with your backed-up copy.
6. Start one of the Mattermost servers.
7. When the server is running, start the other servers.
8. Restart NGINX.

Upgrading to Version 4.0 and Later¶

Starting with Mattermost Server version 4.0, when a server starts up it can automatically discover other servers in the same cluster. You can add and remove servers without the need to make changes to the configuration file, config.json. To support this capability, new items were added to the ClusterSettings section of config.json. When upgrading from 3.10 or earlier to 4.0 or later, you must manually add the new items to your existing config.json.

1. Review the upgrade procedure in Upgrading Mattermost Server.
2. Make a backup of your existing config.json file.
3. Revise your existing config.json to update the ClusterSettings section. The following settings should work in most cases:

"ClusterSettings": {
    "Enable": true,
    "ClusterName": "production",
    "OverrideHostname": "",
    "UseIpAddress": true,
    "UseExperimentalGossip": false,
    "ReadOnlyConfig": true,
    "GossipPort": 8074,
    "StreamingPort": 8075
},

For more information about these settings, see High Availability.

4. Stop NGINX.
5. Upgrade each Mattermost instance.
6. On each server, replace the new config.json file with your modified version.
7. Start one of the Mattermost servers.
8. When the server is running, start the other servers.
9. Restart NGINX.

Does Mattermost support multi-region high availability deployment?¶

Yes. Although not officially tested, you can set up a cluster across AWS regions, for example, and it should work without issues.

Red Server Status¶

When high availability is enabled, the System Console displays the server status as red or green, indicating if the servers are communicating correctly with the cluster. The servers use inter-node communication to ping the other machines in the cluster, and once a ping is established the servers exchange information, such as server version and configuration files.

A server status of red can occur for the following reasons:

• Configuration file mismatch: Mattermost will still attempt the inter-node communication, but the System Console will show a red status for the server since the high availability feature assumes the same configuration file to function properly.
• Server version mismatch: Mattermost will still attempt the inter-node communication, but the System Console will show a red status for the server since the high availability feature assumes the same version of Mattermost is installed on each server in the cluster. It is recommended to use the latest version of Mattermost on all servers. Follow the upgrade procedure in Upgrading Mattermost Server for any server that needs to be upgraded.
• Server is down: If an inter-node communication fails to send a message it makes another attempt in 15 seconds. If the second attempt fails, the server is assumed to be down. An error message is written to the logs and the System Console shows a status of red for that server. The inter-node communication continues to ping the down server in 15 second intervals. When the server comes back up, any new messages are sent to it.

WebSocket Disconnect¶

When a client WebSocket receives a disconnect it will automatically attempt to re-establish a connection every three seconds with a backoff. After the connection is established, the client attempts to receive any messages that were sent while it was disconnected.