High Availability (E20) - Beta¶

Initial Setup Guide for High Availability¶

To ensure your instance and configuration are compatible with high availability, please review the Configuration and Compatibility section.

1. Follow our upgrade guide to upgrade your Mattermost server to v3.3 or later.
2. Setup a new Mattermost server with v3.3 or later following one of our Install Guides. This server must use an identical copy of the configuration file, config.json. Verify the servers are functioning by hitting each independent server through it’s private IP address.
3. Modify the config.json files on both servers to add the ClusterSettings as described in this documentation.
4. Verify the configuration files are identical on both servers then restart each machine in the cluster.
5. Modify your NGINX setup so it proxies to both servers.
6. Open the System Console > Advanced > High Availability (Beta) to verify all the machines in the cluster are communicating as expected with green status indicators. If not, investigate the log files for any extra information.

Adding a Server to the Cluster¶

1. Backup your Mattermost database and file storage location.
2. Setup a new Mattermost server following one of our Install Guides. This server must use an identical copy of the configuration file, config.json. Verify the server is functioning by hitting the private IP address.
3. Modify the config.json files on all servers with the ClusterSettings as described in this documentation. Be sure to add the new server URL to InterNodeUrls.
4. Verify the configuration files are identical on all servers, then restart each machine in the cluster in sequence with 10 or more seconds between each restart.
5. Modify your NGINX setup to add the new server.
6. Open the System Console > Advanced > High Availability (Beta) to verify all the machines in the cluster are communicating as expected with green status indicators. If not, investigate the log files for any extra information.

Removing a Server from the Cluster¶

1. Backup your Mattermost database and file storage location.
2. Modify your NGINX setup to remove the server.
3. On all servers staying active in the cluster, modify the ClusterSettings in config.json to remove the server from InterNodeUrls
4. Verify the configuration files are identical on all servers, then restart each machine in the cluster in sequence with 10 or more seconds between each restart.
5. Open the System Console > Advanced > High Availability (Beta) to verify all the machines in the cluster are communicating as expected with green status indicators. If not, investigate the log files for any extra information.

Mattermost Server Configuration¶

"ClusterSettings": {
      "Enable": false,
      "InterNodeListenAddress": ":8075",
      "InterNodeUrls": []
}

Proxy Server Configuration¶

The proxy server will expose the cluster of Mattermost servers to the outside world. The Mattermost servers are designed for use with a proxy server like NGINX, hardware load balancer, or a cloud service like Amazon Elastic Load Balancer.

If you wish to monitor the server with a health check you can use http://10.10.10.2/api/v3/general/ping and check the response for Status 200, indicating success. Use this health check route to mark the server in-service or out-of-service.

A sample configuration for NGINX is provided below. It assumes you have two Mattermost servers running on private IP addresses of 10.10.10.2 and 10.10.10.4.

upstream backend {
        server 10.10.10.2:8065;
        server 10.10.10.4:8065;
  }

  server {
      server_name mattermost.example.com;

      location / {
            client_max_body_size 50M;
            proxy_set_header Upgrade $http_upgrade;
            proxy_set_header Connection "upgrade";
            proxy_set_header Host $http_host;
            proxy_set_header X-Real-IP $remote_addr;
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
            proxy_set_header X-Forwarded-Proto $scheme;
            proxy_set_header X-Frame-Options SAMEORIGIN;
            proxy_pass http://backend;
      }
}

A setup with multiple proxy servers can be utilized to limit a single point of failure, but is beyond the scope of this documentation.

File Storage Configuration¶

If you’re using the Compliance Reports feature in Enterprise Edition E20, you will need to configure the "ComplianceSettings": "Directory": "./data/", to share between all machines or the reports will only be available from the System Console on the local Mattermost server.

Migrating to NAS or S3 from local storage is beyond the scope of this document.

Database Configuration¶

Scaling the database can be accomplished by utilizing the read-replica feature. The Mattermost server can be set up to use one “master” database and up to 8 read replica databases. Mattermost distributes read requests across all databases, and sends write requests to the master database, and those changes are then sent to update the read replicas.

Red Server Status¶

When high availability is enabled, the System Console displays the server status as red or green, indicating if the servers are communicating correctly with the cluster. The servers use inter-node communication to ping the other machines in the cluster, and once a ping is established the servers exchange information, such as server version and configuration files. Red server status may display for the following reasons:

• Configuration file mismatch: Mattermost will still attempt the inter-node communication, but the System Console will show a red status for the server since the high availability feature assumes the same configuration file to function properly.
• Server version mismatch: Mattermost will still attempt the inter-node communication, but the System Console will show a red status for the server since the high availability feature assumes the same version of Mattermost is installed on each server in the cluster. It is recommended to use the latest version of Mattermost on all servers. Follow the upgrade procedure for any server that needs to be upgraded.
• Server is down: If an inter-node communication fails to send a message it will attempt again in 15 seconds. If the second attempt fails, the server is assumed to be down. An error message is written to the logs and the System Console will show a status of red for that server.

WebSocket Disconnect¶

When a client WebSocket receives a disconnect it will automatically attempt to re-establish a connection every three seconds with a backoff. Once the connection is established the client will attempt to receive any missing messages since it was disconnected.