Bulk Loading Data¶

About the bulk loading command¶

The bulk loading command is interruptible and idempotent

If the import is interrupted for any reason, it continues from where it left off the next time you run it. You can run the command repeatedly with the same data file, and the data is imported only once.

You can run the bulk loading command on a live system

Although you don’t need to shut down Mattermost to run the command, changes made by users of the system between runs can be overwritten if the corresponding fields exist in the data file.

Some data fields are optional

Not all fields are mandatory. If an optional field is missing from the object that is being imported, the field’s current value in the database is not changed.

The bulk loading command is not a synchronization tool

You cannot use the bulk loading command to remove any objects or their fields from the Mattermost database. The command only creates or overwrites fields.

Running the bulk loading command¶

Before running the bulk loading command, you must first create a JSON file that contains the data that you want to import. After you create the file, run the bulk load command in validation mode. In this mode, the data is checked for correctness, but is not written to the database. After validating, run the command in apply mode, which saves the data to the database.

To bulk load data:

1. Create the JSON data file in your Mattermost bin directory. The file can have any name, but in this procedure it’s called data.json. The format of the file is described in the Data Format section.
2. Validate that the file is correct:

1. Change to the Mattermost bin directory.

cd /opt/mattermost/bin (the location might be different on your system)

2. Run the following command:

sudo ./platform import bulk data.json --validate

3. Resolve any errors that are reported, and validate the file again. Do not go to the next step until you can run the validate command without errors.
4. Run the bulk load command in apply mode:

sudo ./platform import bulk data.json --apply

5. When the bulk load command completes, clear all caches. Open the System Console, and click General > Configuration > Purge All Caches.

Data Format¶

The input data file must be a valid JSON file with the following objects, each on its own line in the file. The objects must occur in the file in the order listed.

Version

Mandatory. The Version object must be the first line in the file, and must occur only once.

Team

Optional. If present, Team objects must occur after the Version object and before any Channel objects.

Channel

Optional. If present, Channel objects must occur after all Team objects and before any User objects.

User

Optional. If present, User objects must occur after the Team and Channel objects in the file and before any Post objects. Each User object defines the teams and channels that the user is a member of. If the corresponding teams and channels are not in the data file, then they must exist in the Mattermost database.

Post

Optional. If present, Post objects must occur after the last User object but before any DirectChannel objects. Each Post object defines the team, the channel, and the username of the user who posted the message. If the corresponding team, channel, or user are not in the data file, then they must exist in the Mattermost database.

DirectChannel

Optional. If present, DirectChannel objects must occur after all Post objects in the file and before any DirectPost objects.

DirectPost

Optional. If present, DirectPost objects must occur after all other objects in the file. Each DirectPost object defines the usernames of the channel members and the username of the user who posted the message. If the corresponding usernames are not in the data file, then they must exist in the Mattermost database.

With the exception of the Version object, each object has a field or a combination of fields that is used as the unique identifier of that object. The bulk loader uses the unique identifier to determine if the object being imported is a new object or an update to an existing object.

The identifiers for each object are listed in the following table:

The following fragment is from a file that imports two teams, each with two channels, many users, and many posts.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22

{ type: "version", ... }
{ type: "team", team: { name: "TeamA", ...} }
{ type: "team", team: { name: "TeamB", ...} }
{ type: "channel", channel: { team: "TeamA", name: "ChannelA1", ...} }
{ type: "channel", channel: { team: "TeamA", name: "ChannelA2", ...} }
{ type: "channel", channel: { team: "TeamB", name: "ChannelB1", ...} }
{ type: "channel", channel: { team: "TeamB", name: "ChannelB1", ...} }
{ type: "user", user: { username: "user001", ...} }
{ type: "user", user: { username: "user002", ...} }
{ type: "user", user: { username: "user003", ...} }
{ type: "user", ... }
{ type: "user", ... }
{ type: "user", ... }
.
.
.
{ type: "post", { team: "TeamA", name: "ChannelA1", user: "user001", ...} }
{ type: "post", { team: "TeamA", name: "ChannelA1", user: "user001", ...} }
{ type: "post", { team: "TeamA", name: "ChannelA1", user: "user001", ...} }
.
.
.

{
  type: "version",
  version: 1
}

{
type: "team",
team: {
  name: "team-name",
  display_name: "Team Display Name",
  type: "O",
  description: "The Team Description",
  allow_open_invite: true
  }
}

{
  type: "channel",
  channel: {
    team: "team-name",
    name: "channel-name",
    display_name: "Channel Name",
    type: "O",
    header: "The Channel Header",
    purpose: "The Channel Purpose",
  }
}

{
  type: "user",
  user: {
    username: "username",
    email: "email@example.com",
    auth_service: "",
    auth_data: "",
    password: "passw0rd",
    nickname: "bobuser",
    first_name: "Bob",
    last_name: "User",
    position: "Senior Developer",
    roles: "system_user",
    locale: "pt_BR",
    teams: [
      {
        name: "team-name",
        roles: "team_user team_admin",
        channels: [
          {
            name: "channel-name",
            roles: "channel_user",
            notify_props: {
              desktop: "default",
              mark_unread: "all"
            }
          }
        ]
      }
    ]
  }
}

{
  type: "post",
  post: {
    team: "team-name",
    channel: "channel-name",
    user: "username",
    message: "The post message",
    create_at: 140012340013
  }
}

{
  "type": "direct_channel",
  "direct_channel": {
    "members": [
      "username1",
      "username2",
      "username3"
    ],
    "header": "The Channel Header",
  }
}

{
  "type": "direct_post",
  "direct_post": {
    "channel_members": [
      "username1",
      "username2",
      "username3",
    ],
    "user": "username2",
    "message": "Hello Group Channel",
    "create_at": 140012340013
  }
}