Release Process¶

A. (Code complete date of previous release) Beginning of release¶

Pre-work for the current release begins at the code complete date of the previous release. See “Code Complete” section below for details.

B. (T-minus 15 working days) Cut-off for submitting major features¶

No pull requests for major features should be submitted to the current release after this date (except if release manager decides to add “release-exception” label to Jira ticket).

1. Release Manager:
   • Post this checklist in Release channel
2. PM:
   • Prioritize reviewing major features, ensuring any bugs and UX issues get fixed
   • Check that all major features are behind a feature flag
3. Dev:
   • Prioritize reviewing, updating, and merging of pull requests for major features
4. Marketing:
   • Confirm each Enterprise feature is in the correct pricing SKU, if not alert the release manager

C. (T-minus 12 working days) Cut-off for merging major features¶

No pull requests for major features should be merged to the current release after this date (except if release manager decides to add “release-exception” label to Jira ticket).

1. Release Manager:
   • Post this checklist in Release channel
   • Verify all items in the last posted release checklist are complete
2. Logistics:
   • Start posting a daily Zero Bug Balance query
   • Queue an item for UX meeting to discuss worst UX bug and to do a 10-minute UI/UX bug bash
   • Notify community about upcoming release in Reception
3. PM:
   • PM area owners complete draft of Changelog in a WIP PR with updates for highlights, feature additions, known issues, compatibility updates for deprecated features, config.json, database changes, API changes (search #api-proposal and confirm with Dev) and WebSocket event changes; see example
   • Review and update company roadmap with which major features made it into the release
   • PM feature owners post draft section for the blog post in the Marketing Channel (including screenshots and a hashtag #mattermostXX where XX is the version number, see example thread) and queue a tweet
   • Backlog is reviewed and tickets that won’t make it are moved to next release
4. Docs:
   • Update Upgrade Guide for any steps needed to upgrade to new version
   • Submit NOTICE.txt PR for any new libraries added from dev, if not added already. The following two files contain a list of dependencies:
     • https://github.com/mattermost/platform/blob/master/webapp/package.json
     • https://github.com/mattermost/platform/blob/master/glide.yaml
5. QA:
   • Coordinate testing:
     • Receive testing sign-off from feature area owners (i.e. PM/Dev either signs-off that their area is well tested, or flags potential quality issues that may exist)
     • Prioritize updating the RC Testing Spreadsheet to cover any changes or new features, and confirm that known issues are listed in the relevant tests
     • Assign each area of the release testing spreadsheet to a team member and ensure core team has access permissions
6. (Team) Major Feature Complete Meeting (10:00am San Francisco time):
   • Discuss worst bug currently on master
   • Discuss release highlights for marketing
   • Review status of last feature PRs to be merged
7. Dev:
   • Prioritize reviewing, updating, and merging of pull requests for current release until there are no more tickets in the pull request queue marked for the current release
   • After the cut-off, any PRs that include significant code changes require approval of the release manager before merging.
8. Marketing:
   • Prepare bullet points for release announcement
   • Propose list of key features included in the release GIF and send to marketing lead for review
   • Draft release headline and send to marketing lead for review

D. (T-minus 10 working days) Judgment Day¶

Day when leads and PM area owners decide which major features are included in the release, and which are postponed.

1. Release Manager:
   • Post this checklist in Release channel
   • Verify all items in the last posted release checklist are complete
2. Logistics:
   • Confirm, in Judgement Day meeting, date of marketing announcement for the release and update release channel header if needed
3. Leads:
   • Finalize roadmap for next release, and identify planned marketing bullet points
4. (Team) Judgment Day Meeting (10:00am San Francisco time):
   • Meet to discuss release update and finalize which major features will be in or out for the release
5. PM:
   • Based on discussion, create tickets for features that need to be turned on or off for the release
   • Backlog is reviewed and tickets that won’t make it are moved to next release
   • Post a link to Release Discussion room for query of remaining tickets in this release
   • Update Changelog PR based on what’s in/out of the release
   • Create meta issue for release in GitHub (see example)
   • Review the JIRA tickets remaining in the current release fix version and push those that won’t make it to the next fix version
   • Review any features that are currently in Beta and confirm with leads if there are any to be promoted
6. Marketing:
   • Begins to draft blog post, tweet, and email for the release announcement

E. (T-minus 8 working days) Code Complete and Release Candidate Cut¶

Stabilization period begins when all features for release have been committed. During this period, only bugs can be committed to the release branch. Non-bug pull requests are tagged for next version.

Exceptions can be made by the release manager setting priority to “Highest” and adding a “release-exception” label to the Jira ticket. This will add the ticket to the hotfix list for release candidate.

1. Release Manager:
   • Post this checklist in Release channel
   • Verify all items in the last posted release checklist are complete
2. Logistics:
   • Mail out mugs to any new contributors
   • Update Team page with new contributors
   • Add list of contributors to the Changelog draft
3. Dev:
   • Prioritize reviewing, updating, and merging of pull requests for current release until there are no more tickets in the pull request queue marked for the current release
4. PM:    - Review all Severity 1 bugs (data loss or security) to consider for adding to Hotfix list.
   • Update documentation:
     • Submit Changelog PR
     • Draft Mattermost Security Updates, but do not post until 14 days after official release
   • If there are any breaking compatibility changes in the release, open an issue in the GitLab Omnibus to make sure GitLab is aware. Post a link to the issue in the Release Discussion channel.
5. Docs:
   • Submit all doc PRs for review
   • Confirm changes to config.json in compatibility section of Changelog are written back to settings documentation
6. QA:
   • Confirm all PRs merged into the current release have been tested
7. (Team) Code Complete Meeting (10:00am San Francisco time):
   • (PM) Leads team review of Changelog
   • (Release Manager) Walk through each unfinished item of this checklist
   • (Dev) Last check of tickets that need to be merged before RC1
8. Build:
   • Review all TODO notes, including one for uncommenting upgrade code
   • Confirm all PRs in /enterprise repo have been merged.
   • Master is tagged and branched and “Release Candidate 1″ is cut (e.g. 3.5.0-RC1) according to the Release Candidate Checklist in mattermost/process
   • After branching, the database version in sql_upgrade.go on master is set to the next scheduled release version (e.g. 3.6.0)
   • CI servers are updated to the release branch
   • Translation server is locked to the release branch
9. PM:
   • Merge changelog PR after team review is complete, and update the GitHub meta issue to include a link to the changelog on the documentation branch
   • Tweet announcement that RC1 is ready (see example)

F. (T-minus 7 working days) Release Candidate Testing¶

1. Release Manager:
   • Post this checklist in Release channel
   • Verify all items in the last posted release checklist are complete
2. QA:
   • Update Release Discussion header with links to RC instances and testing spreadsheet
   • Post release testing instructions to Release Discussion channel (example)
   • Post “Known Issues” to Release Discussion channel
3. Dev:
   • Run loadtests against the release candidate to find potential performance issues
4. Logistics:
   • Confirm community testers are directed to the Release Discussion channel
   • Queue an item for UX meeting to do a 10-minute UI/UX bug bash
5. Team:
   • Test assigned areas of the Release Candidate Testing Spreadsheet and file any bugs found in Jira
   • Post a link to any “Blocking” issue that may need a hotfix to the RC in the Release room, with the #blocking tag. If the issue is security related or contains confidential information, post the link in the Confidential Bugs private channel. Blocking issues are considered to be security issues, data loss issues, and issues that break core functionality or significantly impact aesthetics.
   • Daily triage of hotfix candidates and decide on whether and when to cut next RC or final
6. PM:
   • Queue an item for Release Update meeting to discuss worst bug
   • Update the meta issue:
     • Post comments to the meta issue with approved fixes for the next RCs
     • Download links and testing server links to the RCs
   • Post screenshot and links to final tickets for next RC to the Release Discussion room
   • Update Changelog “Known Issues” section with any significant issues that were found and not fixed for the final release
7. Dev:
   • PRs for hotfixes made to release branch
   • Review PRs made from release branch and merge changes into both the release branch and master
8. Build:
   • Verify with Release Manager before cutting any new RCs (approved fixes should be merged)
   • Push next RC to acceptance and announce in Town Square with new RC link
9. QA:
   • Test the new RC to verify fixes merged to the release branch work
   • Post in Release Channel after testing

G. (T-minus 5 working days) Release Candidate Testing Finished¶

1. Release Manager:
   • Post this checklist in Release channel
   • Verify all items in the last posted release checklist are complete
   • Verify that the final translations PR for the release is submitted
2. Team:
   • Finish assigned areas of the Release Candidate Testing Spreadsheet
   • Continue triaging hotfix candidates and decide on whether and when to cut next RC or final
   • If no blocking issues are found, PM, Dev and Ops signs off on the release
3. PM:
   • Check that known issues section of Changelog is updated
   • Check that the contributors section of Changelog is updated (including contributors from all repos)
   • Check that if there is a security fix, the Changelog contains a note mentioning the severity level of the issue, recommending upgrade, and thanking the security researcher (see security process doc for example)
4. Marketing:
   • Finish draft of blog post for mattermost.com and send for marketing lead to review
     • Upgrade should be recommended if there are security fixes in this version, with a note thanking the security researcher
   • Finish drafts of all art work (screenshots, GIFs and twitter banners) used for the blog post and send to marketing lead for review
   • Find www-gitlab-com merge request for latest GitLab release blog post and make request for adding GitLab Mattermost update (see example request, example update). Post to Release Discussion channel with link to request.
5. QA:
   • Organize a mini-release testing spreadsheet for team members to carry out before final version is cut.

H. (T-minus 2 working days) Release Build Cut¶

The final release is cut. If an urgent and important issue needs to be addressed between major releases, a bug fix release (e.g. 1.1.1) may be created.

1. Release Manager:
   • Post this checklist in Release channel
   • Verify all items in the last posted release checklist are complete
2. Build:
   • Tags a new release (e.g. 1.1.0) and runs an official build which should be essentially identical to the last RC
   • Posts SHA key, md5 sum and GPG signatures of the final build to release channel
   • Post in Release Discussion with links to the EE and Team Edition bits
3. PM:
   • Close GitHub meta ticket for the release
   • Submit GitLab MR to take next Mattermost version in the Omnibus (see example):
     • Include changes to Mattermost version number (default_version) and md5 sum of the final TE build (source md5) in config/software/mattermost.rb
     • Include a summary of updates in Team Edition that are relevant to GitLab
     • Include an update to the GitLab changelog
     • Post a link to the MR in the Release Discussion channel
   • Check Security Issues spreadsheet and confirm disclosure text
   • Check the security researcher was added to the Responsible Disclosure Policy page
   • Confirm link to security updates appears in blog post if there are security updates in this release, with a note thanking the security researcher
   • Update deprecated feature list in mattermost.com with new and scheduled deprecations
4. Dev:
   • Verify the hashes (SHA-1, SHA-256 and md5) and GPG signatures are correct for both Team Edition and Enterprise Edition.
   • Test upgrade from previous version to current version, following the Upgrade Guide with database upgrades on both MySQL and Postgres
   • Test upgrade from Team Edition to Enterprise edition based on the Upgrade Guide
   • Review any changes made to install guides, and test if necessary
   • Run loadtests against the final release build to confirm there are no performance issues
5. Logistics:
   • Update the Mattermost server download page with the links to the EE and TE bits
     • Test the download links before and after updating the page
   • Update MVP page with the most valuable contributor of the release
6. Docs:
   • Add the download links and SHA-256 hash upgrade guide
   • Finalize docs
     • If reviews are not complete, hold a 30 minute doc review meeting with PMs and anyone else who has changed or reviewed docs this release and wants to join
     • Merge the docs release branch to master and verify all changes on docs.mattermost.com once the build is up
     • Submit a correction PR for any incorrect formatting or other errors missed during the initial review
   • Submit an MR to update GitLab Mattermost documentation
7. Marketing:
   • Finish draft of animated GIF (for Twitter announcement, MailChimp and blog post) made up of top announcements
   • Finish draft of MailChimp email blast and Twitter announcement and send for marketing lead to review. Once reviewed, schedule for 08:00 PST on the date of marketing announcement
   • Note: If the release contains a security update, also draft a Mailchimp email blast for the Security Bulletin mailing list
   • Finalize blog post for mattermost.com and set timer for 08:00 PST on the day of release
   • Turn on CrazyEgg for blog post page
   • Update feature list on mattermost.com with relevant new features

If a bug fix release is required, run through the following steps:

1. PM:
   • Schedule a team Release Update meeting every day until the dot release is complete
   • Post links to approved tickets for next dot release RC to the Release Discussion channel
   • Make a post in Town Square announcing the dot release. See example
   • Open an issue in the GitLab Omnibus mentioning a dot release is coming
   • Update the GitHub meta issue:
     • Change the title to “Mattermost vx.x.x - RCx”, where vx.x.x is the version number of the dot release
     • Post a comment to the meta issue with approved fixes for the next RC of the dot release
     • Post download links and testing server links for the next RC when it’s cut
   • Update Changelog:
     • Start a WIP PR for the dot release changelog and commit updates as new issues are fixed on the dot release RCs
     • Update “Known Issues” section with any significant issues that were found during RC testing and not fixed for the dot release
   • Push RC versions to acceptance and announce in Town Square with new RC link as they are cut
   • Test the new RC to verify fixes merged to the release branch work. Post in Release Discussion channel after testing
2. Dev:
   • PRs for hotfixes are made to release branch
   • Review PRs made from release branch and merge changes into both the release branch and master
3. Build:
   • Verify with Release Manager before cutting any new dot release RCs (approved fixes should be merged)
   • Push dot release RC’s to CI servers, pre-release and GitLab Mattermost.
4. QA:
   • Test the new RC to verify fixes merged to the release branch work. Post in Release Discussion channel after testing

Once final dot release build is ready to cut:

1. Build:
   • Tag a new release (e.g. 1.1.1) and run an official build
   • Update CI servers, pre-release and GitLab Mattermost to the final version
2. PM:
   • Update Mattermost pricing page if anything has changed
   • Merge the Changelog PR with notes on patch releases (see example entry)
   • Submit GitLab MR to update the version number and MD5 hash to the dot release version. See example
     • Test the upgrade once the MR is merged and the package is released to the GitLab package server
3. QA:
   • Verifies each of the issues in the patch release are fixed
4. Docs:
   • Update the version archive in the upgrade guide
   • Submit an MR to update GitLab Mattermost documentation
5. Logistics:
   • Update Mattermost server download page with the links to the EE and TE bits
     • Test the download links before and after updating the page
6. Marketing:
   • Prepare blog post for mattermost.com, MailChimp email blast, and Twitter announcement, and send for marketing lead to review. Once reviewed, schedule for 08:00 PST on the day after dot release
     • Note: If the release contains a security update, also draft a Mailchimp email blast for the Security Bulletin mailing list
     • Upgrade should be recommended if there are security fixes in the dot release version

I. (T-minus 0 working days) Release Day¶

1. Release Manager:
   • Post this checklist in Release channel
   • Verify all items in the last posted release checklist are complete, if not alert the release manager
2. Logistics:
   • Check for any UserVoice feature suggestions that were completed in the current release. Find the release tweet and insert a link to the tweet next to the feature that shipped with the release.
   • Post key dates for the next release in the header of the Release Discussion channel and remove links to RC candidates and testing spreadsheet.
     • Make sure that statutory holidays for Canada and US are accounted for in the release dates.
   • For the next release, create the following team meetings. If they conflict with existing meetings, check with meeting owner to reschedule or reschedule the release meeting
     • PM Release Update meeting on T-15 at 7:30am San Francisco time
     • Major Feature Complete Meeting on T-12 at 10:00am San Francisco time
     • Judgment Day Meeting on T-10 at 10:00am San Francisco time
     • Code Complete Meeting on T-8 at 10:00am San Francisco time
     • Team-wide Triage each weekday starting at T-8 and ending at T-2 at 9:00am San Francisco time, with optional attendance
     • Release Update each weekday starting at T-7 and ending at T-2 at 10:00am San Francisco time
   • Add “Release Retrospective” item to next team meeting, asking each core team member to give a letter grade (and brief explanation) for:
     • Release Quality
     • Release Process
     • Testing Process
   • Close the release in Jira (releases page)
     • If there are many unresolved tickets in the current release, ask the release manager to review the ticket queue
     • Otherwise, release the fix version (Actions > [...] > Release)
   • Prepare tickets for the next release, with a corresponding vX.X prefix
     • Creating final release candidate
     • Test Gitlab Omnibus RC install of Mattermost
     • Test upgrade to latest release based on upgrade guide
     • Test upgrade from Team Edition to Enterprise Edition
     • RC Build Testing for core team
     • Upgrade gitlab.mattermost.com to RC1
     • Push final build to gitlab.mattermost.com
     • Cut build and set up RC1 servers, including a note to check for all XXX items
   • Update mattermost-docker version
   • Contact owners of community installers or submit PRs to update install version number
     • For Puppet, Heroku and Ansible Playbook, post to Installers and Images channel announcing the new release. See example.
     • For Chef Cookbook, open a new issue to announce the new release. See example.
     • For Yunohost, open a new pull request to update the version. See example.
     • For OpenShift, open a new pull request to update the version. See example.
3. Docs:
   • Create a new branch on docs for the next release - vX.X-documentation
     • Submit a PR for changelog against the vX.X-documentation branch and add a Work in Progress label for it
     • Submit a PR to change version number in docs/source/conf.py against the vX.X-documentation branch
4. Build
   • Put CI servers and translation server back onto master
   • Update ci-linux-mysql-prev to the final release version
5. Dev:
   • Delete RCs after final version is shipped
   • Check if any libraries need to be updated for the next release, and if so bring up in weekly team meeting
   • Test the GitLab RC containing the Mattermost final bits
   • Confirm gitlab.mattermost.com is updated to final build
   • Merge changes made to release branch into master
6. Marketing:
   • Confirm marketing has been posted (animated GIFs, screenshots, mail announcement, tweets, blog posts)
   • Update @mattermosthq Twitter profile with the next release date
   • Prepare retweet of GitLab release tweet (see example here)

J. (T-plus 5 working days) Release Updates¶

1. Release Manager:
   • Post this checklist in Release channel
   • Verify all items in the last posted release checklist are complete
2. Logistics:
   • Confirm the Security Researchers list on the Responsible Disclosure Policy is up to date
   • Review “Community Installers” and update version numbers if there are any discrepancies https://www.mattermost.org/installation/ (move this to ops eventually)
3. Leads:
   • Update company roadmap at mattermost.com
4. Build:
   • Put pre-release back on master

K. (T-plus 10 working days) Update Mattermost Security Page¶

1. PM:
   • Post Mattermost Security Updates after reviewing with security lead
     • If a dot release is shipping with security fixes, do not post new details until T-plus 10 working days from the dot release ship date
   • Update Security Issues spreadsheet with issue number from posted update (e.g. v3.2.0.1)