Release Process¶

A. (Code complete date of previous release) Beginning of release¶

Pre-work for the current release begins at the code complete date of the previous release. See “Code Complete” section below for details.

B. (T-minus 15 working days) Cut-off for submitting major features¶

No pull requests for major features should be submitted to the current release after this date (except if release manager decides to add “release-exception” label to Jira ticket).

1. Logistics:
   • Post this checklist in Release channel
2. PM:
   • Prioritize reviewing major features, ensuring any bugs and UX issues get fixed
   • Check that all major features are behind a feature flag
3. Dev:
   • Prioritize reviewing, updating, and merging of pull requests for major features
4. Marketing:
   • Confirm each Enterprise feature is in the correct pricing SKU, if not alert the release manager

C. (T-minus 12 working days) Cut-off for merging major features¶

No pull requests for major features should be merged to the current release after this date (except if release manager decides to add “release-exception” label to Jira ticket).

1. Logistics:
   • Post this checklist in Release channel
   • Begin posting Zero Bug Balance query daily
   • Verify all items in the last posted release checklist are complete, if not alert the release manager
2. PM:
   • PM area owners complete draft of Changelog in a WIP PR with updates for highlights, feature additions, known issues, compatibility updates for config.json, database changes, API changes (search #api-proposal and confirm with Dev) and WebSocket event changes; see example
   • PM feature owners post draft section for the blog post in the Marketing Channel (including screenshots and a hashtag #mattermostXX where XX is the version number, see example thread) and queue a tweet
   • Update Upgrade Guide for any steps needed to upgrade to new version
   • Submit NOTICE.txt PR for any new libraries added from dev, if not added already. The following two files contain a list of dependancies:
     • https://github.com/mattermost/platform/blob/master/webapp/package.json
     • https://github.com/mattermost/platform/blob/master/glide.yaml
   • Coordinate testing:
     • Queue an item for UX meeting to discuss worst UX bug
     • Receive testing sign-off from feature area owners (i.e. PM/Dev either signs-off that their area is well tested, or flags potential quality issues that may exist)
     • Prioritize updating the RC Testing Spreadsheet to cover any changes or new features, and confirm that known issues are listed in the relevant tests
     • Assign each area of the release testing spreadsheet to a team member and ensure core team has access permissions
     • Post in Reception alerting community of upcoming release and to ask about top issues on master (See example)
3. (Team) Major Feature Complete Meeting (10:15am PST):
   • Discuss worst bug currently on master
   • Review status of last feature PRs to be merged
4. Dev:
   • Prioritize reviewing, updating, and merging of pull requests for current release until there are no more tickets in the pull request queue marked for the current release

D. (T-minus 10 working days) Judgment Day¶

Day when leads and PM area owners decide which major features are included in the release, and which are postponed.

1. Logistics:
   • Post this checklist in Release channel
   • Verify all items in the last posted release checklist are complete, if not alert the release manager
   • Confirm date of marketing announcement for the release and update release channel header if needed
2. Leads:
   • Finalize roadmap for next release, and identify planned marketing bullet points
3. (Team) Judgment Day Meeting (10:15am PST):
   • Meet to discuss release update and finalize which major features will be in or out for the release
4. PM:
   • Based on discussion, create tickets for features that need to be turned on or off for the release
   • Backlog is reviewed and tickets that won’t make it are moved to next release
   • Post a link to Release Discussion room for query of remaining tickets in this release
   • Update Changelog PR based on what’s in/out of the release
   • Create meta issue for release in GitHub (see example)
5. Marketing:
   • Begins to draft blog post, tweet, and email for the release announcement

E. (T-minus 8 working days) Code Complete and Release Candidate Cut¶

Stablization period begins when all features for release have been committed. During this period, only bugs can be committed to the release branch. Non-bug pull requests are tagged for next version.

Exceptions can be made by the release manager setting priority to “Highest” and adding a “release-exception” label to the Jira ticket. This will add the ticket to the hotfix list for release candidate.

1. Logistics:
   • Post this checklist in Release channel
   • Mail out mugs to any new contributors
   • Update Team page with new contributors
   • Verify all items in the last posted release checklist are complete, if not alert the release manager
   • Confirm all PRs merged into the current release have been tested
2. Dev:
   • Prioritize reviewing, updating, and merging of pull requests for current release until there are no more tickets in the pull request queue marked for the current release
3. PM:
   • Review all Severity 1 bugs (data loss or security) to consider for adding to Hotfix list
   • Gather a list of contributors across all public Mattermost GitHub repos, and add to the Changelog draft
   • Zapier manager adds new GitHub automation zap for PR tracking spreadsheet for next release.
   • Update documentation:
     • Submit Changelog PR
     • Submit Changelog PR for /ios and /android repositories
     • Submit all doc PRs for review
     • Confirm changes to config.json in compatibility section of Changelog are written back to settings documentation
     • Prioritize any developer documentation tickets
     • Draft Mattermost Security Updates, but do not post until seven days after official release
4. (Team) Code Complete Meeting (10:15am PST):
   • (PM) Leads team review of Changelog
   • (Logistics) Walk through each unfinished item of this checklist
   • (Dev) Last check of tickets that need to be merged before RC1
5. Build:
   • Review all TODO notes
   • Master is tagged and branched and “Release Candidate 1″ is cut (e.g. 3.5.0-RC1) according to the Release Candidate Checklist in mattermost/process
   • After branching, the database version in sql_upgrade.go on master is set to the next scheduled release version (e.g. 3.6.0)
   • CI servers are updated to the release branch
   • Translation server is locked to the release branch
   • Directory structure is reviewed and large changes posted to the Release Discussion channel
6. PM:
   • Merge changelog PR after team review is complete, and update the GitHub meta issue to include a link to the changelog on the documention branch
   • Tweet announcement that RC1 is ready (see example)

F. (T-minus 7 working days) Release Candidate Testing¶

1. Logistics:
   • Post this checklist in Release channel
   • Verify all items in the last posted release checklist are complete, if not alert the release manager
2. Dev:
   • Test upgrade from previous version to current version, following the Upgrade Guide with database upgrades on both MySQL and Postgres
   • Test upgrade from Team Edition to Enterprise edition based on the Upgrade Guide
   • Review any changes made to install guides, and test if necessary
3. PM:
   • Update Release Discussion header with links to RC instances and testing spreadsheet
   • Post release testing instructions to Release Discussion channel (example)
   • Post to Reception directing community testers to the Release Discussion channel for details
   • Post “Known Issues” to Release Discussion channel
4. Team:
   • Test assigned areas of the Release Candidate Testing Spreadsheet and file any bugs found in Jira
   • Post a link to any “Blocking” issue that may need a hotfix to the RC in the Release room, with the #blocking tag. If the issue is security related or contains confidential information, post the link in the Confidential Bugs private group. Blocking issues are considered to be security issues, data loss issues, and issues that break core functionality or significantly impact aesthetics.
   • Daily triage of hotfix candidates and decide on whether and when to cut next RC or final
5. PM:
   • Queue an item for Release Update meeting to discuss worst bug
   • Update the meta issue:
     • Post comments to the meta issue with approved fixes for the next RCs
     • Download links and testing server links to the RCs
   • Post screenshot and links to final tickets for next RC to the Release Discussion room
   • Update Changelog “Known Issues” section with any significant issues that were found and not fixed for the final release
6. Dev:
   • PRs for hotfixes made to release branch
   • Review PRs made from release branch and merge changes into both the release branch and master
7. Logistics:
   • Test RC fixes as they come in on CI servers
8. Build:
   • Verify with Release Manager before cutting any new RCs (approved fixes should be merged)
   • Push next RC to acceptance and announce in Town Square with new RC link
9. PM:
   • Test the new RC to verify fixes merged to the release branch work
   • Post in Release Channel after testing

G. (T-minus 5 working days) Release Candidate Testing Finished¶

1. Logistics:
   • Post this checklist in Release channel
   • Verify all items in the last posted release checklist are complete, if not alert the release manager
2. Team:
   • Finish assigned areas of the Release Candidate Testing Spreadsheet
   • Continue triaging hotfix candidates and decide on whether and when to cut next RC or final
   • If no blocking issues are found, PM, Dev and Ops signs off on the release
3. PM:
   • Check that known issues section of Changelog is updated
   • Check that the contributors section of Changelog is updated (including contributors from all repos)
4. Marketing:
   • Finish draft of blog post for mattermost.org and send for marketing lead to review
     • Upgrade should be recommended if there are security fixes in this version
   • Finish drafts of all art work (screenshots, GIFs and twitter banners) used for the blog post and send to marketing lead for review

H. (T-minus 2 working days) Release Build Cut¶

The final release is cut. If an urgent and important issue needs to be addressed between major releases, a bug fix release (e.g. 1.1.1) may be created.

1. Logistics:
   • Post this checklist in Release channel
   • Verify all items in the last posted release checklist are complete, if not alert the release manager
2. Build:
   • Tags a new release (e.g. 1.1.0) and runs an official build which should be essentially identical to the last RC
   • Posts SHA key and md5 sum of the final build to release channel
3. PM:
   • Post in Release Discussion with links to the EE and Team Edition bits
   • Update the Mattermost server download page with the links to the EE and TE bits
     • Test the download links before and after updating the page
   • Add the download links, SHA key and md5 sum to upgrade guide
   • Contact owners of community installers or submit PRs to update install version number
   • Close GitHub meta ticket for the release
   • Finalize docs
     • If reviews are not complete, hold a 30 minute doc review meeting with PMs and anyone else who has changed or reviewed docs this release and wants to join
     • Merge the docs release branch to master and verify all changes on docs.mattermost.com once the build is up
     • Submit a correction PR for any incorrect formatting or other errors missed during the intitial review
   • Update MVP page with the most valuable contributor of the release
   • Submit GitLab MR to take next Mattermost version in the Omnibus (see example):
     • Include changes to Mattermost version number (default_version) and md5 sum of the final TE build (source md5) in config/software/mattermost.rb
     • Include a summary of updates in Team Edition that are relevant to GitLab
   • Update Docker preview image to latest version
   • Submit PR to update /mattermost-docker image to latest release
   • Check Security Issues spreadsheet and confirm disclosure text
   • Confirm link to security updates appears in blog post if there are security updates in this release
4. Marketing:
   • Finish draft of animated GIF (for Twitter announcement, MailChimp and blog post) made up of top announcements
   • Finish draft of MailChimp email blast and Twitter announcement and send for marketing lead to review. Once reviewed, schedule for 08:00 PST on the date of marketing announcement
   • Finalize blog post for mattermost.com and set timer for 08:00 PST on the day of release
   • Turn on CrazyEgg for blog post page
   • Find www-gitlab-com merge request for latest GitLab release blog post and make request for adding GitLab Mattermost update (see example request, example update). Post to Release Discussion channel with link to request.
   • Update feature list on mattermost.com with relevant new features

If a bug fix release is required, run through the following steps:

1. PM:
   • Post links to approved tickets for next dot release RC to the Release Discussion channel
   • Make a post in Town Square announcing the dot release. See example
   • Update the GitHub meta issue:
     • Change the title to “Mattermost vx.x.x - RCx”, where vx.x.x is the version number of the dot release
     • Post a comment to the meta issue with approved fixes for the next RC of the dot release
     • Post download links and testing server links for the next RC when it’s cut
   • Update Changelog:
     • Start a WIP PR for the dot release changelog and commit updates as new issues are fixed on the dot release RCs
     • Update “Known Issues” section with any significant issues that were found during RC testing and not fixed for the dot release
   • Push RC versions to acceptance and announce in Town Square with new RC link as they are cut
   • Test the new RC to verify fixes merged to the release branch work. Post in Release Discussion channel after testing
2. Dev:
   • PRs for hotfixes are made to release branch
   • Review PRs made from release branch and merge changes into both the release branch and master
3. Build:
   • Verify with Release Manager before cutting any new dot release RCs (approved fixes should be merged)
   • Push dot release RC’s to CI servers, pre-release and GitLab Mattermost.
4. Logistics:
   • Test RC fixes as they come in on CI servers

Once final dot release build is ready to cut:

1. Build:   - Tag a new release (e.g. 1.1.1) and run an official build   - Update CI servers, pre-release and GitLab Mattermost to the final version
2. PM
   • Update [Mattermost server download page](the https://mattermost.org/download) with the links to the EE and TE bits
     • Test the download links before and after updating the page
   • Update Mattermost pricing page if anything has changed
   • Add the download links to http://docs.mattermost.com/administration/upgrade.html#version-archive
   • Merge the Changelog PR with notes on patch releases (see example entry)
   • Update the version archive in the upgrade guide
   • Sumbit GitLab MR to update the version number and MD5 hash to the dot release version. See example
     • Test the upgrade once the MR is merged and the package is released to the GitLab package server
3. Logistics:
   • Verifies each of the issues in the patch release are fixed

I. (T-minus 0 working days) Release Day¶

1. Logistics:
   • Post this checklist in Release channel
   • Post key dates for the next release in the header of the Release Discussion channel and remove links to RC candidates and testing spreadsheet
   • Verify all items in the last posted release checklist are complete, if not alert the release manager
   • For the next release, create the following team meetings:
     • Major Feature Complete Meeting on T-12
     • Judgment Day Meeting on T-10
     • Code Complete Meeting on T-8
     • Team-wide Triage each weekday starting at T-8 and ending at T-2, with optional attendance
     • Release Update each weekday starting at T-7 and ending at T-2
   • For the next release, create PM Release Update meeting on T-15 one hour before UX meeting
   • Add “Release Retrospective” item to next team meeting in place of Kaizen/User Issues, asking each core team member to give a letter grade (and brief explanation) for:
     • Release Quality
     • Release Process
     • Testing Process
2. PM:
   • Close the release in Jira
   • Prepare tickets for the next release, with a corresponding vX.X prefix
     • Performance improvement ticket
     • Update testing spreadsheet
     • Creating final release candidate
     • Test Gitlab Omnibus RC install of Mattermost
     • Push to private long-running feature branches
     • Test upgrade to latest release based on upgrade guide
     • Test upgrade from Team Edition to Enterprise Edition
     • RC Build Testing for core team
     • Upgrade gitlab.mattermost.com to RC1
     • Push final build to gitlab.mattermost.com
     • Cut build and set up RC1 servers, including a note to check for all XXX items
   • Create PR tracking spreadsheet for next release
     • Populate with any PR’s that have already went in for the next release
   • Create a new branch on docs for the next release - vX.X-documentation
     • Submit a PR for changelog against the vX.X-documentation branch and add a Work in Progress label for it
     • Submit a PR to change version number in docs/source/conf.py against the vX.X-documentation branch
3. Build
   • Put CI servers and translation server back onto master
4. Dev:
   • Delete RCs after final version is shipped
   • Check if any libraries need to be updated for the next release, and if so bring up in weekly team meeting
   • Test the GitLab RC containing the Mattermost final bits
   • Confirm gitlab.mattermost.com is updated to final build
5. Marketing:
   • Confirm marketing has been posted (animated GIFs, screenshots, mail announcement, tweets, blog posts)
   • Prepare retweet of GitLab release tweet (see example here)

J. (T-plus 5 working days) Release Updates¶

1. Logistics:
   • Post this checklist in Release channel
   • Verify all items in the last posted release checklist are complete, if not alert the release manager
2. Leads:
   • Update company roadmap at mattermost.com
   • Post Mattermost Security Updates after reviewing with security lead.
     • If a dot release is shipping with security fixes, do not post new details until T-plus 5 working days from the dot release ship date.
3. PM:
   • Update Security Issues spreadsheet with issue number from posted update (e.g. v3.2.0.1)
   • Confirm the Security Researchers list on the Responsible Disclosure Policy is up to date
   • Review “Community Installers” and update version numbers if there are any discrepencies https://www.mattermost.org/installation/ (move this to ops eventually)
4. Build:
   • Put pre-release back on master