After a version of MySQL Server has been declared GA, Oracle releases regular maintenance updates: the 5.6 GA was 5.6.10 and we are now at 5.6.17.
A lot of work goes into the maintenance releases. The purpose of this blog post is to show you a little bit of what happens on behind the curtains in the Oracle sweatshop that actually produces a maintenance release and to give you an insight into our thinking around testing, quality and bug fixing.
What goes into a maintenance release?
The short answer is “bug fixes”. These fall roughly into two classes,
- The majority of bugs are chosen by Support for their relevance to customers and community and worked on by the Sustaining Team, which is dedicated to fixing the bugs on Support’s list. Yes, Support choose the bugs. That may come as a surprise, but makes sense since they feel the pain points of customers and of community through their work on bug verifying at http://bugs.mysql.com
- In addition, some bugs are requested for inclusion by the developers themselves. These are subject to additional scrutiny by Support and the patch is vetted by them before the push can happen.
- We also extend our test suite on a regular basis to make sure that bugs don’t reappear.
What doesn’t go into a maintenance release?
- Risky bugfixes. We do our best to minimize risk.
- New features. Since we publish milestone releases of 5.7, our hopefully cool new features can be tested and tried out before they come to a GA release.
We sometimes get requests to backport features in a maintenance release, but we don’t do it: most customers and Linux distros don’t want backports, they introduce too much risk and also constitute an extra burden on the engineering team, that has to re-implement the feature in an earlier codebase.
What kind of testing is done upfront?
We do continuous intergration testing of all our release branches (5.5, 5.6 and trunk). This consists of:
- Per-push testing: a subset of the MTR test suite on a selected set of platforms.
- Daily testing: a bigger subset of the MTR test suite on a narrow set of platforms.
- Weekly testing: the entire MTR test suite on all release platforms.
Random Query Generator (RQG) tests are also run on per-push, daily and weekly.
In addition, we run daily performance regression testing, stress/load testing, replication testing, as well as specialized builds like Werror, Valgrind and AddressSanitizer. On a weekly basis we run upgrade-downgrade testing and additional performance tests.
An important aspect of the work on continuous testing is the human factor. The QA team and the release managers follow up on errors introduced by developers and pushes that break tests are not allowed to remain for long without a fix.
Are there no regressions in a maintenance release, then?
Alas, no. We are not perfect, and as a Release Manager I would like to see us extend our test coverage so that regressions are even rarer.
How do we produce a maintenance release?
About a month before the projected release date, the release manager cuts off the maintenance release and creates the release branch.
Packages get built automatically in our combined build/test system and they are handed over for integration testing:
- Testing with other products (connectors, MEB, Workbench…)
- Upgrade-downgrade testing
- Performance testing (sysbench, DBT2)
- Load-stress testing with and without replication (using mysql-stress-test.pl)
- Large volume testing
- Package validation/installation/upgrade (on a wide range of platforms and packages)
- Windows installer testing
From the point where integration testing starts, no more fixes except showstoppers into the release. If a showstopper is found, we respin the release and retest it.
Finally, when everything (including the release notes) is completed and we are convinced that the quality is acceptable, we publish packages to customers and community (on http://dev.mysql.com).
In the meantime, of course, the developers have been busy for a month writing fixes for the next maintenance release. And so the cycle continues…