In my Percona live keynote “Driving MySQL Innovation” on Tuesday I talked about the quality problems we had with 5.0 and 5.1 and the switch to the new Milestone Release model. I explained how this new development process allowed us to deliver both 5.5 and 5.6, on time, including innovative features, and with very high quality.

The description of the Milestone Release model has been available here, for almost two years. So, this is not new but it has now been in operation for 3.5 years and it can be worth looking back and summarize.

Our first MySQL Server Milestone 5.4.3 was released on October 9th, 2009 and was then followed up by 5.5.2 on Feb. 26, 2010. Initially there was a time of trying out and fine tuning the model before it found its current form around 5.5 GA. The stabilized model has been followed consistently throughout the 5.6 development cycle.   Our latest milestone release (5.7.1) was released this week and was first 5.7 milestone release. In total we have now done 11 milestones in 5.5, 5.6 and 5.7 together. In short we refer to these releases as DMR’s (Development Milestone Releases).

Why did we create the milestone release model? The milestone release model came into existence because our previous models failed badly. The problems were many: First, we were not able to ship on time (not even close), and in some cases not at all (maybe a blog for another day, “The MySQL releases that never shipped”). Second, quality was bad or unknown. Third, developers were frustrated. All these issues were fixed by the process change.

So what is the essence of the Milestone Release model? The heart of the milestone release model can be explained by three words: “mysql-trunk is sacred”. The key is to keep soundness and quality in our main development branch at all time. Two secrets here: One is to have massive regression testing on trunk at all times and to immediately fix anything that is broken. Second, to do massive QA on all new features before they are allowed into trunk. New features are developed in independent feature trees while frequently pulling in changes from trunk to keep it in sync. When feature development is done it goes to QA and will stay in QA until all bugs found by QA are fixed and a sign-off is given by QA. At this point, *after* QA is completed, the feature will be merged to trunk.

To summarize our experiences: With the Milestone Release model our historical problems are gone: A problematic feature will be stopped before it comes to trunk and will thus not pollute trunk. We are able to ship trunk at any time since quality is maintained on a daily basis. Developers are no longer blocked by a bad trunk, their feature goes in when it is ready.

This process is working out great internally. The process allows our enginners to focus on their work. However, the biggest benefit is to our customers and users as they are getting predictability, timeliness and quality.