Around nine months ago, we launched the MySQL yum repositories for Enterprise Linux and Fedora, followed by apt repos for Debian and Ubuntu back in May. We’re extremely happy that the repos have proved to be a big hit with the community: the monthly number of downloads hit 100K recently and it is still growing briskly. Indeed, the repos are now our most important community distribution channel for the MySQL product family on these platforms.

Right from the outset, it was clear that a thorough QA regime would be very important. The yum and apt projects involved a complete rearchitecting of our rpm and deb packaging, we expected the repos to become very popular, and not least: a key aspect of the repo distribution model is immediacy. If you release broken packages through your repos, lots of users are going to be hit by problems very quickly. So we made very sure that QA was placed front and center of the repo projects.

In this posting, we will describe the way we currently test MySQL packages that go into the repos. We will focus on the newest MySQL releases (5.6.20 and 5.5.39, both coming very shortly), and the most recently added platform, namely Enterprise Linux 7. The principles and overall approach are very similar between the yum and apt repos, while the specific test scope and types of install scenarios we test vary between different MySQL versions and OS flavors.

Objectives and strategy

Boiling down the motivation for our repo QA efforts to a few bullet points, this is what we get:

• Provide users with overall high quality packages
• MySQL is a core system component: our packages must not break the system in any way
• Make sure we completely replace the distro native “mysql” without breaking compatibility
• Make sure that distro native third party applications that depend on MySQL continue to work properly
• Build and maintain user confidence in the official MySQL repos

The repository testing takes as its input a set of packages that comes from our continuous integration build and test system. The main focus is on testing the packaging, as opposed to the package contents; most of the testing of the product itself happens in a separate, parallel QA process. The entire process is applied for each and every maintenance release of MySQL.

The overall scope of testing covers numerous scenarios involving installation and upgrade, including verification of the resulting installs, and testing to make sure that third party applications that depend on MySQL work as expected. This is also the point where our chosen example becomes particularly interesting: as we have highlighted previously, Red Hat chose to ship an older, forked version of MySQL natively in EL7, and this meant we had to add some new testing scenarios in this latest round in order to make sure we got things right.

Scope and approach

Here is an overview of the different categories of repository package testing we do and the scenarios we cover, specifically for MySQL on EL7:

Installation testing

• We install the following mysql community packages: mysql-community-bench, mysql-community-client, mysql-community-common, mysql-community-devel, mysql-community-embedded, mysql-community-embedded-devel, mysql-community-libs, mysql-community-server, mysql-community-test packages, all at once.
• After installation, we validate that we have a running service, we connect to it, we verify plugins, do version checks and check mysql system tables, information_schema tables etc. in order to ensure that we have a valid install.
• We test multilib support to ensure that users can run 32 bit applications with 64 bit MySQL packages.
• We do individual package installation in order to ensure that the correct dependencies get installed automatically. E.g. the mysql-community-server package depends on mysql-community-client, mysql-community-common and mysql-community-libs.

Upgrade testing

As part of upgrade testing, we cover a number of different upgrade flows: upgrades between different server releases from the MySQL repo; upgrade from distro native packages to MySQL repo packages; upgrade from older MySQL repo packages to new ones, and finally that none of these three upgrade types break multilib support.

Let us take a more detailed look at each of these flows:

Upgrades between MySQL release series in the MySQL repo

As we have multiple MySQL major release series (5.5, 5.6 and 5.7) in the repository, we cover relevant upgrade flows between these releases in order to ensure that all the packages we deliver are compatible with each other:

mysql55-community → mysql56-community
mysql56-community → mysql57-community
mysql55-community → mysql57-community

Upgrade from distro native to MySQL repo with different MySQL versions

Here we test upgrade or replacement of native distro packages with real MySQL packages in order to ensure that nothing breaks. Example: EL7 comes with MariaDB 5.5.35 by default, we want to offer users a move to the real MySQL from MariaDB without any compatibility issues, so we thoroughly test upgrades from the distro native MariaDB to the different MySQL versions that we offer from the MySQL repo:

mariadb5.5.35 → mysql55-community
mariadb5.5.35 → mysql56-community

We verify and validate the upgrade process with various checks like monitoring information_schema and mysql database tables, installing plugins, verifying already installed plugins, defined users, running mysql_secure_installation, doing storage engine and version checks etc.

Upgrade from “old style” MySQL rpms to repo rpms with different MySQL versions

As we mentioned above, the repository rpms represent a deep rearchitecting of our “old style” rpms. We continue to support the old style packages, but we also give users a choice to “upgrade” to the repo style ones and thus simplify future upgrades and administration. To make sure this works seamlessly, we test all relevant version combinations of old style rpms and yum repo rpms (note: for EL7 there are no legacy old style packages, so this scenario does not apply there)

Upgrade from older to new MySQL packages in the MySQL repo

It is critically important that we don’t break the chain of upgradability between what we have delivered already and what we are delivering. This means that we need to test upgrades from older MySQL versions in the repo to new versions. For MySQL 5.6.20 and 5.5.39, this means:

mysql55-community(e.g. 5.5.38) → mysql55-community(e.g. 5.5.39)
mysql55-community(e.g. 5.5.38) → mysql56-community(e.g. 5.6.20)
mysql56-community(e.g. 5.6.19) → mysql56-community(e.g. 5.6.20)

(The precise combinations above will of course vary with the actual server versions in the repository at any given time)

Upgrade between 32 bit and 64 bit packages

We offer multilib support in our repo packages, meaning that users can install 32 bit and 64 bit packages side by side on a 64 bit system. In this testing we cover installation of individual 32 bit packages and verify that those still exist on the system after installing all 64 bit packages.

Third party application dependency testing

Most distros come with many third party applications which internally depend on some MySQL-like client, libraries or server being available on the system. Our testing in this area aims to ensure that dependent applications continue to work correctly after a user installs the real MySQL.

A good test strategy often involves reducing the number of combinations to be tested to a workable set, so we have been identifying three different dependency patterns and one  or more third party MySQL-dependent applications for each of these. We then check that these applications are unaffected by the installation.

These are the dependency patterns and their representative applications:

• applications that depend on client and libs: dpm-name-server-mysql
• applications that depend on libs: mysql-connector-odbc,mysql++
• applications that depend on devel and libs: mysql++-devel

Overview of test scenarios and combinations

Here is the detailed test specification matrix for 5.6.20 repository package testing:

So, what is it good for?

For an activity such as this, a key metric will always be cost versus benefit. The QA regime we describe in this posting is resource consuming, both in terms of initial investment and ongoing test development and execution for each release, and careful management of test scope and intense automation is important in order to keep especially the recurring execution costs down.

So, has it been worth it so far? The answer is a resounding “yes”, and we have data points to back that up. First and foremost, in spite of well over half a million downloads and a steady stream of new releases, the number of community reported issues has been very low. Of course, that doesn’t mean that we haven’t seen any issues, just that we tend to catch them before release. Here are a few bugs encountered during the 5.6.20 / 5.5.39 test cycle. Please note that this was the first test cycle run on the final GA version of EL7, so some issues were indeed expected:

• MySQL bug #73073: update to community-server fails if mariadb-galera-server already installed
• MySQL bug #73113: yum update fails when community repo enabled and mariadb-embedded-devel installed
• MySQL bug #73194: 5.5 packages failed to install after installing mysql-bench from native repo

What’s next?

As the distro landscape develops and changes, our packages and our testing regime need to change as well. An example is the evolution of EL7: Red Hat made a number of big MySQL related packaging changes beginning with the EL7 beta version they released back in December 2013, to the release candidate in April and finally to the GA that appeared some weeks ago. For each of these milestones, we adjusted and expanded our testing in order to make sure that our packages worked as they should.

Going forward, we will of course continue to adapt to any changes. We will also continue to pursue efficiency by investing in more automation. In particular, we will add more automatic validation points at different stages of the install and upgrade process, we will enhance the system for test triggering and we will hook everything into our automated test reporting system. This should all hopefully allow us to spend even more time to ensure that we follow the distros closely and in general that we test the right things as thoroughly as possible.

And finally, we’d be delighted to hear your comments, questions and thoughts on all this. Ping us in the comments section!

Authors

This article was jointly authored by Ramana Yeruva and Yngve Svendsen from the MySQL Engineering team. We would also like to thank the MySQL Repository Project team for helpful suggestions and for valuable QA of the article contents.