MySQL :: MySQL Package Verification: Making sure we always ship correct, complete and installable packages

What is MySQL Package Verification?

Package verification (Pkgver for short) refers to black box testing of MySQL packages across all supported platforms and across different MySQL versions. In Pkgver, packages are tested in order to ensure that the basic user experience is as it should be, focusing on installation, initial startup and rudimentary functionality. When tests are written, we put ourselves into the role of the user and write tests accordingly. This involves checks for sanity and correctness of contents, installability and smoke testing. The MySQL user manual constitutes the underlying reference guide to write the tests, since this manual is the one that most users will follow.

Background

MySQL has an ever growing test suite that performs regression tests on the MySQL code, testing old and new functionality. Our dedicated Quality Assurance (QA) teams do additional in-depth testing with main focus on verifying functional and non-functional (e.g. performance, high volume, stress, upgrade/downgrade etc.) characteristics of the product once it is up and running. What is needed in addition is testing to ensure that the initial and basic steps of getting up and running can be accomplished with the product packages we ship. And that is what Package verification does.

Objective and Scope

The basic objective of Pkgver is to make sure that the user can get MySQL up and running.

The Pkgver suite runs on MySQL versions 5.5, 5.6, 5.7, and what will eventually become 5.8, and on Cluster 7.2, 7.3 and 7.4 on several OS flavors and versions. It runs on release branches in our continuous integration test system, on weekly-trunk (where development builds happen on weekly basis) and on certain Worklog branches (where feature development happens, and where packaging and basic functionality may be involved.)

Approach

The Pkgver test suite attempts to install the relevant MySQL package, start the server, run some sanity and functionality tests, then uninstall and report if there are any issues.

Pkgver testing consists of the following phases:

The Pkgver Testing Environments

Pkgver testing is done in chroot environments (MySQL 5.5 and 5.6) or in dedicated Virtualbox VMs (5.7 onwards).

Chroots

This involves creation and maintenance of chroot images that mimic a clean system as closely as possible, except for the addition of the minimal set of files/libraries/executables required for doing the testing, all tied together with some Python scripts.

During a Pkgver test run, the chroot image corresponding to the package/platform in consideration is extracted, the Pkgver process then enters the chroot jail, executes the tests and reports success/failure.

During the course of automating the tests using chroot environments, we realized that this approach has some drawbacks:

As the product evolves, so do its dependencies, and the chroot image needs to keep up. So we got into a constant race to keep the chroot environments updated. Chroot maintenance is cumbersome; even keeping OS patch levels reasonable requires significant work, so our Pkgver work over time became more about fixing chroots than adding or automating new tests.
Often, if the environment is found to be missing something in the chroot image of one particular platform, the same is the case for the images of other platforms as well. So the effort required by the item above often gets a sizable multiplier due to the wide range of platforms we support.
It is also a problem that a chroot is only a very weakly isolated system. It shares e.g. ports and other facilities with the underlying host OS; this causes issues with Pkgver testing from time to time, and those issues can be quite hard to track down.
Our introduction of systemd support caused issues: Service commands available with el7 and related newer platforms do not get executed in chroot enviroments (https://bugzilla.redhat.com/show_bug.cgi?id=1110675). Inside a chroot jail, you simply get the following response to a service mysqld start:
```
$ service mysqld start
Redirecting to /bin/systemctl start mysqld.service
Running in chroot, ignoring request.
```

Virtual machines on the other hand give you much stronger isolation and the option of using regular system maintenance tools to keep images up to date. Hence the decision to replace chroots with Virtualbox VMs as our testing environment, and we are now using VMs to test 5.7+ packages.

Virtual machines

For MySQL 5.7 and above VMs now form the basic infrastructure for Package verification. For each Pkgver run, we create a new VM instance, start the VM, run Pkgver on it, get the results and destroy it. By creating a new VM instance for every Pkgver run, we ensure that we have a clean and pristine environment during testing.

Overview of Test Scenarios

The tests are performed in phases as follows:

Phase I – INSTALL: Install MySQL package

Test 1:

For .deb, run `dpkg -i <pkg-name>`
For .tar.gz, unpack the tarball, symlink to ‘mysql’, and run mysql_install_db to initialize database(for MySQL >= 5.7.6, run mysqld –initialize)
For RPM, it’s rpm -Uvh(in case of new-style RPMs, it is `yum localinstall`)
For Solaris .pkg file, it’s pkgadd,
For .msi, do a batch install (using msiexec command line)
For MySQL version >= 5.6, this test also grabs the random password either from .mysql_secret file or mysqld error log(in case of MySQL version >= 5.7.7) and sets a new password.

Phase II – post-INSTALL: Perform post-installation checks

Test 2: Find .la Files

Checks if there are any .la files present in the package and reports error if they are present.

Test 3: Check Standard Plugins in Package

Checks that the standard plugins are present in the package.

Test 4: Check package dependencies

Ensures that the package does not have unwanted dependencies. This test is implemented only for RPMs. Specifically, the package should not depend on Intel icc runtime libraries like libimf, libipr, etc. (https://bugs.mysql.com/bug.php?id=16662)

Test 5: Contains ChangeLog

Ensures that the ChangeLog exists and has the correct content.

Test 6: Contains errmsg files

Ensures that errmsg file (errmsg.sys) exists and is in the correct location.

Test 7: Contains include directory

Ensures that the include directory exists and contains mysql.h.

Test 8: Contains libraries

Ensures that relevant libraries exist (libmysqlclient.a in Unix and debug/mysqlclient.lib in Windows) and are in the correct location.

Test 9: Contains debug binaries

Ensures that the mysqld-debug binary exists.

Test 10: mysqld has debugging symbols

Ensure that mysqld and mysqld-debug are not stripped.

Test 11: Check mandatory common plugins in 5.6+ packages

Checks that the mandatory plugin validate_password.so/.dll is present in both commercial and community 5.6+ packages.

Test 12: Check permissions of files/binaries/libraries in 5.7.6+ packages on Unix platforms

Verifies that the permissions of files/binaries/libraries are correct as per WL5608/WL8129. This test is not applicable to Windows.

Test 13: Check permissions and ownership of the data directory and its contents in 5.7.6+ packages

Verifies that the data directory has mysql:mysql ownership and permissions as per WL5608 (750 for dir and 640 for files).
This test is applicable for new-style RPMs and DEBs only.
NB: RPMs have data dir permissions of 755.

Test 14: Check package signature

This test is applicable to OS X DMG only.
Verifies that the DMG has the correct Oracle Certificate.

Phase III – START SERVER

Test 15: Perform server start

Phase IV – post-START SERVER: Performs post-start checks

Test 16: Smoke Test – Nuke & Restart

Kills the mysqld process and verifies that the mysqld_safe process triggers a new mysqld process. Else errors out.

Test 17: Smoke Test – Version Variables

Displays version variables.

Test 18: Smoke Test – Check HELP Contents

An arbitrary test of the server’s HELP contents – checks if HELP SELECT mentions GROUP BY.

Test 19: Smoke Test – Simple MySQL Test

Runs commands for creation of a dummy database, a table, insertion of data, display of rows and dropping of the table and database. These commands are run for the MyISAM, Memory and InnoDB engines.

Test 20: Check that the mysql user is unable to login on 5.7.6+

Verifies that the mysql user has /bin/false as its default shell.
This test is applicable for new-style RPMs and DEBs only(WL5608).

Test 21: Check the value of the secure_file_priv variable and data export/import functionality

Checks that the secure_file_priv variable has the correct path of the mysql-files directory.
Verifies that the export/import of data happens correctly.
This is implemented in 5.7.6+ packages, as a part of WL6782.

Test 22: Check that dtrace is enabled in the package

This test is implemented for OS X DMG and Solaris 11 as of now.
Runs 2 tests, viz:
1. Using PID provider, checks if MySQL probes are present.
2. Verifies MySQL query and connection probes by running specific D scripts. This is to check that mysql providers are present in the package.

Phase V – STOP SERVER

Test 23: STOP SERVER

Tests server stop.

Phase VI – UNINSTALL

Test 24: UNINSTALL

Uninstalls the package.

Future

We will keep adding new tests as new Worklogs that affect the functionality that Pkgver verifies get implemented. We will also be porting Pkgver to new OSes that get added to our list of supported platforms, and we will of course make sure that any future packaging related issue that may arise gets the headstone that every bug deserves: a new regression test in the test suite.

And finally, if you have viewpoints, comments or suggestions for improvements, please let us know in a comment below.