The first DMR of MySQL 8 was recently released. While a DMR (or Developer Milestone Release) is not yet a GA product, it does come with our commitment that each feature has been tested and qualified for inclusion.

In MySQL 8.0 we have made additional improvements to the MySQL test framework (MTR) and the test suite.

Ability to mark tests as Fail when check-testcase fails(WL#7730)

MTR has an option called check-testcase which checks if tests are cleaning up after each run. It does this by  checking if there is any change in the system state before  and after the test run. The current behavior is such that, a warning is issued if there is a change in the system  state, but the test does not fail. This approach has the problem that warnings are not highlighted in regression test environment and check-testcase failures are  getting ignored.  We have solved this by introducing a new option --fail-check-testcase. When the  option is ON tests will fail when check-testcase fails .  The option is OFF by default, but is turned ON in our regression test suite.  This feature is implemented  by Sayantan Dutta.

MTR to initialize Data Directory using mysqld (WL#8403)

This enhancement implemented by Sayantan Dutta, makes use of the new method for initializing Data Dictionary that was introduced in 5.7.  Prior to 5.7, the method for initialization was platform dependent. On some platforms it had to be done manually using mysql_install_db tool and on some others it was done automatically as part of the initialization. Since tests have to run uniformly on all platforms, MTR had to implement its own method for initializing the database. In fact most users were relying on MTR to start the database because of the complexity of other methods. Now that mysqld is made capable of doing the initialization the process has become platform independent and MTR is  able to use it instead of the custom code. This change ensures that MTR starts server the same way as a user does. A big thanks to Georgi Kodinov for implementing this easy method for initialization. You may read more about the server feature in this post.

MTR to run with default values of server variable (WL#8795)

This work done by Deepa Dixit makes MTR tests run with the default values of server variables wherever possible. This makes our tests run in an environment that is as close as possible to that used by community.

New MTR option to avoid tests that are skipped in regression test runs (WL#9073)

MTR has a command called “skip” that allow tests to be skipped if  conditions necessary for running the test are not met. This  is a good feature and perhaps unavoidable considering the  wide array of configuration options and platforms supported by  MySQL. However this also has the undesirable effect of some  tests never running in the regression test environment. As  a result we end up with some features that may not get  tested regularly. We have now added a new MTR option called --no-skip. If this is set tests that do not meet required  conditions will fail instead of getting skipped and will be  noticed in the regression test run. This option is OFF by default. This feature was  implemented by Mohit Joshi.

Restructuring of test suite to reduce usage of skip (WL#8036, WL#9027, WL#9170 )

In addition to adding the --no-skip option test suite was restructured to minimize the usage of skip itself.Some of the changes are:

• Removed redundant skips. i.e. removed checks for various charsets since they cannot be compiled out in 8.0.0
• Avoided usage of skip to check for certain value for a system variable. The values were set in the test itself thereby making the skip condition unnecessary.

There are more such improvements and Mohit will be writing about it in a separate blog.

Migration of tests in main suite to run with InnoDB(WL#9272, WL#7264)

This is a continuation of the migration project that was started in 5.7. I have blogged about it at here previously. We have come a long way with this project and out of the 6000+ MTR tests, fewer than 400 remain to be migrated. These migration tasks were implemented by Mohit Joshi and Viswanatham Gudipati.

Other MTR Enhancements

New MTR options:
* –do-suite – Runs all suites that begins with a given prefix or matcches a regex. This can be useful if one wants to run all rpl suites, all innodb suites etc

New MTR commands:
–force-rmdir – Recursively removes directory.
–copy_files_wildcard – copies all files that match a pattern from a source directory to a destination directory
Above enhancements were implemented by Pavan Naik and Deepa Dixit.

Please refer to the release notes  for full list of MTR enhancements.

Along with the test infrastructure improvements MySQL test suite has also been steadily growing. MySQL 8.0.0 release has 5102 MTR tests which is 229 more than 5.7. This shows that all our features are subjected to a rigorous testing regime.

MTR documentation is updated with information on the new features.  Please try them out and let us know if you have any feedback. We will also be happy to evaluate any new feature requests for MTR.

Stay tuned for more detailed blogs from MySQL Server QA team members on some of these features.