Test Framework Programs

The MySQL test framework uses several programs:

The mysql-test-run.pl Perl script is the main application used to run the test suite. It invokes mysqltest to run individual test cases.
mysqltest runs test cases.
The mysql_client_test program is used for testing aspects of the MySQL client API that cannot be tested using mysqltest and its test language.
The mysql-stress-test.pl Perl script performs stress-testing of the MySQL server.
A unit-testing facility is provided so that individual unit test programs can be created for storage engines and plugins.

Test suite programs can be found in these locations:

For a source distribution, mysqltest is in the client directory. For a binary distribution, it is in the MySQL bin directory.
For a source distribution, mysql_client_test is in the tests directory. For a binary distribution, it is in the MySQL bin directory.
The other programs are located in the mysql-test directory. For a source distribution, mysql-test is found under the source tree root. For a binary distribution, the location of mysql-test depends on the layout used for the distribution format.

Test Framework Directories and Files

The test suite is located in the mysql-test directory, which contains the following components:

The mysql-test-run.pl and mysql-stress-test.pl programs that are used for running tests.
The t directory contains test case input files. A test case file might also have additional files associated with it.
- A file name of the form test_name.test is a test case file for a test named test_name. For example, subquery.test is the test case file for the test named subquery.
- A file name of the form test_name-master.opt provides options to associate with the named test case. mysql-test-run.pl restarts the server with the options given in the file if the options are different from those required for the currently running server.
  
  Note that the -master.opt file is used for the “main” server of a test, even if no replication is involved.
- A file name of the form test_name-slave.opt provides slave options.
- Server options which need to be passed during initialization of the server can be passed in both test_name-master.opt and test_name-slave.opt. Each bootstrap variable must be passed as the value of a --initialize option so that mysql-test-run.pl can recognize that the variable should be used during server initialization, remove the existing data directory, and initialize a new data directory with the options provided. mysql-test-run.pl will also start the server afterwards, with the bootstrap options, and other options specified.
- A file name of the form test_name-client.opt provides mysqltest client options.
- A file name of the form test_name.cnf contains additional entries for the config file to be used for this test.
  - During a test run, each server is started with a different defaults-group-suffix value and this suffix value can be used with a group name to specify server specific options. For example, section [mysqld.1] is used to specify options for mysqld server with suffix value ".1".
  - The [mysqltest] section is used to specify options for mysqltest client.
  - The [ENV] section is used to specify environment variables for a test case.
  - To include the options mentioned in another .cnf file, !include path_to_cnf_file is used. Path to the included .cnf should either be an absolute path or a relative path from current location or from mysql-test directory.
  Here is a sample .cnf file.
```
!include include/default_my.cnf

[mysqld.1]
Options for server mysqld.1

[mysqld.2]
Options for server mysqld.2

[mysqltest]
ps-protocol

...
...
...

[ENV]
SERVER_MYPORT_1= @mysqld.1.port
SERVER_MYPORT_2= @mysqld.2.port
```
- A file name of the form test_name.combinations provides section of options for each test run.
- A file name of the form suite.opt provides server options to all the tests within the suite it belongs to.
  
  If a test run is started with more than one server, the options specified in a suite.opt are used by all of them. It is also possible to pass the server options needed during the server initialization in the suite.opt file.
  
  The options mentioned in the suite.opt file are overridden by those found in a test_name-master.opt file or in a test_name-slave.opt file.
- A file name of the form test_name-master.sh is a shell script that will be executed before the server is started for the named test case. There may also be a test_name-slave.sh which is executed before the slave server is started.
  
  These files will not work on Windows and may therefore be replaced with a more portable mechanism in the future.
- The disabled.def file contains information about deferred/disabled tests. When a test is failing because of a bug in the server and you want it to be ignored by mysql-test-run.pl, list the test in this file.
  
  The format of a line in the disabled.def file looks like this, where fields are separated by one or more spaces.
```
suite_name.test_name [@platform|@!platform] : <BUG|WL>#<XXXX> [<comment>]
```
  Example:
```
main.1st @linux : BUG#18980 Test fails randomly
```
  suite_name is the suite name. test_name is the test case name.
  
  An additional element after the test name and before the colon you may add a @platform to have the test disabled only on a specific platform. Basic OS names as reported by $^O in Perl, or 'windows' are supported. Alternatively, @!platform will disable the test on all except the named platform.
  
  BUG#nnnnn or WL#nnnn indicates the bug or the WL related to the test that causes it to fail (and thus requires it to be disabled).
  
  The comment text following the BUG#nnnnn or the WL#nnnn is not part of the mandatory syntax for this file, mysql-test-run.pl will not actually care what is written here. Length of a comment text must not be more than 80 characters.
  
  A comment line can be written in the file by beginning the line with a “#” character.
The r directory contains test case result files:
- A file name of the form test_name.result is the expected result for the named test case. A file r/test_name.result is the output that corresponds to the input in the test case file t/test_name.test.
- A file name of the form test_name.reject contains output for the named test case if the test fails due to output mismatch (but not it it fails for other reasons).
For a test case that succeeds, the .result file represents both the expected and actual result. For a test case that fails, the .result file represents the expected result, and the .reject file represents the actual result.

If a .reject file is created because a test fails, mysql-test-run.pl removes the file later the next time the test succeeds.

When --check-testcases option is enabled, a test case not having its corresponding .result file is marked as failed by MTR. See mysql-test-run.pl.
The include directory contains files that are included by test case files using the source command. These include files encapsulate operations of varying complexity into a single file so that you can perform the operations in a single step. See Using Include Files to Simplify Test Cases.
The lib directory contains library files used by mysql-test-run.pl, and database initialization SQL code.
The std_data directory contains data files used by some of the tests.
The var directory is used during test runs for various kinds of files: log files, temporary files, trace files, Unix socket files for the servers started during the tests, and so forth. This directory cannot be shared by simultaneous test runs.
The suite directory contains a set of subdirectories containing named test suites. Each subdirectory represents a test suite with the same name.

A test suite directory contains the following components:
- The t directory contains test cases.
- The r directory contains test case result files.
- The include directory contains files that are included by the test cases belonging to that suite.
- A file name of the form combinations provides section of options for each test run. See Controlling the Binary Log Format Used for an Entire Test Run.
- A file name of the form my.cnf contains additional entries for the config file used by all the tests within the suite it belongs to.
  
  A suite specific my.cnf file is overridden by a test_name.cnf file.
The collections directory contains collections of test runs that are run during integration and release testing of MySQL. They are not directly useful outside this context, but need to be part of the source repository and are included for reference.

Unit test-related files are located in the unittest directory. Additional files specific to storage engines and plugins may be present under the subdirectories of the storage or plugin directories.

Test Execution and Evaluation

There are a number of targets in the top-level Makefile that can be used to run sets of tests. make test only runs unit tests. Other targets run subsets of the tests, or run tests with specific options for the test programs. Have a look at the Makefile to see what targets are available.

A “test case” is a single file. The case might contain multiple individual test commands. If any individual command fails, the entire test case is considered to fail. Note that “fail” means “does not produce the expected result.” It does not necessarily mean “executes without error,” because some tests are written precisely to verify that an illegal statement does in fact produce an error. In such an instance, if the statement executes successfully without producing the expected error, that is considered failure of the test.

Test case output (the test result) consists of:

Input SQL statements and their output. Each statement is written to the result followed by its output. Columns in output resulting from SQL statements are separated by tab characters.
The result from mysqltest commands such as echo and exec. The commands themselves are not echoed to the result, only their output.

The disable_query_log and enable_query_log commands control logging of input SQL statements. The disable_result_log and enable_result_log commands control logging of SQL statement results, and warning or error messages resulting from those statements.

mysqltest reads a test case file from its standard input by default. The --test-file or -x option can be given to name a test case file explicitly.

mysqltest writes test case output to the standard output by default. The --result-file or -R option can be used to indicate the location of the result file. That option, together with the --record option, determine how mysqltest treats the test actual and expected results for a test case:

If the test produces no results, mysqltest exits with an error message to that effect, unless --result-file is given and this file is empty.
Otherwise, if --result-file is not given, mysqltest sends test results to the standard output.
With --result-file but not --record, mysqltest reads the expected results from the given file and compares them with the actual results. If the results do not match, mysqltest writes a .reject file in the log directory and exits with an error. It will also print out a diff of the expected and actual result, provided it can find an appropriate diff tool to use.
With both --result-file and --record, mysqltest updates the given file by writing the actual test results to it. The file does not need to pre-exist.

mysqltest itself knows nothing of the t and r directories under the mysql-test directory. The use of files in those directories is a convention that is used by mysql-test-run.pl, which invokes mysqltest with the appropriate options for each test case to tell mysqltest where to read input and write output.

Table of Contents

Test Framework Programs

Test Framework Directories and Files

Test Execution and Evaluation