This content is no longer updated. Any further updates to test framework documention take place in the MySQL Source Code documentation and can be accessed at The MySQL Test Framework, Version 2.0.
Invoke mysql-test-run.pl in the
mysql-test directory like this:
shell> mysql-test-run.pl [options] [test_name] ...
test_name argument names a test
case. The test case file that corresponds to the test name is
If no suffix is given for the test name, a suffix of
.test is assumed. Any leading path name is
ignored. These commands are equivalent:
shell> mysql-test-run.pl mytest shell> mysql-test-run.pl mytest.test shell> mysql-test-run.pl t/mytest.test
A suite name can be given as part of the test name. That is, the syntax for naming a test is:
If a suite name is given, mysql-test-run.pl
looks in that suite for the test. The test file corresponding to
a test named
There is also an implicit suite name
the tests in the top
t directory. With no
suite name, mysql-test-run.pl looks in the
default list of suites for a match and runs the test in any
suites where it finds the test. Suppose that the default suite
rpl, and that a test
mytest.test exists in the
rpl suites. With
an argument of
mysql-test-run.pl will run
mytest.test from the
To run a family of test cases for which the names share a common
prefix, use the
option. For example,
--do-test=rpl runs the
replication tests (test cases that have names beginning with
--skip-test has the
opposite effect of skipping test cases for which the names share
a common prefix.
The argument for the
--skip-test options also allows more flexible
specification of which tests to perform or skip. If the argument
contains a pattern metacharacter other than a lone period, it is
interpreted as a Perl regular expression and applies to test
names that match the pattern. If the argument contains a lone
period or does not contain any pattern metacharacters, it is
interpreted the same way as previously and matches test names
that begin with the argument value. For example,
--do-test=testa matches tests that begin with
matches tests in the
main test suite that
--do-test=main.*testa matches test names that
main followed by
testa with anything in between. In the latter
case, the pattern match is not anchored to the beginning of the
test name, so it also matches names such as
As of MySQL 5.7, it is possible to put a list of test names in a
file and have mysql-test-run.pl run those
tests, using the option
The tests should be listed one per line in the file, using the
fully qualified name
A space may be used in place of the period. A line beginning
# indicates a comment and is ignored.
As of MySQL 8.0, mysql-test-run.pl supports a
--do-suite option, which is similar to
--do-test but permits specifying entire suites
of tests to run.
To perform setup prior to running tests,
mysql-test-run.pl needs to invoke
mysqld with the
--skip-grant-tables options. If MySQL was
built with the compiler flag
--init-file will be disabled. To handle this,
MYSQLD_BOOTSTRAP environment variable
to the full path name of a server that has all options enabled.
mysql-test-run.pl will use that server to
perform setup; it is not used to run the tests.
init_file test will fail if
--init-file is disabled. This is an expected
failure in this case.
To run mysql-test-run.pl on Windows, you'll
need either Cygwin or ActiveState Perl to run it. You may also
need to install the modules required by the script. To run the
test script, change location into the
mysql-test directory, set the
MTR_VS_CONFIG environment variable to the
configuration you selected earlier (or use the
--vs-config option), and invoke
mysql-test-run.pl. For example (using Cygwin
and the bash shell):
shell> cd mysql-test shell> export MTR_VS_CONFIG=debug shell> ./mysqltest-run.pl --force --timer shell> ./mysqltest-run.pl --force --timer --ps-protocol
mysql-test-run.pl uses several environment variables. Some of them are listed in the following table. Some of these are set from the outside and used by mysql-test-run.pl, others are set by mysql-test-run.pl instead, and may be referred to in tests.
|If set, defines which port number range is used for the server|
|If set to anything, will run tests with files in "memory" using tmpfs or
ramdisk. Not available on Windows. Same as
|If set, defines maximum number of parallel threads if
|Setting of a timeout in minutes or seconds, corresponding to command
|If set, defines number of parallel threads executing tests. Same as
|If set, defines which port number range is used for the server|
|Path name to mysql_config_editor binary. Supported as of MySQL 5.6.6.|
|Path name to mysqltest binary|
|Full path to the |
|Path name to login file used by mysql_config_editor.
If not set, the default is
|Path to temp directory used for temporary files during tests|
|Full path to server executable used in tests. Supported as of MySQL 5.5.17.|
|Full path name to mysqld that has all options enabled|
|Full command line used for initial database setup for this test batch|
|Command line for starting server as used in tests, with the minimum set of required arguments. Supported as of MySQL 5.5.17.|
|Path name to the |
|Path name to a file containing ThreadSanitizer suppressions. Supported as of MySQL 8.0.1.|
MTR_PORT_BASE is a more logical
replacement for the original variable
MTR_BUILD_THREAD. It gives the actual port
number directly (will be rounded down to a multiple of 10). If
MTR_BUILD_THREAD, the port number is
found by multiplying this by 10 and adding 10000.
Tests sometimes rely on certain environment variables being
defined. For example, certain tests assume that
MYSQL_TEST is defined so that
mysqltest can invoke itself with
Other tests may refer to the last three variables listed in the
preceding table, to locate files to read or write. For example,
tests that need to create files will typically put them in
$MYSQLD_CMD will include any
server options added with the
to mysql-test-run.pl, but will not include
server options added specifically for the currently running
Display a help message and exit.
Allow tests marked as "big" to run. Tests can be thus marked by including the line
--source include/big_test.inc, and they will only be run if this option is given, or if the environment variable
BIG_TESTis set to 1.
This is typically done for tests that take very long to run, or that use very much resources, so that they are not suitable for running as part of a normal test suite run.
Run the mysqld server used for bootstrapping the database through the dbx debugger. This option is available from MySQL 5.5.17.
Run the mysqld server used for bootstrapping the database through the ddd debugger. This option is available from MySQL 5.5.17.
Run the mysqld server used for bootstrapping the database through the gdb debugger. This option is available from MySQL 5.5.17.
See also the
Specify a number to calculate port numbers from. The formula is 10 *
build_thread+ 10000. Instead of a number, it can be set to
auto, which is also the default value, in which case mysql-test-run.pl will allocate a number unique to this host.
The value (number or
auto) can also be set with the
This option is kept for backward compatibility. The more logical
--port-baseis recommended instead.
Instructs valgrind to use callgrind.
Specify the default character set for the
testdatabase. The default value is
This option was added in MySQL 8.0.1.
Check test cases for side effects. This is done by checking the system state before and after each test case; if there is any difference, a warning to that effect is written, but the test case is not marked as failed because of it. This check is enabled by default. To disable it, use the
Clean up the
vardirectory with logs and test results etc. after the test run, but only if there were no test failures. This option only has effect if also running with option
--mem. The intent is to alleviate the problem of using up memory for test results, in cases where many different test runs are being done on the same host.
The path to the directory where client binaries are located.
Start mysqltest in the dbx debugger. Support for dbx is available from MySQL 5.5.12.
Start mysqltest in the ddd debugger.
Start mysqltest in the named debugger.
Start mysqltest in the gdb debugger.
The path to the directory where client libraries are located.
Extra option to pass to mysqld. The value should consist of a single mysqld option including dashes. This option is similar to
--mysqldbut has a different effect. mysql-test-run.pl executes multiple test runs, using the options for each instance of
--combinationin successive runs. If
--combinationis given only once, it has no effect. For test runs specific to a given test suite, an alternative to the use of
--combinationis to create a
combinationsfile in the suite directory. The file should contain a section of options for each test run. See Section 4.9, “Passing Options from mysql-test-run.pl to mysqld or mysqltest”.
strto the output within lines filled with
#, as a form of banner.
Compress all information sent between the client and the server if both support compression.
--cursor-protocoloption to mysqltest (implies
Start mysqld in the dbx debugger. Support for dbx is available from MySQL 5.5.12.
Start mysqld in the ddd debugger.
Dump trace output for all clients and servers.
Start mysqld using the named debugger.
This option works similar to
--debugbut turns on debug only for the debug macro keywords
query, info, error, enter, exitwhich are considered the most commonly used.
mysqld.debug(if available) instead of
mysqldas server. If it does find
mysqld.debug, it will search for plugin libraries in a subdirectory
debugunder the directory where it's normally located. This option does not turn on trace output and is independent of the
Controls whether the Debug Sync facility for testing and debugging is enabled. The option value is a timeout in seconds. The default value is 300. A value of 0 disables Debug Sync. The value of this option also becomes the default timeout for individual synchronization points.
--loose-debug-sync-timeout=to mysqld. The
--looseprefix is used so that mysqld does not fail if Debug Sync is not compiled in.
For information about using the Debug Sync facility for testing, see Section 4.15, “Thread Synchronization in Test Cases”.
MyISAMas the default storage engine for all except
InnoDB-specific tests. This option is on by default in MySQL 5.5 and 5.6, but is off by default as of MySQL 5.7. See also
Use the named file as fixed config file template for all tests.
Add setting from the named file to all generated configs.
Attempt to preload
discover, the Developer Studio Memory Error Discovery Tool when starting mysqld. Reports from
discovermay be found in
log/mysqld.%p.txtunder the directory given by
--vardir. This option was added in MySQL 8.0.1. It is supported only on SPARC-M7 systems.
Run all test cases from suites having a name that begins with the given
prefixvalue or matches the regular expression. If the argument matches no existing suites, mysql-test-run.pl aborts.
The argument for the
--do-suiteoption allows more flexible specification of which tests to perform. See the description of the
--do-testoption for details.
--do-suiteoption was added in MySQL 8.0.
Run all test cases having a name that begins with the given
prefixvalue or matches the regular expression. This option provides a convenient way to run a family of similarly named tests.
The argument for the
--do-testoption allows more flexible specification of which tests to perform. If the argument contains a pattern metacharacter other than a lone period, it is interpreted as a Perl regular expression and applies to test names that match the pattern. If the argument contains a lone period or does not contain any pattern metacharacters, it is interpreted the same way as previously and matches test names that begin with the argument value. For example,
--do-test=testamatches tests that begin with
--do-test=main.testamatches tests in the
maintest suite that begin with
--do-test=main.*testamatches test names that contain
testawith anything in between. In the latter case, the pattern match is not anchored to the beginning of the test name, so it also matches names such as
Run all tests listed in the file
file. In this file, tests should be listed one per line in the form
testor alternatively, with a space instead of the period. A line beginning with
#will be ignored and can be used for comments.
--do-test-listoption is available from MySQL 5.7.
Use a version of mysqltest built with the embedded server. This option was removed in MySQL 8.0.
disabled.deffile, and run also tests marked as disbaled. Success or failure of those tests will be reported the same way as other tests.
Specify a file that contains a list of test cases that should be displayed with the
[ exp-fail ]code rather than
[ fail ]if they fail.
For an example of a file that might be specified using this option, see
It is also possible to supply more than one
--experimental, test cases listed in all the files will be treated as experimental.
EXPLAIN EXTENDEDon all
Use an already running server. The option/value pair is what is needed by the mysql client to connect to the server. Each
--externcan only take one option/value pair as argument, so it you need more you need to repeat
--externfor each of them. Example:
./mysql-test-run.pl --extern socket=var/tmp/mysqld.1.sock alias
Note: If a test case has an
.optfile that requires the server to be restarted with specific options, the file will not be used. The test case likely will fail as a result.
Do not perform controlled shutdown when servers need to be restarted or at the end of the test run. This is equivalent to using
Enabling this option when a test is run, causes it to fail if MTR's internal check of the test case fails. If this option is disabled, only a warning is generated while the test passes. This option is enabled by default. For additional information, see the description of the
--fail-check-testcasesoption was added in MySQL 8.0.
Normally, mysql-test-run.pl exits if a test case fails.
--forcecauses execution to continue regardless of test case failure.
Always restart the server(s) between each tast case, whether it's needed or not. Will also restart between repeated runs of the same test case. This may be useful e.g. when looking for the source of a memory leak, as there will only have been one test run before the server exits.
Run tests with the gcov test coverage tool.
Start mysqld in the gdb debugger.
Run tests with the gprof profiling tool.
Run also tests that need Cluster.
EXPLAIN FORMAT=JSONon all SELECT, INSERT, REPLACE, UPDATE and DELETE queries. The
json-explain-protocoloption is available from MySQL 5.6.
This option is similar to
--boot-gdbbut attaches the debugger to the server during the bootstrapping process, permitting the use of a remote debugger. This option is available from MySQL 5.7.14.
Use a server that has already been started by the user in the dbx debugger. Support for dbx is available from MySQL 5.5.12.
Use a server that has already been started by the user in the ddd debugger.
Use a server that has already been started by the user in a debugger.
Use a server that has already been started by the user in the gdb debugger.
Marks progress with timing (in milliseconds) and line number in
The maximum number of simultaneous server connections that may be used per test. If not set, the maximum is 128. Minimum allowed limit is 8, maximum is 5120. Corresponds to the same option for mysqltest.
Limit the number of core files saved, to avoid filling up disks in case of a frequently crashing server. Defaults to 5, set to 0 for no limit. May also be set with the environment variable
Limit the number of data directories saved after failed tests, to avoid filling up disks in case of frequent failures. Defaults to 20, set to 0 for no limit. May also be set with the environment variable
Stop execution after the specified number of tests have failed, to avoid using up resources (and time) in case of massive failures. retries are noe counted, nor are failures of tests marked experimental. Defaults to 10, set to 0 for no limit. May also be set with the environment variable
This option is not supported on Windows.
Run the test suite in memory, using tmpfs or ramdisk. This can decrease test times significantly, in particular if you would otherwise be running over a remote file system. mysql-test-run.pl attempts to find a suitable location using a built-in list of standard locations for tmpfs and puts the
vardirectory there. This option also affects placement of temporary files, which are created in
The default list includes
/dev/shm. You can also enable this option by setting the environment variable
dir_nameis given, it is added to the beginning of the list of locations to search, so it takes precedence over any built-in locations.
Once you have run tests with
mysql-testdirectory, a soflink
varwill have been set up to the temporary directory, and this will be re-used the next time, until the soflink is deleted. Thus, you do not have to repeat the
--memoption next time.
Extra option to pass to mysqld. Only one option may be specified in
value; to specify more than one, use additional
--mysqldoptions. See Section 4.9, “Passing Options from mysql-test-run.pl to mysqld or mysqltest”.
Sets (or changes) an environment variable before starting mysqld. Varibles set in the environment from which you run mysql-test-run.pl will normally also be propagated to mysqld, but there may be cases where you want a setting just for a single run, or you may not want the setting to affect other programs. You may use additional
--mysqld-envoptions to set more than one variable.
Extra options to pass to mysqltest.
This option was added in MySQL 8.0.0.
--ndb-connectstring=to the master MySQL server. This option also prevents mysql-test-run.pl from starting a cluster. It is assumed that there is already a cluster running to which the server can connect with the given connectstring.
Disable the check for test case side effects. For additional information, see the description of the
For MySQL 5.5 or 5.6, do not override the build-in default engine to use MyISAM instead for non-InnoDB tests. Since the existing collection of tests were originally adapted for MyISAM as default, many tests will fail when this option is used, because the test behaves differently or produces different output when the engine switches to InnoDB.
From MySQL 5.7, the default engine for tests has been changed to InnoDB and this option will have no effect.
Do not reorder tests to reduce number of restarts, but run them in exactly the order given. If a whole suite is to be run, the tests are run in alphabetic order, though similiar combinations will be grouped together. If more than one suite is listed, the tests are run one suite at a time, in the order listed.
This option forces all tests to run, ignoring any
--skipcommands used in the test. This ensures that all tests are run. An excluded list (
excludenoskip.list) is maintained to track which tests should continue to be skipped. The
--no-skipoption continues to skip the tests that are named in the excluded list. The default value of
--no-skipintroduced variable is OFF, which implies users are not forced to run all tests unless the
--no-skipis explicitly used.
shell> mysql-test-run.pl --suite=innodb --no-skip
Cause mysqltest not to generate a timing file. The effect of this is that the report from each test case does not include the timing in milliseconds as it normally does.
Do not run unit tests, overriding default behavior or setting of the
Running of unit tests was enabled from MySQL 5.5.11.
Do not look for and report errors and warning in the server logs.
This option causes only big tests to run. Normal (non-big) tests are skipped. If both
--only-big-testswas added in MySQL 8.0.1.
Run tests using
Nparallel threads. By default, 1 thread is used. Use
MTR_PARALLELenvironment variable to
Nhas the same effect as specifying
MTR_MAX_PARALLELenvironment variable, if set, specifies the maximum number of parallel workers that can be spawned when the
--parallel=autooption is specified. If
--parallel=autois not specified,
MTR_MAX_PARALLELvariable has no effect.
Specify base of port numbers to be used; a block of 10 will be allocated.
Pshould be divisible by 10; if it is not, it will be rounded down. If running with more than one parallel test thread, thread 2 will use the next block of 10 and so on.
If the port number is given as
auto, which is also the default, mysql-test-run.plwill allocate a number unique to this host. The value may also be given with the environment variable
--port-basewas added in MySQL 5.1.45 as a more logical alternative to
--build-thread. If both are used,
Do not run any tests, but print details about all tests, in the order they would have been run.
--ps-protocoloption to mysqltest.
--recordoption to mysqltest. This option requires a specific test case to be named on the command line.
Reorder tests to minimize the number of server restarts needed. This is the default behavior. There is no guarantee that a particular set of tests will always end up in the same order.
Run each test
Nnumber of times.
Display the output of
SHOW VARIABLES. This can be used to verify that binaries are built with all required features.
At the end of the test run, write a summary of how much time was spent in various phases of execution. If you run with
--parallel, the total will exceed the wall clock time passed, since it will be summed over all threads.
The times reported should only be treated as approximations, and the exact points where the time is taken may also change between releases. If the test run is aborted, including if a test fails and
--forceis not in use, the time report will not be produced.
--report-timesis available from MySQL 5.5.
If a test fails, it is retried up to a maximum of
Nruns, but will terminate after 2 failures. Default is 3, set to 1 or 0 for no retries. This option has no effect unless
--forceis also used; without it, test execution will terminate after the first failure.
--retry-failureoptions do not affect how many times a test repeated with
--repeatmay fail in total, as each repetition is considered a new test case, which may in turn be retried if it fails.
Allow a failed and retried test to fail more than the default 2 times before giving it up. Setting it to 0 or 1 effectively turns off retries
This option was added in MySQL 8.0.0. As of MySQL 8.0.1, the
TSAN_OPTIONSenvironment variable can be set to specify the path name of a file containing ThreadSanitizer suppressions.
Max number of seconds to wait for servers to do controlled shutdown before killing them. Default is 10.
Do not apply combinations; ignore combinations file or option.
Do not start NDB Cluster; skip Cluster test cases. This option only has effect if you do have NDB, if not it will have no effect as it cannot run those tests anyway.
Do not start an NDB Cluster slave.
Skip replication test cases.
Do not start mysqld with support for SSL connections.
Specify a regular expression to be applied to test case names. Cases with names that match the expression are skipped. tests to skip.
The argument for the
--skip-testoption allows more flexible specification of which tests to skip. If the argument contains a pattern metacharacter other than a lone period, it is interpreted as a Perl regular expression and applies to test names that match the pattern. See the description of the
--do-testoption for details.
Specify a file listing tests that should be skipped (disabled).
The file has the same format as the
disabled.deffile listing disabled tests. With this option, disabling can be done on a case by case basis. The
--skip-test-listoption is supported from MySQL 5.5.
--skip-*options not otherwise recognized by mysql-test-run.pl are passed to the master server.
--sp-protocoloption to mysqltest.
If mysql-test-run.pl is started with the
--ssloption, it sets up a secure connection for all test cases. In this case, if mysqld does not support SSL, mysql-test-run.pl exits with an error message:
Couldn't find support for SSL
Initialize and start servers with the startup settings for the specified test case. You can use this option to start a server to which you can connect later. For example, after building a source distribution you can start a server and connect to it with the mysql client like this:
shell> cd mysql-test shell> ./mysql-test-run.pl --start alias & shell> ../mysql -S ./var/tmp/master.sock -h localhost -u root
If no tests are named on the command line, the server(s) will be started with settings for the first test that would have been run without the
mysql-test-run.pl will stop once the server has been started, but will terminate if the server dies. If killed, it will also shut down the server.
This is similar to
--start, but mysql-test-run.pl terminates once the server has been started, leaving just the server process running.
This is similar to
--start, but will skip the database initialization phase and assume that database files are already available. Usually this means you must have run another test first.
mysql-test-run.pl sorts the list of names of the test cases to be run, and then begins with
Create strace output for mysqltest. Will produce default strace output as
mysqltest.strace. Note that this will be overwritten for each new test case, so it's most useful for running only one test.
strace-clientoption is functional from MySQL 5.5.20, and only supported on Linux. The option was available in earlier versions too, but was not working properly.
Create strace output for the server. Will produce default strace output as
mysqld.1.strace. Note that this will be overwritten each time the server is restarted, so it's most useful for running a single test, or if you want trace from the first test that fails.
strace-serveroption is available from MySQL 5.5.20, on Linux only.
Start a server, but instead of running a test, run mysql-stress-test.pl with the supplied arguments. Arguments needed to communicate with the server will be automatically provided, the rest should be given as arguments to this option. Command line options for mysql-stress-test.pl should be separeted by a comma.
stressoption was added in MySQL 5.5.17, it is not a direct replacement for the option of the same name that exists in version 1 of mysql-test-run.pl.
Run the named test suite. The default name is
main(the regular test suite located in the
Specify the maximum test suite runtime in minutes.
Generate a plain text version of the test summary only and write it to the file named as the option argument. The file is suitable for sending by email. This option was added in MySQL 8.0.1.
Display the percentage of tests remaining. This option was added in MySQL 5.7.19.
Specify the maximum test case runtime in minutes.
Adds to each test report for a test case, the total time in sconds and milliseconds passed since the preceding test ended. This option can only be used together with
--timestamp, and has no effect without it.
Cause mysqltest to generate a timing file. The default file is named
Prints a timestamp before the test case name in each test report line, showing when the test ended.
The directory where temporary file are stored. The default location is
./var/tmp. The environment variable
MYSQL_TMP_DIRwill be set to the path for this directory, whether it has the default value or has been set explicitly. This may be referred to in tests.
Force running of unit tests, overriding default behavior or setting of the
Running of unit tests was enabled from MySQL 5.5.11.
Extend the unit test run by also outputting the log from the test run, independently of whether it succeeded or not. This option implies
--unit-testsso it is not necessary to specify both. The
--unit-tests-reportoption is available in MySQL 5.5 from version 5.5.44, in 5.6 from version 5.6.25 as well as in MySQL 5.7.
The MySQL user name to use when connecting to the server.
Drops all non-essential command line arguments to the mysqld server, except those supplied with
--mysqldarguemnts, if any. Only works in combination with
--start-dirty, and only if no test name is given.
Run mysqltest and mysqld with valgrind. This and the following
--valgrindoptions require that the executables have been build with valgrind support.
When the server is run with valgrind, an extra pass over the server log file(s) will be performed after all tests are run, and any report with problems that have been reported at server shutdown will be extracted and printed. The most common warnings are memory leaks. With each report will also be listed all tests that were run since previous server restart; one of these is likely to have caused the problem.
From MySQL 5.5.13, a final "pseudo" test named
valgrind_reportis added to the list of tests when the server is run in valgrind. This test is reported as failed if any such shutdown warnings were produced by valgrind. Pass or failure of this test is also added to the total test count reported.
Run all clients started by
.testfiles with valgrind. This option requires valgrind 3.9 or later.
--valgrind-clientswas added in MySQL 5.7.9.
Run the mysqld server with valgrind.
Run mysqltest with valgrind.
Extra options to pass to valgrind.
Specify the path name to the valgrind executable.
Specify the path where files generated during the test run are stored. The default location is
./var. The environment variable
MYSQLTEST_VARDIRwill be set to the path for this directory, whether it has the default value or has been set explicitly. This may be referred to in tests.
Give more verbose output regarding test execution. Use the option twice to get even more output. Note that the output generated within each test case is not affected.
Write when and why servers are restarted between test cases.
--view-protocoloption to mysqltest.
Specify the configuration used to build MySQL (for example,
--vs-config=release). This option is for Windows only.
--start-dirtyis used, wait for all servers to exit before termination. Otherise, it will terminate if one (of several) servers is restarted.
Search the server log for errors or warning after each test and report any suspicious ones; if any are found, the test will be marked as failed. This is the default behavior, it may be turned off with
Run only test cases that have
ndbin their name.
The hostname resolves to 127.0.0.1 and not to the actual IP address.