Documentation Home
The MySQL Test Framework, Version 1.0
Download this Manual
PDF (US Ltr) - 443.9Kb
PDF (A4) - 445.6Kb
EPUB - 101.9Kb


The MySQL Test Framework, Version 1.0  /  MySQL Test Programs  /  mysql-test-run.pl — Run MySQL Test Suite

5.3 mysql-test-run.pl — Run MySQL Test Suite

The mysql-test-run.pl Perl script is the main application used to run the MySQL test suite. It invokes mysqltest to run individual test cases. (Prior to MySQL 4.1, a similar shell script, mysql-test-run, can be used instead.)

Invoke mysql-test-run.pl in the mysql-test directory like this:

shell> mysql-test-run.pl [options] [test_name] ...

Each test_name argument names a test case. The test case file that corresponds to the test name is t/test_name.test.

For each test_name argument, mysql-test-run.pl runs the named test case. With no test_name arguments, mysql-test-run.pl runs all .test files in the t subdirectory.

If no suffix is given for the test name, a suffix of .test is assumed. Any leading path name is ignored. These commands are equivalent:

shell> mysql-test-run.pl mytest
shell> mysql-test-run.pl mytest.test
shell> mysql-test-run.pl t/mytest.test

As of MySQL 5.1.23, a suite name can be given as part of the test name. That is, the syntax for naming a test is:

[suite_name.]test_name[.suffix]

If a suite name is given, mysql-test-run.pl looks in that suite for the test. With no suite name, mysql-test-run.pl looks in the default list of suites for a match and runs the test in any suites where it finds the test. Suppose that the default suite list is main, binlog, rpl, and that a test mytest.test exists in the main and rpl suites. With an argument of mytest or mytest.test, mysql-test-run.pl will run mytest.test from the main and rpl suites.

To run a family of test cases for which the names share a common prefix, use the --do-test=prefix option. For example, --do-test=rpl runs the replication tests (test cases that have names beginning with rpl). --skip-test has the opposite effect of skipping test cases for which the names share a common prefix.

As of MySQL 5.0.54/5.1.23/6.0.5, the argument for the --do-test and --skip-test options allows more flexible specification of which tests to perform or skip. If the argument contains a pattern metacharacter other than a lone period, it is interpreted as a Perl regular expression and applies to test names that match the pattern. If the argument contains a lone period or does not contain any pattern metacharacters, it is interpreted the same way as previously and matches test names that begin with the argument value. For example, --do-test=testa matches tests that begin with testa, --do-test=main.testa matches tests in the main test suite that begin with testa, and --do-test=main.*testa matches test names that contain main followed by testa with anything in between. In the latter case, the pattern match is not anchored to the beginning of the test name, so it also matches names such as xmainytestz.

To perform setup prior to running tests, mysql-test-run.pl needs to invoke mysqld with the --bootstrap and --skip-grant-tables options If MySQL was configured with the --disable-grant-options option, --bootstrap, --skip-grant-tables, and --init-file will be disabled. To handle this, set the MYSQLD_BOOTSTRAP environment variable to the full path name of a server that has all options enabled. mysql-test-run.pl will use that server to perform setup; it is not used to run the tests.

The init_file test will fail if --init-file is disabled. This is an expected failure that can be handled as follows:

shell> export MYSQLD_BOOTSTRAP
shell> MYSQLD_BOOTSTRAP=/full/path/to/mysqld
shell> make test force="--skip-test=init_file"

To run mysql-test-run.pl on Windows, you'll need either Cygwin or ActiveState Perl to run it. You may also need to install the modules required by the script. To run the test script, change location into the mysql-test directory, set the MTR_VS_CONFIG environment variable to the configuration you selected earlier (or use the --vs-config option), and invoke mysql-test-run.pl. For example (using Cygwin and the bash shell):

shell> cd mysql-test
shell> export MTR_VS_CONFIG=debug
shell> ./mysqltest-run.pl --force --timer
shell> ./mysqltest-run.pl --force --timer --ps-protocol

If you have a copy of mysqld running on the machine where you want to run the test suite, you do not have to stop it, as long as it is not using ports 9306 or 9307. If either of those ports is taken, you should set the MTR_BUILD_THREAD environment variable to an appropriate value, and the test suite will use a different set of ports for master, slave, NDB, and Instance Manager). For example:

shell> export MTR_BUILD_THREAD=31
shell> ./mysql-test-run.pl [options] [test_name]

mysql-test-run.pl defines several environment variables. Some of them are listed in the following table.

VariableDescription
MYSQL_TESTPath name to mysqltest binary
MYSQLTEST_VARDIRPath name to the var directory that is used for logs, temporary files, and so forth
MYSQLD_BOOTSTRAPFull path name to mysqld that has all options enabled
MASTER_MYPORT???
MASTER_MYSOCK???

Tests sometimes rely on certain environment variables being defined. For example, certain tests assume that MYSQL_TEST is defined so that mysqltest can invoke itself with exec $MYSQL_TEST.

mysql-test-run.pl supports the options in the following list. An argument of -- tells mysql-test-run.pl not to process any following arguments as options. (A description of differences between the options supported by mysql-test-run.pl and mysql-test-run appears following the list.)

  • --help, -h

    Display a help message and exit.

  • --bench

    Run the benchmark suite.

  • --benchdir=path

    The directory where the benchmark suite is located. The default path is ../../mysql-bench.

  • --big-test

    Pass the --big-test option to mysqltest.

  • --check-testcases

    Check test cases for side effects.

  • --client-bindir

    The path to the directory where client binaries are located. This option was added in MySQL 5.0.66/5.1.27.

  • --client-ddd

    Start mysqltest in the ddd debugger.

  • --client-debugger

    Start mysqltest in the named debugger.

  • --client-gdb

    Start mysqltest in the gdb debugger.

  • --client-libdir

    The path to the directory where client libraries are located. This option was added in MySQL 5.0.66/5.1.27.

  • --combination=value

    Extra options to pass to mysqld. The value should consist of one or more comma-separated mysqld options. This option is similar to --mysqld but should be given two or more times. mysql-test-run.pl executes multiple test runs, using the options for each instance of --combination in successive runs. If --combination is given only once, it has no effect. For test runs specific to a given test suite, an alternative to the use of --combination is to create a combinations file in the suite directory. The file should contain a section of options for each test run. See Section 4.9, “Passing Options from mysql-test-run.pl to mysqld or mysqltest.

    This option was added in MySQL 5.1.23/6.0.4.

  • --comment=str

    Write str to the output.

  • --compress

    Compress all information sent between the client and the server if both support compression.

  • --cursor-protocol

    Pass the --cursor-protocol option to mysqltest (implies --ps-protocol).

  • --ddd

    Start mysqld in the ddd debugger.

  • --debug

    Dump trace output for all clients and servers.

  • --debugger

    Start mysqld using the named debugger.

  • --debug-sync-timeout=seconds

    Controls whether the Debug Sync facility for testing and debugging is enabled. The option value is a timeout in seconds. The default value is 300. A value of 0 disables Debug Sync. The value of this option also becomes the default timeout for individual synchronization points.

    mysql-test-run.pl passes --loose-debug-sync-timeout=seconds to mysqld. The --loose prefix is used so that mysqld does not fail if Debug Sync is not compiled in.

    For information about using the Debug Sync facility for testing, see Section 4.14, “Thread Synchronization in Test Cases”.

    This option was added in MySQL 5.1.41/5.5.0/6.0.6.

  • --do-test=prefix

    Run all test cases having a name that begins with the given prefix value. This option provides a convenient way to run a family of similarly named tests.

    As of MySQL 5.0.54/5.1.23/6.0.5, the argument for the --do-test option allows more flexible specification of which tests to perform. If the argument contains a pattern metacharacter other than a lone period, it is interpreted as a Perl regular expression and applies to test names that match the pattern. If the argument contains a lone period or does not contain any pattern metacharacters, it is interpreted the same way as previously and matches test names that begin with the argument value. For example, --do-test=testa matches tests that begin with testa, --do-test=main.testa matches tests in the main test suite that begin with testa, and --do-test=main.*testa matches test names that contain main followed by testa with anything in between. In the latter case, the pattern match is not anchored to the beginning of the test name, so it also matches names such as xmainytestz.

  • --embedded-server

    Use a version of mysqltest built with the embedded server.

  • --experimental=file_name

    Specify a file that contains a list of test cases that should be displayed with the [ exp-fail ] code rather than [ fail ] if they fail. This option was added in MySQL 5.1.33/6.0.11.

    For an example of a file that might be specified using this option, see mysql-test/collections/default.experimental.

  • --extern

    Use an already running server.

    Note: If a test case has an .opt file that requires the server to be restarted with specific options, the file will not be used. The test case likely will fail as a result.

  • --fast

    Do not clean up from earlier test runs.

  • --force

    Normally, mysql-test-run.pl exits if a test case fails. --force causes execution to continue regardless of test case failure.

  • --gcov

    Run tests with the gcov test coverage tool.

  • --gdb

    Start mysqld in the gdb debugger.

  • --gprof

    Run tests with the gprof profiling tool.

  • --im-mysqld1-port

    TCP/IP port number to use for the first mysqld, controlled by Instance Manager.

  • --im-mysqld2-port

    TCP/IP port number to use for the second mysqld, controlled by Instance Manager.

  • --im-port

    TCP/IP port number to use for mysqld, controlled by Instance Manager.

  • --log-warnings

    Pass the --log-warnings option to mysqld.

  • --manual-debug

    Use a server that has already been started by the user in a debugger.

  • --manual-gdb

    Use a server that has already been started by the user in the gdb debugger.

  • --master-binary=path

    Specify the path of the mysqld binary to use for master servers.

  • --master_port=port_num

    Specify the TCP/IP port number for the first master server to use. Observe that the option name has an underscore and not a dash.

  • --mem

    Run the test suite in memory, using tmpfs or ramdisk. This can decrease test times significantly. mysql-test-run.pl attempts to find a suitable location using a built-in list of standard locations for tmpfs and puts the var directory there. This option also affects placement of temporary files, which are created in var/tmp.

    The default list includes /dev/shm. You can also enable this option by setting the environment variable MTR_MEM[=dir_name]. If dir_name is given, it is added to the beginning of the list of locations to search, so it takes precedence over any built-in locations.

    This option was added in MySQL 4.1.22, 5.0.30, and 5.1.13. It is not supported on Windows.

  • --mysqld=value

    Extra options to pass to mysqld. The value should consist of one or more comma-separated mysqld options. See Section 4.9, “Passing Options from mysql-test-run.pl to mysqld or mysqltest.

  • --mysqltest=value

    Extra options to pass to mysqltest. The value should consist of one or more comma-separated mysqltest options. See Section 4.9, “Passing Options from mysql-test-run.pl to mysqld or mysqltest. This option was added in MySQL 6.0.6.

  • --ndb-connectstring=str

    Pass --ndb-connectstring=str to the master MySQL server. This option also prevents mysql-test-run.pl from starting a cluster. It is assumed that there is already a cluster running to which the server can connect with the given connectstring.

  • --ndb-connectstring-slave=str

    Pass --ndb-connectstring=str to slave MySQL servers. This option also prevents mysql-test-run.pl from starting a cluster. It is assumed that there is already a cluster running to which the server can connect with the given connectstring.

  • --ndb-extra-test

    Unknown.

  • --ndbcluster-port=port_num, --ndbcluster_port=port_num

    Specify the TCP/IP port number that NDB Cluster should use.

  • --ndbcluster-port-slave=port_num

    Specify the TCP/IP port number that the slave NDB Cluster should use.

  • --netware

    Run mysqld with options needed on NetWare.

  • --notimer

    Cause mysqltest not to generate a timing file.

  • --parallel={N|auto}

    Run tests using N parallel threads. By default, 1 thread is used. Use --parallel=auto for auto-setting of N. This option was added in MySQL 5.1.36.

  • --ps-protocol

    Pass the --ps-protocol option to mysqltest.

  • --record

    Pass the --record option to mysqltest. This option requires a specific test case to be named on the command line.

  • --reorder

    Reorder tests to minimize the number of server restarts needed.

  • --report-features

    Display the output of SHOW ENGINES and SHOW VARIABLES. This can be used to verify that binaries are built with all required features.

    This option was added in MySQL 4.1.23, 5.0.30, and 5.1.14.

  • --script-debug

    Enable debug output for mysql-test-run.pl itself.

  • --skip-im

    Do not start Instance Manager; skip Instance Manager test cases.

  • --skip-master-binlog

    Do not enable master server binary logging.

  • --skip-ndbcluster, --skip-ndb

    Do not start NDB Cluster; skip Cluster test cases.

  • --skip-ndbcluster-slave, --skip-ndb-slave

    Do not start an NDB Cluster slave.

  • --skip-rpl

    Skip replication test cases.

  • --skip-slave-binlog

    Do not enable master server binary logging.

  • --skip-ssl

    Do not start mysqld with support for SSL connections.

  • --skip-test=regex

    Specify a regular expression to be applied to test case names. Cases with names that match the expression are skipped. tests to skip.

    As of MySQL 5.0.54/5.1.23/6.0.5, the argument for the --skip-test option allows more flexible specification of which tests to skip. If the argument contains a pattern metacharacter other than a lone period, it is interpreted as a Perl regular expression and applies to test names that match the pattern. See the description of the --do-test option for details.

  • --skip-*

    --skip-* options not otherwise recognized by mysql-test-run.pl are passed to the master server.

  • --slave-binary=path

    Specify the path of the mysqld binary to use for slave servers.

  • --slave_port=port_num

    Specify the TCP/IP port number for the first master server to use. Observe that the option name has an underscore and not a dash.

  • --sleep=N

    Pass --sleep=N to mysqltest.

  • --small-bench

    Run the benchmarks with the --small-tests and --small-tables options.

  • --socket=file_name

    For connections to localhost, the Unix socket file to use, or, on Windows, the name of the named pipe to use.

  • --sp-protocol

    Pass the --sp-protocol option to mysqltest.

  • --ssl

    If mysql-test-run.pl is started with the --ssl option, it sets up a secure connection for all test cases. In this case, if mysqld does not support SSL, mysql-test-run.pl exits with an error message: Couldn't find support for SSL

  • --start

    Initialize and start servers with the startup settings for the first specified test case. For example:

    shell> cd mysql-test
    shell> ./mysql-test-run.pl --start alias &
    

    This option was added in MySQL 5.1.32/6.0.11.

  • --start-and-exit

    Initialize and start servers with the startup settings for the specified test case or cases, if any, and then exit. You can use this option to start a server to which you can connect later. For example, after building a source distribution you can start a server and connect to it with the mysql client like this:

    shell> cd mysql-test
    shell> ./mysql-test-run.pl --start-and-exit
    shell> ../mysql -S ./var/tmp/master.sock -h localhost -u root
    
  • --start-dirty

    Start servers (without initialization) for the specified test case or cases, if any, and then exit. You can then manually run the test cases.

  • --start-from=test_name

    mysql-test-run.pl sorts the list of names of the test cases to be run, and then begins with test_name.

  • --strace-client

    Create strace output for mysqltest.

  • --stress

    Run the stress test. The other --stress-xxx options apply in this case.

  • --stress-init-file=file_name

    file_name is the location of the file that contains the list of tests. The default file is stress_init.txt in the test suite directory.

  • --stress-loop-count=N

    In sequential stress-test mode, the number of loops to execute before exiting.

  • --stress-mode=mode

    This option indicates the test order in stress-test mode. The mode value is either random to select tests in random order or seq to run tests in each thread in the order specified in the test list file. The default mode is random.

  • --stress-suite=suite_name

    The name of the test suite to use for stress testing. The default suite name is main (the regular test suite located in the mysql-test directory).

  • --stress-test-count=N

    For stress testing, the number of tests to execute before exiting.

  • --stress-test-duration=N

    For stress testing, the duration of stress testing in seconds.

  • --stress-test-file=file_name

    The file that contains the list of tests to use in stress testing. The tests should be named without the .test extension. The default file is stress_tests.txt in the test suite directory.

  • --stress-threads=N

    The number of threads to use in stress testing. The default is 5.

  • --suite=suite_name

    Run the named test suite. The default name is main (the regular test suite located in the mysql-test directory).

  • --suite-timeout=minutes

    Specify the maximum test suite runtime in minutes.

  • --testcase-timeout=minutes

    Specify the maximum test case runtime in minutes.

  • --timer

    Cause mysqltest to generate a timing file. The default file is named ./var/log/timer.

  • --tmpdir=path

    The directory where temporary file are stored. The default location is ./var/tmp.

  • --unified-diff, --udiff

    Use unified diff format when presenting differences between expected and actual test case results.

  • --use-old-data

    Do not install the test databases. (Use existing ones.)

  • --user-test=val

    Unused.

  • --user=user_name

    The MySQL user name to use when connecting to the server.

  • --valgrind

    Run mysqltest and mysqld with valgrind.

  • --valgrind-all

    Like --valgrind, but passes the --verbose and --show-reachable options to valgrind.

  • --valgrind-mysqltest

    Run mysqltest with valgrind.

  • --valgrind-mysqltest-all

    Like --valgrind-mysqltest, but passes the --verbose and --show-reachable options to valgrind.

  • --valgrind-options=str

    Extra options to pass to valgrind.

  • --valgrind-path=path

    Specify the path name to the valgrind executable.

  • --vardir=path

    Specify the path where files generated during the test run are stored. The default location is ./var.

  • --view-protocol

    Pass the --view-protocol option to mysqltest.

  • --vs-config=config_val

    Specify the configuration used to build MySQL (for example, --vs-config=debug --vs-config=release). This option is for Windows only. It is available as of MySQL 4.1.23, 5.0.30, and 5.1.14.

  • --wait-timeout=N

    Unused?

  • --warnings

    Search the server log for errors or warning after each test and report any suspicious ones; if any are found, the test will be marked as failed. This is the default behavior, it may be turned off with --nowarnings.

  • --with-ndbcluster

    Use NDB Cluster and enable test cases that require it.

  • --with-ndbcluster-all

    Use NDB Cluster in all tests.

  • --with-ndbcluster-only

    Run only test cases that have ndb in their name.

  • --with-ndbcluster-slave

    Unknown.

  • --with-openssl

    This option is a synonym for --ssl.

Note

mysql-test-run supports the following options not supported by mysql-test-run.pl: --local, --local-master, --ndb-verbose, --ndb_mgm-extra-opts, --ndb_mgmd-extra-opts, --ndbd-extra-opts, --old-master, --purify, --use-old-data, --valgrind-mysqltest-all.

Conversely, mysql-test-run.pl supports the following options not supported by mysql-test-run: --benchdir, --check-testcases, --client-ddd, --client-debugger, --cursor-protocol, --debugger, --im-mysqld1-port, --im-mysqld2-port, --im-port, --manual-debug, --netware, --notimer, --reorder, --script-debug, --skip-im, --skip-ssl, --sp-protocol, --start-dirty, --suite, --suite-timeout, --testcase-timeout, --udiff, --unified-diff,, --valgrind-path, --vardir, --view-protocol.


Download this Manual
PDF (US Ltr) - 443.9Kb
PDF (A4) - 445.6Kb
EPUB - 101.9Kb