Test cases that do not perform a proper cleanup turn out to be a nightmare for test engineers or anyone who is tasked with the job of monitoring regression test results. It leads to test failures that are difficult to analyse and fix.
A good test tool should provide a mechanism which helps to detect such faulty tests. Our primary test tool mysql-test-run(MTR) already has an option 'check-testcases'(MTR documentation) which is enabled by default to perform this check. But this merely prints an informative message about missing cleanup and the test run succeeds.
First let us look at current behaviour of MTR with 'check-testcases' option. Consider a simple test 'sample.test' which creates a table, performs SELECT operation but misses dropping the table.
|
1 2 3 |
CREATE TABLE t1(c1 INT); INSERT INTO t1 VALUES (1), (2); SELECT * FROM t1; |
Let’s run this test using MTR.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 |
perl mysql-test-run.pl sample.test worker[1] Using MTR_BUILD_THREAD 300, with reserved ports 13000..13009 CREATE TABLE t1(c1 INT); INSERT INTO t1 VALUES (1), (2); SELECT * FROM t1; c1 1 2 main.sample [ pass ] 13 MTR's internal check of the test case 'main.sample' failed. This means that the test case does not preserve the state that existed before the test case was executed. Most likely the test case did not do a proper clean-up. It could also be caused by the previous test run by this thread, if the server wasn't restarted. This is the diff of the states of the servers before and after the test case was executed: tables_in_test +t1 tables_in_mysql mysql.column_stats mysql.columns_priv Check of testcase failed for: main.sample Completed: All 1 tests were successful. |
As you see from the above run, 'check-testcases' triggers only a warning about the missing cleanup and informs about the observed differences in the system states. Since MTR does not report it as a failure, it doesn’t get noticed in regression runs. As a consequence, new tests that do not perform proper cleanup are getting added to the repository.
To resolve this issue, we have introduced a new MTR option 'fail-check-testcases' to enforce strict cleanup at the end of the test run. When this option is enabled, the check-testcase failure gets reported as an error and the test run fails. This ensures that if a test does not perform a proper cleanup at the end, it is caught while running the regression test suite. Please note that this option is disabled by default.
Now, let us run the above test case with the option enabled.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 |
perl mysql-test-run.pl sample.test –fail-check-testcases worker[1] Using MTR_BUILD_THREAD 300, with reserved ports 13000..13009 CREATE TABLE t1(c1 INT); INSERT INTO t1 VALUES (1), (2); SELECT * FROM t1; c1 1 2 main.sample [ fail ] MTR's internal check of the test case 'main.sample' failed. This means that the test case does not preserve the state that existed before the test case was executed. Most likely the test case did not do a proper clean-up. It could also be caused by the previous test run by this thread, if the server wasn't restarted. This is the diff of the states of the servers before and after the test case was executed: tables_in_test +t1 tables_in_mysql mysql.column_stats mysql.columns_priv mysqltest: Result content mismatch not ok safe_process[7724]: Child process: 7725, exit: 1 Check of testcase failed for: main.sample Completed: Failed 1/1 tests, 0.00% were successful. Failing test(s): main.sample |
As you can see above, the test run has failed by reporting an error.
This option would be enabled by default in the near future to ensure any missing cleanup is caught while enhancing the test suite.
Please try out the new MTR option 'fail-check-testcases'( MTR documentation) and share your feedback in improving our test tool further.