WL#9073: Add an MTR option --no-skip to force all the tests to run
Affects: Server-8.0 — Status: Complete — Priority: Medium
Problem description: 1. In MTR, tests are skipped if all pre-requisites of the test are not met. This can be risky because we may end up with tests that are never run because the conditions are never met in the regression test environment. for ex - In case of a missing plugin or a utility on which the test depends, the test will skip. In these cases, the skip behaviour is incorrect because a genuine dependency miss will go undetected in MTR test runs. 2. Second part of the task is to identify the tests for which skip is the correct behaviour. These tests should continue to skip to avoid unnecessary failures. for ex - a. OS dependent files. b. Different binlog formats. - Since replication suite same test with all 3 binlog formats. Some tests cannot run with some formats and should be skipped c. Different compile options. 3. Third part of the task is, we do not want to change user experience. Proposed solution: 1. A new MTR option (--no-skip) will be introduced. This option will allow each MTR test to run without any skip. This will help us solve the first problem stated. 2. An excluded list will be maintained to keep track of tests which should continue to skip to avoid unwanted failures. This will help us solve the second problem stated. 3. The default value of this introduced variable will be OFF. This implies that users will not be forced to run all the tests and they will not have to deal with unwanted failures due to missing dependencies. To use/not-use this new option will be at the user's disposal.
FR1 : There shall be an MTR option , --no-skip that prevents tests from skipping when needed dependencies are not met FR2 : There shall be an option to specify a list of inc files and tests that shall continue to be skipped even if --no-skip option is specified. This is needed to check for conditions (eg: OS dependency) that are impossible to satisfy always. FR3 : The --no-skip option must be OFF by default and tests shall be skipped when dependencies are not met
WL#9027: Remove usage of skip from tests(except rpl and binlog) that need specific mysqld options to run
An exclude list of include(.inc) and test(.test) files is created and kept at the location include/excludedList.list Based on the excluded list, following tests will continue to skip: 1. All tests which will source the include files mentioned in the excluded list. 2. All tests which are mentioned in the excluded list. The exclude list needs to be updated in future in case a new MTR test is introduced which we would like to skip depending on a newly introduced condition. Below is the current excluded list: /* OS dependent files*/ Reason for inclusion: Below files are OS specific and tests will run only on selective platform where conditions are met and skip on others. 1. have_32bit.inc 2. have_64bit.inc 3. have_case_insensitive_file_system.inc 4. have_case_sensitive_file_system.inc 5. have_crypt.inc /*supported on Linux platform */ 6. have_lowercase0.inc 7. have_lowercase1.inc 8. have_lowercase2.inc 9. linux.inc 10. windows.inc 11. not_windows.inc 12. have_open_file_limit_more_than_5000.inc 13. have_partition_open_file_limit.inc 14. table_compress_2.test /*checks for DATADIR that supports hole punching*/ 15. table_compress_3.test /*checks for DATADIR that supports hole punching*/ /* Client uses the posix locale as default client character set.*/ /* Older unices have the whole collection by default. Also el6 and el7 where test runs fine. But el8 is like modern ubuntu, with just a small set of utf8 locales. On Ubuntu, there is a list of supported locales (/usr/share/i18n/SUPPORTED). From this list, new locales may be generated with a locale generator (locale-gen). The reason for this change is that the whole collection of locales have gotten pretty huge (and is still growing) and increases the size of glibc The reason for retaining this inc file in the excluded list is that new Unix- installations may not have the locales needed for the test and without the inc- file the test will fail. Also, the test is not supported on Windows. */ 16. have_posix_locale.inc /* Binlog formats */ Reason for inclusion: Below files are included because rpl suite runs each test with 3 different binlog format combinations. The test may not be meant to run on all binlog formats and will skip. 17. have_binlog_format_mixed_or_row.inc 18. have_binlog_format_mixed_or_statement.inc 19. have_binlog_format_mixed.inc 20. have_binlog_format_row_or_statement.inc 21. have_binlog_format_row.inc 22. have_binlog_format_statement.inc /* Compile options */ Reason for inclusion: Below files are included because few tests are meant to be tested with specific compile options which is not available in all builds. /* Test requires --debug-server option*/ 23. have_debug_sync.inc 24. have_debug.inc 25. have_nodebug.inc 26. mysql_have_debug.inc /*have_log_bin.inc will be set using an opt file with option (--log-bin). The inc file is removed from the excluded list. There will be no duplicate runs because the usage of have_log_bin.inc is not eliminated from the tests due to internal MTR dependency on this inc file being sourced. */ /*not_log_bin.inc can be set using an opt file with option (--skip-log-bin) and usage of this inc can be removed from the test. However, this will cause the same tests to run again when --log-bin option will be used to run the suites. This will cause duplicate test runs leading to increased test execution time.Removal of not_log_bin.inc does not come in the scope of this worklog and will be done seperately.*/ 27. not_log_bin.inc /* Test is currently running on Dev build linux-x86-64bit-devbld */ 28. mysqlbinlog_have_debug.inc 29. have_innochecksum_debug.inc /*Test with yassl is supported on community build */ 30. have_yassl.inc /*Test with openssl is supported on commercial build */ 31. have_openssl.inc 32. have_openssl_binary.inc /*checks for openssl binary */ /* variable have_ssl is compiled default YES but in case server is started with --skip-ssl or if the test is run with invalid certificates; in that case have_ssl value would be NO or DISABLED */ 33. have_ssl_communication.inc /*checks for have_ssl value to be YES*/ 34. have_ssl_crypto_functs.inc /*checks for have_ssl value to be YES|DISABLED*/ /* Innodb page size */ Reason: Page sizes 4K,8K,16K,32K,64K is set within the test using a custom bootstrap file. However, below is a special case which can run with 4K,8K and 16K at the same time and hence required to skip tests on 32K and 64K 35. have_innodb_max_16k.inc /* IP versions */ 36. have_ipv4_mapped.inc /* System variable which checks whether server has statement_timeout feature or not.The value can be NO if the background thread used by this feature could not be initialized during server start */ 37. have_statement_timeout.inc /*Miscellaneous */ Reason for inclusion: Below .inc files are included because of the need of running the test with different protocols, different builds. 38. have_ps_protocol.inc 39. have_no_protocol.inc 40. no_view_protocol.inc 41. no_sp_protocol.inc 42. no_opt_trace_protocol.inc 43. no_explain_protocol.inc 44. no_cursor_protocol.inc 45. have_embedded.inc 46. not_embedded.inc 47. have_mysqld_safe.inc 48. have_valgrind.inc /* Few skipped test-cases */ Reason for inclusion: Few tests have custom skip logic written within the test and needs to be skipped. /* checks for mysql_embedded executables to be present */ 49. mysql_embedded.test
When an MTR test is run with (--no-skip) option, MTR (mysql-test-run.pl) will pass the --no-skip option & the excluded list of include files to the mysql client (mysqltest.cc) The mysql client will read the test file and also the excluded include files and based on the match or no-match take the decision to skip or run the test. In case --no-skip option is not passed, the excluded list will also be ignored by MTR (mysql-test-run.pl). The test will run as per current behaviour. Below are the list of plugins that will be benefited by this implementation. have_archive_plugin.inc have_audit_log_plugin.inc have_blackhole_plugin.inc have_blackhole.inc have_daemon_example_plugin.inc have_example_plugin.inc have_federated_plugin.inc have_group_replication_plugin.inc have_mecab.inc have_memcached_plugin.inc have_mysql_no_login_plugin.inc have_ngram.inc have_not_innodb_plugin.inc have_null_audit_plugin.inc have_partition.inc have_plugin_auth.inc have_plugin_interface.inc have_plugin_server.inc have_simple_parser.inc have_udf.inc have_validate_password_plugin.inc
Copyright (c) 2000, 2019, Oracle Corporation and/or its affiliates. All rights reserved.