WL#8036: Review usage of skip option in MTR tests

Affects: Server-8.0   —   Status: Complete 

MTR provides this option to skip tests if all pre-requisites of the test are not
met. If not used properly this can prove to be risky. We may end up with tests
that are never run because the conditions it needs are never met . 

In general, the principle shall be that it is acceptable to skip a test if the
conditions for running the test are not there. Example:

- Some tests can't be run in embedded mode
- Some tests can't be run with GTID mode on
- Some tests require certain OS

On the other hand, tests shall not be skipped if certain parts of the server
bundle cannot be found, certain server option needs to be set etc:

- Plugins that we ship
- Character sets that are part of our offering

In the case where a feature is offered only for the enterprise packages, the
test must determine whether it is running on an enterprise build and act
accordingly, not test for the presence/non-presence of a certain feature like a
plugin.

A part of this task is thus to go through all include/have_* files in
mysql-test/ and make sure that the tests fail (and don't skip) if the necessary
conditions are not met.

Note - this is an umbrella WL and actual implementation is done in the sub WLs

No documentation needed since this WL does not make any changes. There is
documentation for sub WLs.
FR1 - In regression test environment if conditions required for a test execution
is not met, the test shall fail instead of skipping
FR2 - There shall be a variable (--no-skip) to determine if the test should fail
or skip.  By default variable will be OFF and tests will skip. This is done so
that external users are not forced to run all tests
After evaluation of inc files we concluded that it is not possible to completely
do away with skip option. For eg: some tests can run only with a certain
binlog_format. Suite like rpl and binlog are run with all 3 formats and hence it
should be possible to skip for formats that are not supported by the test. 

This WL will attempt to remove skip wherever possible. Inc files used to
skip were divided into several categories and a strategy corresponding to each
is defined below.

Tests that require specific values for mysqld option - Neither skip nor fail.
Options to be set in the test or through opt files.
Tests that expect certain Compile options to be ON (eg; some charsets)- Fail.
PB2 binaries should be built with all options needed to run complete test suite
OS specific options - skip because it is impossible to run OS dependent tests on
all platforms (eg : have_lowercase.inc)
Checks for availability of plugins - fail. PB2 binaries should be built with all
plugins
Availability of 3rd party components - eg: have_util_unzip. Fail. Required utils
should be installed on PB2 machines
binlog format - Used by tests that can run only with certain binlog formats.
skip. This will get covered because rpl suite by default runs with all 3 binlog
options

We may also consider grouping together tests that have similar properties under
a separate suite. One example is GTID tests. Currently these are part of rpl
suite and the tests are skipped if suite is run with GTID OFF. This can lead to
reduced test coverage of GTID if tests are never run with GTID ON. 




List of inc files used for skipping tests
----------------------------------------------
./suite/opt_trace/include/filesort_pq.inc
./suite/rpl/include/rpl_mixed_ddl.inc
./suite/rpl/include/rpl_mixed_dml.inc
./suite/perfschema/include/no_protocol.inc
./suite/perfschema/include/upgrade_check.inc
./suite/engines/rr_trx/include/check_for_error_rollback_skip.inc
./extra/rpl_tests/rpl_mts_crash_safe.inc
./include/have_valgrind.inc
./include/have_innodb_4k.inc
./include/have_util_nc.inc
./include/not_var_link.inc
./include/have_mysql_no_login_plugin.inc
./include/have_federated_plugin.inc
./include/have_dynamic_loading.inc
./include/not_lowercase1.inc
./include/have_archive_plugin.inc
./include/have_slave_parallel_type_logical_clock.inc
./include/have_slave_repository_type_file.inc
./include/have_innochecksum_debug.inc
./include/have_blackhole_plugin.inc
./include/have_null_audit_plugin.inc
./include/mysqld--help.inc
./include/have_log_bin.inc
./include/not_crashrep.inc
./include/have_plugin_interface.inc
./include/not_gtid_enabled.inc
./include/have_plugin_server.inc
./include/have_simple_parser.inc
./include/have_undo_tablespaces.inc
./include/have_max_indexes_64.inc
./include/diff_servers.inc
./include/have_daemon_example_plugin.inc
./include/have_validate_password_plugin.inc
./include/have_QC_Disabled.inc
./include/have_example_plugin.inc
./include/have_semisync_plugin.inc
./include/have_innodb_16k.inc
./include/have_myisam.inc
./include/have_udf.inc
./include/have_slave_parallel_type_database.inc
./include/not_parallel.inc
./include/no_valgrind_without_big.inc
./include/have_innodb_8k.inc
./include/have_grep.inc
./include/have_expect.inc
./include/have_innodb.inc
./include/have_max_indexes_128.inc
./include/mysqlbinlog_have_debug.inc
./include/not_ndb_default.inc
./include/have_plugin_auth.inc
~



List of have_*.inc files in all suites

./include/have_utf32.inc
./include/have_innodb_16k.inc
./include/have_case_insensitive_file_system.inc
./include/have_geometry.inc
./include/have_32bit.inc
./include/have_koi8r.inc
./include/have_max_indexes_128.inc
./include/have_slave_parallel_type_logical_clock.inc
./include/have_binlog_format_statement.inc
./include/have_innodb_8k.inc
./include/have_ndb.inc
./include/have_local_infile.inc
./include/have_archive_plugin.inc
./include/have_utf16.inc
./include/have_ipv6.inc
./include/have_64bit.inc
./include/have_validate_password_plugin.inc
./include/have_partition.inc
./include/have_profiling.inc
./include/have_ssl_communication.inc
./include/have_euckr.inc
./include/have_ndb_debug.inc
./include/have_binlog_checksum_off.inc
./include/have_innodb_4k.inc
./include/have_undo_tablespaces.inc
./include/have_semijoin.inc
./include/have_plugin_interface.inc
./include/have_daemon_example_plugin.inc
./include/have_myisam.inc
./include/have_max_indexes_64.inc
./include/have_lowercase0.inc
./include/have_QC_Disabled.inc
./include/have_binlog_format_row.inc
./include/have_gbk.inc
./include/have_partition_open_file_limit.inc
./include/have_archive.inc
./include/have_federated_plugin.inc
./include/have_ndb_extra.inc
./include/have_expect.inc
./include/have_binlog_format_mixed_or_row.inc
./include/have_latin2_ch.inc
./include/have_gtid.inc
./include/have_perfschema.inc
./include/have_not_innodb_plugin.inc
./include/have_udf.inc
./include/have_index_condition_pushdown.inc
./include/have_outfile.inc
./include/have_openssl.inc
./include/have_firstmatch.inc
./include/have_crypt.inc
./include/have_exampledb.inc
./include/have_nodebug.inc
./include/have_innodb.inc
./include/have_no_undo_tablespaces.inc
./include/have_mysql_upgrade.inc
./include/have_ssl.inc
./include/have_materialization.inc
./include/have_lowercase2.inc
./include/have_query_cache_disabled.inc
./include/have_dynamic_loading.inc
./include/have_ucs2.inc
./include/have_null_audit_plugin.inc
./include/have_grep.inc
./include/have_blackhole_plugin.inc
./include/have_query_cache.inc
./include/have_sha256_rsa_auth.inc
./include/have_binlog_format_mixed.inc
./include/have_ipv4_mapped.inc
./include/have_ndbapi_examples.inc
./include/have_loosescan.inc
./include/have_optimizer_trace.inc
./include/have_big5.inc
./include/have_binlog_format_mixed_or_statement.inc
./include/have_innochecksum_debug.inc
./include/have_ssl_crypto_functs.inc
./include/have_debug_sync.inc
./include/have_valgrind.inc
./include/have_slave_parallel_type_database.inc
./include/have_engine_condition_pushdown.inc
./include/have_lowercase1.inc
./include/have_gb2312.inc
./include/have_plugin_auth.inc
./include/have_simple_parser.inc
./include/have_compress.inc
./include/have_ujis.inc
./include/have_cp1251.inc
./include/have_symlink.inc
./include/have_eucjpms.inc
./include/have_mrr.inc
./include/have_gb18030.inc
./include/have_mysql_no_login_plugin.inc
./include/have_plugin_server.inc
./include/have_cp1250_ch.inc
./include/have_utf8mb4.inc
./include/have_utf8.inc
./include/have_example_plugin.inc
./include/have_util_nc.inc
./include/have_log_bin.inc
./include/have_sjis.inc
./include/have_multi_ndb.inc
./include/have_myisam_tmp_storage_engine.inc
./include/have_blackhole.inc
./include/have_binlog_format_row_or_statement.inc
./include/have_cp866.inc
./include/have_cp932.inc
./include/have_slave_repository_type_file.inc
./include/have_memcached_plugin.inc
./include/have_semisync_plugin.inc
./include/have_case_sensitive_file_system.inc
./include/have_statement_timeout.inc
./include/have_debug.inc
./include/have_tis620.inc
./suite/ndb/include/have_java.inc
./suite/ndb/include/have_clusterj.inc
./suite/ndb/include/have_clusterj_jpa.inc
./suite/ndb/include/have_ndb_rqg.inc
./suite/ndb/include/have_ndbjtie_junit.inc
./suite/ndb/include/have_connectorj.inc
./suite/ndb/include/have_little_endian.inc
./suite/ndb/include/have_openjpa.inc
./suite/ndb/t/have_ndb_dist_priv.inc
./suite/ndb/t/have_ndb_error_insert.inc
./suite/ndb/t/have_ndbinfo.inc
./suite/perfschema/include/have_aligned_memory.inc
./suite/ndb_memcache/include/have_memcache.inc
./suite/auth_sec/include/have_ssl_communication.inc
./suite/auth_sec/include/have_openssl.inc
./suite/auth_sec/include/have_ssl.inc
./suite/auth_sec/include/have_sha256_rsa_auth.inc
./suite/federated/include/have_federated_db.inc
./suite/sys_vars/inc/have_open_file_limit_more_than_5000.inc


Copyright (c) 2000, 2024, Oracle Corporation and/or its affiliates. All rights reserved.