WL#14194: Replace old terms in replication system variables, options, and strings
Affects: Server-8.0 — Status: Complete
EXECUTIVE SUMMARY ================= Replace the terms "master", "slave", and "mts" by "source", "replica", and "mta" in system variable names, status variables, SQL functions, command-line options (for server and all tools), semisync plugin names and file names, C++ filenames and header guards, debug symbols, help texts, and in performance_schema instrumentation names for locks, condition variables, threads, execution stages, allocators, and thread commands. Create aliases so that users can use the old names for system variables, SQL functions, and command-line options, and deprecate the use of those aliases. USER STORIES ============ U1. As a MySQL admin user: - I want to use configuration that does not contain offensive terms, - so that I do not get offended by them and/or expose them to other people (e.g., customers of my application built on top of MySQL). U2. As a MySQL admin user: - I want to monitor the product without seeing offensive terms, - so that I do not get offended by them and/or expose them to other people (e.g., customers of my application built on top of MySQL). SCOPE ===== This worklog is part of a broader effort to change old terminology across the entire MySQL product line. This worklog focuses only on changing *replication* terminology. For the word "master", we only change it where it appears in a replication context; we do not change "master key" or "master thread". We only change status variables, SQL functions, command-line options, semisync plugin names and filenames, C++ filenames and header guards, debug symbols, help texts, and performance_schema instrumentation names. (So we do not change privilege names, column names or other metadata for system tables, error messages, ER_* symbols, or SQL commands.) LIMITATIONS =========== We do not change identifiers related to NDB. We do not change SQL printed by mysqldump or issued by mysqladmin, MEB, or other external tools except mysqlbinlog. The identifiers we change, are not changed when they appear in error messages. REFERENCES ========== - https://mysqlhighavailability.com/mysql-terminology-updates/
FR1. The words "master", "slave", and "mts" shall be replaced by "source", "replica", and "mta", respectively, in the contexts listed below, except where "master" occurs in a non-replication context such as "master key" or "master thread". Where possible, we keep the old names for backward compatibility, as indicated in each item in the list. Where possible the old names generate deprecation warnings, as indicated in each item in the list: FR1.1. All command-line options for mysqld, except those that will be deprecated in WL#14526, and those that are defined by NDB. The old names shall remain, and using them shall generate a deprecation warning in the error log. FR1.2. All command-line options for mysqladmin. The old names shall remain and make the tool print deprecation warnings. FR1.3. All command-line options for mysqlbinlog. The old names shall remain and make the tool print deprecation warnings. FR1.4. All command-line options for mysqldump. The old names shall remain and make the tool print deprecation warnings. FR1.5. All system variables, except those that have been deprecated already. The old names shall remain. The use of them in SQL statement shall make the client return a deprecation warning. The use of them on the command-line or configuration file shall generate a deprecation warning in the error log. The following shall hold for persisted variables: FR1.5.1. When the server reads persisted variables, it shall accept either the old name, or the new name, or both. FR1.5.2. When the server writes persisted variables, it shall persist both the old and the new name, even if only one of them was specified in a SET statement. FR1.5.3. When RESET PERSIST is used with a renamed system variable, regardless of using the old or new name in the statement: - The old name shall be removed if it is persisted; - The new name shall be removed if it is persisted; - A warning/error shall only be generated if neither the old nor the new name was persisted. Both the old and the new variables shall exist in the following tables and commands, but no deprecation warning shall be issued while reading them: performance_schema.global_variables performance_schema.session_variables performance_schema.variables_by_thread performance_schema.persisted_variables performance_schema.variables_info SHOW VARIABLES FR1.6. All status variables defined in the server, except those defined by NDB. Both the old and the new variables shall exist in the following places, and no deprecation warning shall be generated when reading them: performance_schema.global_status performance_schema.session_status performance_schema.status_by_thread SHOW STATUS FR1.7. All user variables that the replica sets on the connection before it sends a COM_BINLOG_DUMP command to start replication. FR1.7.1. The source shall accept either the old or the new names. FR1.7.2. The replica shall set both the old-named and the new-named variable. No deprecation warning shall be generated. FR1.8. All SQL functions defined in the server. The old names shall remain and the use of them shall make the client return deprecation warnings. FR1.9. The following identifiers instrumented in performance_schema: FR1.9.1. Thread commands FR1.9.2. Instrumented locks FR1.9.3. Instrumented condition variables FR1.9.4. Thread stages FR1.9.5. Thread names FR1.9.6. Instrumented memory alloctions The old names shall be removed. FR1.10. All C++ source file names. The old names shall be removed. FR1.11. All header guards in C++ source filenames. The old names shall be removed. FR1.12. All debug symbols (as in `DBUG_EXECUTE_IF("debug_symbol", ...)`). The old names shall be removed. FR1.13. The following names related to the two semisync libraries: FR1.13.1. The library filenames. FR1.13.2. The plugin names. FR1.13.2. All system variables defined by the plugins. FR1.13.2. All status variables defined by the plugins. The libraries shall be duplicated so that old filenames can be used with the old plugin names, and loading them shall define the old-named system variables and status variables. Deprecation warnings shall be generated when loading the old-named libraries. An error shall be generated if the user loads both the old-named and the new-named source libraries, or both the old-named and the new-named replica libraries. FR1.14. Compiled-in help texts for system variables and command-line options: - Any occurrence of renamed identifiers shall use the new name. - Any occurrence of master/slave/mts shall be replaced by source/replica/mta. - Any mistakes spotted while editing the text shall be fixed. - Any deprecated options shall be documented as deprecated, with a reference to the new option.
SUMMARY OF THE APPROACH ======================= Replace the old terms by the new terms in all the required places. Where possible, maintain the old terms as aliases. This is possible for configuration and other things that appear as identifiers in SQL, but not for strings that appear in performance_schema tables, so we break compatibility for them. This needs to change more than 16000 files. Therefore, we perform the replacements automatically. To make that practical, we use a tool that reads a file containing replacement rules and applies each rule to the full source tree. The tool needs to express things in different ways than what standard tools like grep can, so we code a new tool for this. After applying the replacements, we code the aliases in a sequence of follow-up steps. CHANGES IN USER INTERFACE ========================= Here we list the full set of replaced identifiers. Unless otherwise specified, we uniformly replace "master" by "source", "slave" by "replica", and "mts" by "mta". * Changed system variables defined in the server: For the variables below, the old name remains and is deprecated, and the new name is added. Both names will exist in all contexts where variables appear, i.e., --command-line-option, @@global.variable, @@session.variable, SET PERSIST, SHOW VARIABLES, performance_schema.global_variables, performance_schema.persisted_variables, performance_schema.variables_info. Deprecation warnings are emitted when using the variable on the command line, in a SET statement, and when accessing it using @@ notation. No warning is emitted when the variable is accessed through a table or SHOW statement. The built-in help for the old name states that it is deprecated. master_verify_checksum sync_master_info init_slave rpl_stop_slave_timeout log_slow_slave_statements slave_max_allowed_packet slave_compressed_protocol slave_exec_mode slave_type_conversions slave_sql_verify_checksum slave_parallel_type slave_preserve_commit_order log_slave_updates slave_allow_batching slave_load_tmpdir slave_net_timeout sql_slave_skip_counter slave_skip_errors slave_checkpoint_period slave_checkpoint_group slave_transaction_retries slave_parallel_workers slave_pending_jobs_size_max pseudo_slave_mode skip_slave_start The following variables are not changed: - ndb_slave_conflict_role: We do not change NDB now. - binlog_rotate_encryption_master_key_at_startup: This refers to the "master key", which is a metaphor without offensive connotations. This use of master is considered unproblematic and will not be replaced. - slave_rows_search_algorithms: Deprecated already - master_info_repository: Deprecated already. * Changed command-line options for mysqld: For the following option, the old name remains and is deprecated, and the new name is added. A deprecation warning is issued when the command-line option is used. The built-in help for the old name states that it is deprecated. (Most command-line options are also system variables, and are included in the list in the preceding section instead. The list here includes only the command-line options which are not system variables.) show-slave-auth-info The following command-line options are not changed: - abort-slave-event-count: Will be deprecated in WL#14526. - disconnect-slave-event-count: Will be deprecated in WL#14526. - master-info-file: Deprecated already. - master-retry-count: Deprecated already. * Changed command-line options for mysqlbinlog: For the following option, the old name remains and is deprecated, and the new name is added. A deprecation warning is issued when the command-line option is used. The built-in help for the old name states that it is deprecated. read-from-remote-master The following variable is not changed: - stop-never-slave-server-id: Deprecated already. * Changed command-line options for mysqldump: For the following options, the old name remains and is deprecated, and the new name is added. A deprecation warning is issued when the command-line option is used. The built-in help for the old name states that it is deprecated. apply-slave-statements delete-master-logs dump-slave include-master-host-port master-data * Changed command-line options for mysqladmin: For the following options, the old name remains and is deprecated, and the new name is added. A deprecation warning is issued when the command-line option is used. The built-in help for the old name states that it is deprecated. start-slave stop-slave * Changed built-in SQL functions: For the following function, the old name remains and is deprecated, and the new name is added. A deprecation warning is issued when the function is used. MASTER_POS_WAIT * Changed status variables: Status variables are displayed with SHOW STATUS LIKE 'name' and performance_schema.status_variables. The old name remains and the new name is added. No deprecation warning is used. Slave_open_temp_tables Slave_rows_last_search_algorithm_used Rpl_semi_sync_master_status Rpl_semi_sync_master_clients Rpl_semi_sync_master_yes_tx Rpl_semi_sync_master_no_tx Rpl_semi_sync_master_wait_sessions Rpl_semi_sync_master_no_times Rpl_semi_sync_master_timefunc_failures Rpl_semi_sync_master_wait_pos_backtraverse Rpl_semi_sync_master_tx_wait_time Rpl_semi_sync_master_tx_waits Rpl_semi_sync_master_tx_avg_wait_time Rpl_semi_sync_master_net_wait_time Rpl_semi_sync_master_net_waits Rpl_semi_sync_master_net_avg_wait_time Rpl_semi_sync_slave_status All NDB-related status variables are skipped. The following variables were already removed in 8.0: Slave_running Slave_retried_transactions Slave_heartbeat_period Slave_received_heartbeats Slave_last_heartbeat (There is some dead server code that makes it look like they exist, if you read the code. Hence they were listed in an early version of this worklog. We do not consider them in this worklog, and will remove them in an independent bugfix.) * Changed thread names in PERFORMANCE_SCHEMA: Thread names are visible in the NAME column of performance_schema.threads. The old name is replaced by the new name. This is an incompatible change. slave_io slave_sql slave_worker * Changed thread commands in PERFORMANCE_SCHEMA: Thread command is the value shown in the following places: - The PROCESSLIST_COMMAND column of performance_schema.threads, - the COMMAND column of performance_schema.processlist, - the COMMAND column of SHOW PROCESSLIST, - the COMMAND column of INFORMATION_SCHEMA.PROCESSLIST. In the following case, the old name is replaced by the new name. This is an incompatible change. Register Slave * Changed names of instrumented locks in PERFORMANCE_SCHEMA: Mutex names are visible with a 'wait/synch/mutex/' prefix, in the following places: - the NAME column of performance_schema.mutex_instances - the EVENT_NAME column of all performance_schema.events_waits_* tables In the following cases, the old name is replaced by the new name. This is an incompatible change. Master_info::data_lock Master_info::run_lock Master_info::sleep_lock Master_info::info_thd_lock Master_info::rotate_lock Slave_reporting_capability::err_lock Relay_log_info::slave_worker_hash_lock key_mts_temp_table_LOCK key_mts_gaq_LOCK LOCK_slave_list * Changed names of instrumented condition variables in PERFORMANCE_SCHEMA: Condition variable names are visible with a 'wait/synch/cond/' prefix in the following places: - the NAME column of performance_schema.cond_instances. - the EVENT_NAME column of all performance_schema.events_waits_* tables In the following cases, the old name is replaced by the new name. This is an incompatible change. Master_info::data_cond Master_info::start_cond Master_info::stop_cond Master_info::sleep_cond Master_info::rotate_cond LOCK_slave_trans_dep_tracker Relay_log_info::mts_gaq_cond * Changed names of thread stages in PERFORMANCE_SCHEMA: Thread stages are visible in the following places: - the PROCESSLIST_STATE column of performance_schema.threads, and - the EVENT_NAME column of all performance_schema.events_stages_* tables, and - the STATE column of performance_schema.processlist, and - the STATE column of SHOW PROCESSLIST, and - the STATE column of INFORMATION_SCHEMA.PROCESSLIST. In the following cases, the old name is replaced by the new name. This is an incompatible change. Checking master version Connecting to master Flushing relay log and master info repository. Queueing master event to the relay log Waiting until MASTER_DELAY seconds after master executed event Waiting for master to send event Waiting for master update Killing slave Master has sent all binlog to slave; waiting for more updates Registering slave on master Sending binlog event to slave Slave has read all relay log; waiting for more updates Waiting for slave workers to process their queues Waiting for Slave Worker queue Waiting for Slave Workers to free pending events Waiting for Slave Worker to release partition Waiting for slave mutex on exit Waiting for slave thread to start Waiting for the slave SQL thread to advance position The following thread stages are defined in the old semi-sync plugin. The new name is used in the new semi-sync plugin. This is a *compatible* change, since users must enable the new library in order to observe the new names. Waiting for semi-sync ACK from slave Waiting for semi-sync slave connection Reading semi-sync ACK from slave The following thread stages are changed in more complex way than the scripted word substitution: Waiting for the slave SQL thread to free enough relay log space Waiting for the replica SQL thread to free relay log space - The old text is exactly the maximum allowed length, and replacing 'slave' by 'replica' makes it one character too long. Hence removing the redundant word 'enough'. Changing master Changing replication source - This stage referred to the SQL statement CHANGE MASTER, which has been renamed to CHANGE REPLICATION SOURCE, so the renamed stage shall use the new SQL command name. * Changed instrumented memory allocations: Slave_job_group::group_relay_log_name rpl_slave::check_temp_dir SLAVE_INFO show_slave_status_io_gtid_set memory/sql/Relay_log_info::mts_coor The old name is replaced by the new name. This is an incompatible change. * Changed C++ filenames: rpl_slave.cc rpl_slave.h rpl_slave_until_options.cc rpl_slave_until_options.h rpl_slave_commit_order_manager.cc rpl_slave_commit_order_manager.h rpl_master.cc rpl_master.h rpl_mts_submode.cc rpl_mts_submode.h semisync_master.cc semisync_master.h semisync_master_ack_receiver.cc semisync_master_ack_receiver.h semisync_master_plugin.cc semisync_master_socket_listener.h semisync_slave.cc semisync_slave.h semisync_slave_plugin.cc This is an internal change and does not change the API. Compatibility is not an issue. * Changed C++ header guards: RPL_SLAVE_H DEFINED_RPL_SLAVE_UNTIL_OPTIONS_H RPL_SLAVE_COMMIT_ORDER_MANAGER RPL_MASTER_H_INCLUDED SEMISYNC_MASTER_H SEMISYNC_MASTER_ACK_RECEIVER_DEFINED SEMISYNC_MASTER_SOCKET_LISTENER SEMISYNC_SLAVE_H MTS_SUBMODE_H This is an internal change and does not change the API. Compatibility is not an issue. * Changed C++ debug symbols: after_locking_channel_map_in_start_slave after_start_slave begin_master_pos_wait begin_start_slave check_slave_debug_group check_slave_master_info dbug.before_get_MASTER_UUID dbug.mts.force_clock_diff_eq_0 dbug.return_null_MASTER_UUID debug_crash_slave_time_out enable_mts_wokrer_failure_in_recovery_finalize enable_mts_worker_failure_init fail_generating_rotate_event_on_write_rotate_to_master_pos fail_to_flush_master_info fake_5_5_version_slave fault_injection_get_slave_worker fix_slave_net_timeout force_rollback_in_slave_on_transactional_ddl_commit FORCE_SLAVE_TO_RECONNECT_DUMP FORCE_SLAVE_TO_RECONNECT_EVENT FORCE_SLAVE_TO_RECONNECT_REG get_master_server_id. get_master_server_id.ER_NET_READ_INTERRUPTED mts_checkpoint mts_checkpoint_end mts_checkpoint_start mts_debug_concurrent_access mts_debug_recovery_reset_fails mts_debug_reset_workers_fails mts_distribute_round_robin mts_slave_worker_init_at_gaps_fails mts_worker_thread_fails mts_worker_thread_init_fails ndb_slave_fail_marking_epoch_committed old_row_based_repl_4_byte_map_id_master query_log_event_mts_corrupt_db_names remove_slave_load_file_before_write request_master_log_pos_3 rotate_slave_debug_group rpl_semisync_master_commit_trx_before_lock rpl_semisync_simulate_add_slave_failure rpl_slave_debug rpl_slave_debug_point signal.get_master_uuid signal.got_stop_slave simulate_hold_run_locks_on_stop_slave simulate_io_slave_error_on_init simulate_master_has_gtid_mode_off_something simulate_master_has_gtid_mode_on_something simulate_master_has_unknown_gtid_mode simulate_register_slave_killed simulate_slave_delay_at_terminate_bug38694 simulate_slave_unaware_checksum simulate_slave_unaware_gtid simulate_sql_slave_error_on_init simulate_stop_when_mts_in_group slave_acquired_backup_lock slave_crash_after_commit slave_crash_after_commit_no_atomic_ddl slave_crash_before_commit slave_reconnect_with_gtid_set_executed slave_worker_ends_group_before_signal_lwm stop_slave_dont_release_backup_lock stop_slave_middle_group stop_when_mts_in_group undefined_algorithm_on_slave uninitialized_master-info_structure unrecognized_master_version This is an internal change and does not change the API. Compatibility is not an issue. * Changed plugin library filenames: semisync_slave.so semisync_master.so The old-named file remains and is deprecated, and the new-named file is added. A deprecation warning is issued when loading the old-named file. * Changed plugin names: rpl_semi_sync_slave rpl_semi_sync_master The old name exists only in the old-named file, and the new name exists only in the new file. A deprecation warning is issued when loading the old-named file. * Changed system variables defined in the semisync plugins: rpl_semi_sync_master_wait_for_slave_count rpl_semi_sync_master_wait_no_slave rpl_semi_sync_slave_enabled rpl_semi_sync_slave_trace_level rpl_semi_sync_master_wait_for_slave_count rpl_semi_sync_master_wait_no_slave rpl_semi_sync_master_enabled rpl_semi_sync_master_timeout rpl_semi_sync_master_trace_level rpl_semi_sync_master_wait_point The old name exists only in the old-named plugin, and the new name exists only in the new-named plugin. A deprecation warning is issued when loading the old-named plugin. * Changed status variables defined in the semisync plugins: Rpl_semi_sync_slave_enabled Rpl_semi_sync_master_status Rpl_semi_sync_master_clients Rpl_semi_sync_master_yes_tx Rpl_semi_sync_master_no_tx Rpl_semi_sync_master_wait_sessions Rpl_semi_sync_master_no_times Rpl_semi_sync_master_timefunc_failures Rpl_semi_sync_master_wait_pos_backtraverse Rpl_semi_sync_master_tx_wait_time Rpl_semi_sync_master_tx_waits Rpl_semi_sync_master_tx_avg_wait_time Rpl_semi_sync_master_net_wait_time Rpl_semi_sync_master_net_waits Rpl_semi_sync_master_net_avg_wait_time The old name exists only in the old-named plugin, and the new name exists only in the new-named plugin. A deprecation warning is issued when loading the old-named plugin. * Changed user variables recognized by dump threads: (When replica connects to source, it sets some user variables on the connection before issuing the COM_BINLOG_DUMP[_GTID] command.) master_heartbeat_period master_binlog_checksum slave_uuid rpl_semi_sync_slave This is an internal change only and compatibility is not an isuse. * Changed cloud configuration: In internal/cloud/mysql-test/config/cloud.cnf, we use the new library in the following line: plugin-load-add="connection_control.so;semisync_master.so;thread_pool.so" * Changed packaging definitions: In all package definition files, we retain the existing semisync libraries and add the new ones, so that they all get included in packages. * Changed help texts: The help texts are too many to list here. Generally, we replace the old terms by the new terms, and additionally rewrite the help texts a bit where they can be improved. See the source patch for the full list of changes. DEPLOYMENT AND INSTALLATION =========================== This adds two new files: semisync_source.so and semisync_replica.so. They shall both be included in distributions. The existing semisync_master.so and semisync_replica.so shall remain included in distributions, too. SECURITY ======== No security implications. PROCEDURE TO MIGRATE APPLICATIONS TO NEW TERMINOLOGY ==================================================== If needed, we may publish the scripts that update the identifiers, to aid in migration. PROCEDURE TO MERGE UP FROM 5.7 ============================== After this worklog, patches written in 5.7 and merged to 8.0 may need to rename identifiers. There are several cases: 1. New files, for example test, which use old identifiers. These can be fixed by running a script after the merge. 2. Modified files, which add new lines that use old identifiers. These can be fixed by running a script after the merge. 3. Modified files, which modify existing lines that use old identifiers. These will generate merge conflicts. It seems hard to use scripts to resolve the conflicts, so this will likely have to be handled manually. PROTOCOL ======== Existing behavior ----------------- When an asynchronous replica connects to a source, it sets the following user variables. These are user variables (not system variables), which the dump thread interprets: @master_heartbeat_period @master_binlog_checksum @slave_uuid When a semi-synchronous replica connects, it checks if the source has set the system variable @@global.rpl_semi_sync_source_enabled, and also sets the user variable @rpl_semi_sync_slave, in addition to the three user variables above. New behavior ------------ We make the above logic more flexible, so that: - For all the three user variables used in asynchronous replication, the replica sets *both* the new-named variable and the old-named variable when it connects. - For the system variable checked in semi-synchronous replication, the replica checks if *either* the new-named variable or the old-named variable has been set. - For the user variable set in semi-synchronous replication, the replica sets *both* the new-named variable and the new-named variable. CROSS-VERSION REPLICATION ========================= Asynchronous replication ------------------------ The changes in the protocol imply the following for *asynchronous* cross-version replication. Here, OLD refers to a version before this worklog. NEW refers to a version having this worklog. FUTURE refers to a future version where the old terminology has been removed. OLD->NEW: Works because NEW replica sets old-named variables. NEW->OLD: Works because NEW source accepts old-named variables. NEW->FUTURE: Works because NEW source accepts new-named variables. FUTURE->NEW: Work because NEW replica sets new-named variables. OLD->FUTURE: Unsupported. Does not work beacuse FUTURE sets only new-named variables and OLD accepts only old-named variables. FUTURE->OLD: Unsupported. Does not work beacuse FUTURE sets only new-named variables and OLD accepts only old-named variables. Semi-synchronous replication ---------------------------- The changes in the protocol, and the way we alias semisync libraries, imply the following for *semi-synchronous* replication. In the following, we assume all servers install a semisync library. On NEW, there are two semisync libraries available, so we use NEW(old_semi) to refer to a server having this worklog, on which the user has installed the old-named semisync library, and we use NEW(new_semi) to refer to a server having this worklog, on which the user has installed the new-named semisync library. When a NEW server talks with an OLD server, we require that NEW uses old_semi, and when NEW talks with FUTURE, we requrie that NEW uses new_semi. The cases are as follows: OLD->NEW(old_semi): Works because NEW(old_semi) recognizes old-named system variable and sets new-named user variable. OLD->NEW(new_semi): Unsupported: user should upgrade all servers before using new_semi on NEW. Despite that, this works, because NEW(new_semi) recognizes old-named system variable and sets old-named user variable. NEW(old_semi)->OLD: Works beacuse NEW(old_semi) exports old-named system variable and recognizes old-named user variable. NEW(new_semi)->OLD: Unsupported: user should upgrade all servers before using new_semi. Does not work because OLD expects old-named system variable rpl_semi_sync_source_enabled to be defined, which NEW(new_semi) does not define. NEW(old_semi)->NEW(new_semi): Works because NEW(new_semi) recognizes old-named system variable and sets old-named user variable. NEW(new_semi)->NEW(old_semi): Works because NEW(old_semi) recognizes new-named system variable and sets new-named user variable. NEW(old_semi)->FUTURE: Unsupported: user should enable new_semi before upgrading to FUTURE. Does not work because FUTURE expects new-named system variable, which NEW(old_semi) does not define. NEW(new_semi)->FUTURE: Works because NEW(new_semi) defines new-named system variables and accepts new-named user variables. FUTURE->NEW(old_semi): Unsupported: user should enable new_semi before upgrading to FUTURE. Despite that, this works because NEW(old_semi) recognizes new system variable and sets new user variable. FUTURE->NEW(new_semi): Works because NEW(new_semi) recognizes new-named system variables and sets new-named user variables. OLD->FUTURE: Unsupported (see asynchrounous replication case) FUTURE->OLD: Unsupported (see asynchrounous replication case) mysqlbinlog ----------- When mysqlbinlog connects to a source server to receive binary logs, it uses the same logic as an asynchronous replica. Thus, the same cross-version compatibility constraints apply to the mysqlbinlog connection as stated above for an asynchronous source-replica connection. mysqlbinlog prints one variable that is renamed in this worklog, namely pseudo_slave_mode. In order for an old replica to process the output from mysqlbinlog, in this worklog we retain the old variable name in the output from mysqlbinlog. A possible future transition to using only the new name is: - OLD mysqlbinlog prints pseudo_slave_mode and recognizes only pseudo_slave_mode. - NEW mysqlbinlog prints pseudo_slave_mode and recognizes both pseudo_slave_mode and pseudo_replica_mode. - NEW+1 prints pseudo_replica_mode and recognizes both pseudo_slave_mode and pseudo_replica_mode. - NEW+2 prints pseudo_replica_mode and recognizes only pseudo_replica_mode. With this chain of changes, each version is able to interpret the output both from a one version older mysqlbinlog and from a one version newer mysqlbinlog. APPLICATION COMPATIBILITY ========================= Application changes required when upgrading from an older version to a version having the feature: - The changes in system variables, command-line options, status variables, built-in SQL functions, and semisync plugins are fully backward compatible, since we maintain the old names as aliases. An application that uses them will work without change, but may observe new deprecation warnings. - The changes in peformance_schema instrumentation - thread names, thread commands, locks, condition variables, thread stages, and memory allocation - are incompatible. An application that depends on the exact texts for these will have to be modified. - The changes in C++ filenames, header guards, and debug symbols are internal and not part of the user interface, so they do not count as incompatible. However, future patches written in 5.7 will have to deal with the renames while up-merging them. Application changes required before upgrading from a version having the feature to a future version where the old names are removed: - The changes in system variables, command-line options, status variables, built-in SQL functions, and semisync plugins require that the user replaces the deprecated terminology by the new terminology after upgrading to the version having the feature and before the next upgrade. Application changes on downgrade: - An application that has upgraded to the version having the feature, but not started to use the new names yet, can downgrade without problem. - An application that has upgraded to the version having the feature, and started to adopt the new names in configuration files and SQL, has to first revert to the old terminology before downgrading. - An application that has upgraded to the version having the feature, adopted the new names in configuration files and SQL, and upgraded to the next version, can downgrade to the version where the feature was introduced. From there, they can revert to the old terminology and then downgrade to the version before the feature. INTERNAL CODE CHANGES ===================== We change other internal code identifiers when it is convenient, for instance, when a C++ identifier matches a replacement pattern by chance.
We implement this in several steps. We split the work in three sets of patches: 1. Refactorings that simplify later steps, and scripts ====================================================== 1.1.Various preparation steps This contains miscelanneous preparation steps needed before the automatic replacement: - Eliminate multi-line string literals from the replication code, so that string literals are easy to find by line-based search tools, and so that the code formatting looks better. This was found by searching for lines containing an odd number of double quotes. The replacements were performed manually. - Add a copyright header in a file that we are going to modify later, so that git push won't complain. Since later steps are auto-generated it's a bit easier to add the copyright header before them than after them. - Fix a command-line option that mixes dashes and underscores (slave-preserve_commit-order), since the rename script only searches for identifiers with only dashes or only underscores. 1.2.Fix mysqltest's escape function * mysqltest's escape function should allow multiple escaped characters mysqltest's escape function takes two arguments. The first one is a character to escape and the second one is a string in which the character will be escaped. The usual use case for escape is to prepare a string for being used in string context in an SQL statement. Therefore, both the quote character (e.g. single quote) and the escape character (backslash) need to be escaped. That requires two calls to escape, which is a bit cumbersome. Therefore, we extend the function to allow multiple escaped characters. The reason we disallowed multiple escaped characters was to allow any character including comma to be escaped, and still use comma as a separator between the arguments. We make that work by requiring that the user places any comma occurring among the escaped characters first, and make the function look for the separator only after the first character. * Adapt test utilities to use the escape function - assert.inc, eval.inc: These should have used 'escape' from the beginning, so the caller wouldn't have to bother escaping strings. Now we have too many tests using them and escaping strings, so making it an option. - escape_sql.inc: This was a workaround for not having proper 'escape' function in the language. Removed it now. - show_slave_status.inc: Updated it to use the escape function instead of escape_sql.inc 1.3.Fix min/max values for sysvars with block_size!=1 Background: System variables have a feature known as "block_size". Numeric system variables can only be set to values that are multiples of block_size. Most variables use a block size of 1. Numeric variables are additionally limited to a maximum and a minimum value. Problem: In case the minimum and maximum values are not specified as multiples of the block_size, the effective minimum and maximum are different than the ones specified. This typically happens when using a block_size that is a power of two, for instance 1024, and a maximum that is a numeric limit such as MAXINT, which may have values such as 2^64-1, i.e., one-less than a multiple of the block_size. This has no observable effect, other than the maximum value displayed in performance_schema.system_variables_info being a value that is not actually allowed to use for the variable. This is a litte inconsistent, and also made it hard to write the scripted tests in later steps of the worklog. Also, the indentation was wrong all over Sys_var_integer. Fix: - Round the maximum down to the next smaller multiple of block_size. - Round the minimum up to the next grater multiple of block_size. - Fix the indentation. 1.4.Scripts This contains the scripts used to execute the following steps. See 00README for an overview. These scripts are only kept in the server branch during the development of WL#14194. They will not be pushed to the main repositories, but possibly published elsewhere in case it eases migration for anyone. 2. Scripted replacements ======================== Perform all automated search-and-replace operations. The following replacements are performed this way: 2.0. Replace "master" by "mistir" and "slave" by "slivi" in contexts where we want to preserve the old terms. We replace this back in the last search-and-replace operation. This hides it from the other search-and-replace operations. 2.1. system variables 2.2. command-line options for mysqld 2.3. command-line options for mysqlbinlog 2.4. command-line options for mysqldump 2.5. command-line options for mysqladmin 2.6. SQL functions 2.7. status variables 2.8. thread commands 2.9. instrumented locks 2.10. instrumented condition variables 2.11. thread stages 2.12. thread names 2.13. instrumented memory allocations 2.13. C++ filenames 2.14. header guards in C++ header files 2.15. debug symbols 2.16. semisync 2.17. user variables used in the replication protocol handshake 2.99. Replace "mistir" by "master" and "slivi" by "slave" Each of these categories is committed in a separate step. This includes some C++ identifiers and mtr variables that contain the replaced variables as substrings. This is only good. It is necessary to rename filenames, since some .inc files in the test suite contain e.g. system variable names in their names, so the place where they are sourced will have their names changed. 3. Manual replacements, aliases, compatibility code, and tests ============================================================== 3.1.Fix tests Fix test failures introduced by previous patch. There were two tests that wrote variable names in alphabetic order to their result files. So we re-record those result files. 3.2.Help texts Remove the words "master/slave" from help texts for command line options and system variables. The search is automatic, finding occurrences of 'master' and 'slave' within double-quoted strings, in sys_vars.cc mysqld.cc, and semisync_*_plugin.cc (for server variables and command-line options), and in mysqlbinlog.cc, mysqladmin.cc, mysqldump.cc (for client command-line options). However, the replacement is manual. Therefore we also clarify and correct some texts. Since help texts rarely occur in the test suite, this will not have too much unpredictable consequences: we just need to update the result files for mysqld--help-[not]win. We also fix some other texts occurring in these files, since it is easier to do it now than to postpone it: - Error messages in utilities. These usually don't occur in tests. - Plugin description strings for the semisync plugins. - Deprecation warning for --master-retry-count. - Comments printed by mysqldump. - Source code identifiers in mysqladmin.cc. 3.3.Aliases for command line options For all renamed command-line options, we re-introduce the old name as a deprecated alias. This includes command-line options in mysqladmin, mysqlbinlog, mysqldump, and the server. For the server, it only includes command-line options that are not defined through a system variable. 3.4.Aliases for status variables For all renamed status variables, we re-introduce the old name as an alias. This does not include semisync status variables. 3.5.Aliases for semi-sync libraries and variables * Server changes: For the semisync libraries, we previously renamed both the filename, the plugin name, the system variables defined in the libraries, and the status variables defined in the libraries. Now we create aliases for these as follows. We duplicate the libraries so there exists one library having the old filename, the old plugin name, the old-named system variables, and the old-named status variables. We generate a deprecation warning only when the old libraries are loaded, not when using the old-named variables. Additionally, we generate an error when using the two duplicated source-side libraries or the two duplicated replica-side libraries on the same server. * Package definition changes: To make the packages aware of the new library, we update the following files: - mysql-test/include/plugin.defs - packaging/**/* * Test suite changes: - Add a test utility to disable/restore binlogging in the current session. Changed files: - disable_binlog.inc - restore_binlog.inc - Add a test utility to use a JSON object as a key-value store and lookup the value when using a mysqltest $variable as a key. New file: - json_lookup.inc - Add test utilities to assert that some specific messages were written to the error log. New test utilities: - save_error_log_position.inc: Store the current position in the log in an internal variable. - assert_error_log.inc: Assert that a given message (or sequence of messages) was written to the log since the last saved position. - show_error_log.inc: Show all messages generated since the last saved position. The above utilites need to respect the global error suppressions defined in mtr_warnings.inc. To make use of that, we refactor mtr_warnings.inc and related files. Changed files: - check-warnings.test - load_error_log.inc - mtr_warnings.sql A result file had to be updated, since the test printed all tables in the mtr database to the result log, and we needed another one in order to implement the utilities above: - default_row_format_01.test - Add test utility that simplifies the use of mtr.add_suppression. This automates the suppression of messages on all servers in a replication topology, and the disabling/re-enabling of binary logging on the current session. Changed files: - suppress_messages.inc - Add test case for the feature: rpl_semi_sync_alias. 3.6.Cross-version-compatible connection user vars When the replica connects to the source, it first connects, then sets some user variables using regular SQL SET statements, and then starts replication. We previously renamed some of those user variables. Now we create aliases for those user variables in order to make cross-version replication work, as follows: - On the replica side, we set both the old-named variable and the new-named variable. - On the source side, we accept either the old-named variable or the new-named variable. 3.7.Aliases for SQL functions This creates the deprecated alias MASTER_POS_WAIT for the renamed SQL function SOURCE_POS_WAIT. 3.8.Aliases for system variables We previously renamed several system variables. Now we introduce a framework for defining aliases for system variables, and define deprecated aliases for all the renamed system variabes. The framework consists of a subclass of Sys_var called Sys_var_deprecated_alias. This class represents an alias for some other Sys_var. It copies most member fields of that variable, so setting one of the variables changes the value of both. We also make it so that SET PERSIST writes the name of both variables. When loading persisted variables, when either the alias or the real name appears in the file, it loads both of them into the memory cache. This ensures that both the real variable and the alias will be persisted the next time the persisted variable file is written, which ensures backward and forward compatibility in future upgrade/downgrade scenarios. In order to load the aliases while loading persisted variables, we need to access the system variable framework (which contains the alias definitions) while loading persisted variables. To do that, we have to move the initialization of the system variable framework a bit earlier - hence the change in mysqld.cc. This is only the initialization of the framework, which makes it possible to access metadata about the variable definitions; it is not the initialization of system variable values. The system variable framework initialization is self-contained and does not depend on other things, so this change should not have any other side effects. When using RESET PERSIST, we now reset both the variable and the alias. When handling command-line options, we make the old name generate a deprecation warning. This is done in mysqld.cc:get_one_option. We need changes to make the "variable_source" information accurate when using an alias on the command line. (The variable_source is used to populate columns of the performance_schema.variables_info table.) When an alias is used on the command line, the command-line parsing framework can only store the variable_source for the alias, since it does not know that the alias is an alias. Therefore, we copy the variable_source from the alias to the base variable from code that has this information; namely, in mysqld.cc:get_one_option. We remove the function clear_user_host_timestamp since it was unused, and otherwise we would have to define it in Sys_var_alias. We make sys_var::set_and_truncate return early if source and destination are identical. Otherwise Valgrind complains. This happens in the new call to sysvar->set_source_name from mysqld.cc:mysqld_get_one_option. 3.9.Tests for system variable aliases 1. Test that the system variable aliases introduced in the previous commit work as expected. To make this less repetitive, introduce a framework to test system variables using a declarative syntax. To make that framework possible, we introduce another, lower-level framework, which provides control flow based on JSON documents: in particular, it allows test cases to iterate over JSON arrays, and unpack JSON objects into a set of mtr variables. Relying on this JSON framework, we can define properties of system variables using JSON, and then iterate over all such definitions, unpack the properties of the sysvars into mtr variables, and invoke the higher-level framework to test the system variable. The high-level framework is described in mysql-test/suite/sys_vars/inc/control_for_sysvars_with_aliases.inc. The JSON specifications are in mysql-test/suite/sys_vars/inc/json_sysvar_specs.inc. The JSON framework is in mysql-test/include/create_json_iterator.inc, mysql-test/include/create_json_unpacker.inc, and mysql-test/include/create_unpacking_json_iterator.inc. Persisted variables are tested in mysql-test/suite/sys_vars/t/persisted_sysvars_with_aliases.inc Also minor improvements in assert.inc and eval.inc. Also added: mysql-test/include/check_64bit_machine.inc, mysql-test/include/check_64bit_ulong.inc, mysql-test/include/have_32bit_ulong.inc, and mysql-test/include/have_64bit_ulong.inc, and updated: mysql-test/include/have_32bit.inc and mysql-test/include/have_64bit.inc The new funcationliaty was necessary because some 64-bit windows platforms use only 32 bits for ulong. Also extended sys_vars.pseudo_replica_mode_basic. This is a session variable, and therefore not covered by the new framework. We could have written framework for session variables, but it's too much work for just one variable. 2. Fix tests that broke when adding aliases: binlog_nogtid.binlog_persist_only_variables: - This test used wildcards like %master% and %slave% to select all replication-related system variables from performance_schema tables. The automatic replacements changed this to %source% and %replica%, which made it match a few more variables, so the test started to fail already in that step. - After adding aliases, we need to make the match %master% and %slave% again, in addition to %source% and %replica%, to include all the aliases. - We also need to modify the logic of the test when the test executes RESET PERSIST. Before, it ran RESET PERSIST once for each variable. However, when doing RESET PERSIST for a variable having an alias, it will reset both the variable and the alias. Then when the test issues RESET PERSIST for the alias, it has already been reset and generates an error. Fixed by excluding aliases from this part of the test. persisted_variables_extended: - This test used SET PERSIST replica_net_timeout and then SELECT ... FROM performance_schema.persisted_variables ... Therefore we update the result file to include the alias. mysql-test/suite/binlog_nogtid/r/binlog_persist_variables.result mysql-test/suite/binlog_nogtid/r/binlog_persist_variables.test - This test counts the number of variales. Updated accordingly. - The test lists replication variables, heuristically guessing which variables are replication variables using a pattern. Updated the pattern to include the aliases. - Updated the result file to include the aliases and their deprecation warnings - The result file was auto-updated by the scripted renames. It contains a list of variables in alphabetic order. The rename broke the alphabetic order, so updated the result file accordingly. mysql-test/r/read_only_persisted_plugin_variables.result mysql-test/r/read_only_persisted_variables.result mysql-test/r/reset_persisted_variables.result mysql-test/r/variables.result mysql-test/suite/perfschema/r/persisted_variables.result mysql-test/suite/perfschema/r/variables_info.result mysql-test/suite/sys_vars/r/all_vars.result mysql-test/t/all_persisted_variables.test - These tests iterate over all variables, or some subset of them, or assert that the number of variables is equal to some constant. Updated result files to include the aliases. mysql-test/r/mysqlcheck.result mysql-test/r/partition_innodb_tablespace.result mysql-test/suite/funcs_1/r/is_key_column_usage.result mysql-test/suite/funcs_1/r/is_key_column_usage_ci.result mysql-test/suite/funcs_1/r/is_statistics_ci.result mysql-test/suite/funcs_1/r/is_statistics_cs.result mysql-test/suite/funcs_1/r/is_table_constraints.result mysql-test/suite/funcs_1/r/is_table_constraints_ci.result mysql-test/suite/innodb/r/case_insensitive_fs.result mysql-test/suite/innodb/r/choose_tbsp_location-discard.result mysql-test/suite/innodb/r/create_tablespace.result mysql-test/suite/innodb/r/create_tablespace_16k.result mysql-test/suite/innodb/r/create_tablespace_32k.result mysql-test/suite/innodb/r/create_tablespace_4k.result mysql-test/suite/innodb/r/create_tablespace_64k.result mysql-test/suite/innodb/r/create_tablespace_8k.result mysql-test/suite/innodb/r/create_tablespace_debug.result mysql-test/suite/innodb/r/create_tablespace_replication.result mysql-test/suite/innodb/r/default_row_format.result mysql-test/suite/innodb/r/default_row_format_compatibility.result mysql-test/suite/innodb/r/default_row_format_tablespace.result mysql-test/suite/innodb/r/discard_tablespace.result mysql-test/suite/innodb/r/hidden_directory_dotfile.result mysql-test/suite/innodb/r/hidden_directory_win.result mysql-test/suite/innodb/r/innodb-index-online-fk.result mysql-test/suite/innodb/r/innodb_tablespace.result mysql-test/suite/innodb/r/known_dir_unique_undo.result mysql-test/suite/innodb/r/known_directory.result mysql-test/suite/innodb/r/partition_autoinc.result mysql-test/suite/innodb/r/portability_tablespace.result mysql-test/suite/innodb/r/rename_table.result mysql-test/suite/innodb/r/tablespace_per_table.result mysql-test/suite/innodb/r/unknown_directory.result mysql-test/suite/innodb_fts/r/tablespace_location.result mysql-test/suite/innodb_fts/r/tablespace_location_error.result mysql-test/suite/innodb_zip/r/16k.result mysql-test/suite/innodb_zip/r/4k.result mysql-test/suite/innodb_zip/r/8k.result mysql-test/suite/innodb_zip/r/restart.result mysql-test/suite/innodb_zip/r/zip.result mysql-test/suite/parts/r/partition_reorganize_innodb.result - These tests count or list all tables defined. We added two tables in the framework for assert_error_log.inc in step 3.5, so updated the result file. mysql-test/include/mysqld--help.inc - This has a list of variables whose default is nondeterministic, and one of the aliases had to be added there. mysql-test/r/non_table_atomic_ddl.result mysql-test/r/trigger_mdl.result mysql-test/suite/binlog/r/binlog_truncate_kill.result - The removal of begin_include_file.inc from filter_file.inc also got rid of a 'connection
' printout from the result log in tests using show_binlog_events.inc. Updated result files. 3.10.Justify use of old term in mysqlbinlog Make mysqlbinlog print a comment to clarify why we are still using the old terminology.
Copyright (c) 2000, 2021, Oracle Corporation and/or its affiliates. All rights reserved.