WL#14194: Replace old terms in replication system variables, options, and strings
Affects: Server-8.0
—
Status: Complete
EXECUTIVE SUMMARY
=================
Replace the terms "master", "slave", and "mts" by "source", "replica",
and "mta" in system variable names, status variables, SQL functions,
command-line options (for server and all tools), semisync plugin names
and file names, C++ filenames and header guards, debug symbols, help
texts, and in performance_schema instrumentation names for locks,
condition variables, threads, execution stages, allocators, and thread
commands. Create aliases so that users can use the old names for
system variables, SQL functions, and command-line options, and
deprecate the use of those aliases.
USER STORIES
============
U1. As a MySQL admin user:
- I want to use configuration that does not contain offensive
terms,
- so that I do not get offended by them and/or expose them to
other people (e.g., customers of my application built on top of
MySQL).
U2. As a MySQL admin user:
- I want to monitor the product without seeing offensive terms,
- so that I do not get offended by them and/or expose them to
other people (e.g., customers of my application built on top of
MySQL).
SCOPE
=====
This worklog is part of a broader effort to change old terminology
across the entire MySQL product line.
This worklog focuses only on changing *replication* terminology. For
the word "master", we only change it where it appears in a replication
context; we do not change "master key" or "master thread".
We only change status variables, SQL functions, command-line options,
semisync plugin names and filenames, C++ filenames and header guards,
debug symbols, help texts, and performance_schema instrumentation
names. (So we do not change privilege names, column names or other
metadata for system tables, error messages, ER_* symbols, or SQL
commands.)
LIMITATIONS
===========
We do not change identifiers related to NDB.
We do not change SQL printed by mysqldump or issued by mysqladmin,
MEB, or other external tools except mysqlbinlog. The identifiers we
change, are not changed when they appear in error messages.
REFERENCES
==========
- https://mysqlhighavailability.com/mysql-terminology-updates/
FR1. The words "master", "slave", and "mts" shall be replaced by
"source", "replica", and "mta", respectively, in the contexts
listed below, except where "master" occurs in a non-replication
context such as "master key" or "master thread".
Where possible, we keep the old names for backward compatibility,
as indicated in each item in the list.
Where possible the old names generate deprecation warnings, as
indicated in each item in the list:
FR1.1. All command-line options for mysqld, except those that
will be deprecated in WL#14526, and those that are defined
by NDB.
The old names shall remain, and using them shall generate
a deprecation warning in the error log.
FR1.2. All command-line options for mysqladmin.
The old names shall remain and make the tool print
deprecation warnings.
FR1.3. All command-line options for mysqlbinlog.
The old names shall remain and make the tool print
deprecation warnings.
FR1.4. All command-line options for mysqldump.
The old names shall remain and make the tool print
deprecation warnings.
FR1.5. All system variables, except those that have been deprecated
already.
The old names shall remain. The use of them in SQL
statement shall make the client return a deprecation
warning. The use of them on the command-line or
configuration file shall generate a deprecation warning in
the error log.
The following shall hold for persisted variables:
FR1.5.1. When the server reads persisted variables, it
shall accept either the old name, or the new
name, or both.
FR1.5.2. When the server writes persisted variables, it
shall persist both the old and the new name, even
if only one of them was specified in a SET
statement.
FR1.5.3. When RESET PERSIST is used with a renamed system
variable, regardless of using the old or new name
in the statement:
- The old name shall be removed if it is
persisted;
- The new name shall be removed if it is
persisted;
- A warning/error shall only be generated if
neither the old nor the new name was persisted.
Both the old and the new variables shall exist in the
following tables and commands, but no deprecation warning
shall be issued while reading them:
performance_schema.global_variables
performance_schema.session_variables
performance_schema.variables_by_thread
performance_schema.persisted_variables
performance_schema.variables_info
SHOW VARIABLES
FR1.6. All status variables defined in the server, except those
defined by NDB.
Both the old and the new variables shall exist in the
following places, and no deprecation warning shall be
generated when reading them:
performance_schema.global_status
performance_schema.session_status
performance_schema.status_by_thread
SHOW STATUS
FR1.7. All user variables that the replica sets on the connection
before it sends a COM_BINLOG_DUMP command to start
replication.
FR1.7.1. The source shall accept either the old or the new
names.
FR1.7.2. The replica shall set both the old-named and the
new-named variable.
No deprecation warning shall be generated.
FR1.8. All SQL functions defined in the server.
The old names shall remain and the use of them shall make
the client return deprecation warnings.
FR1.9. The following identifiers instrumented in performance_schema:
FR1.9.1. Thread commands
FR1.9.2. Instrumented locks
FR1.9.3. Instrumented condition variables
FR1.9.4. Thread stages
FR1.9.5. Thread names
FR1.9.6. Instrumented memory alloctions
The old names shall be removed.
FR1.10. All C++ source file names.
The old names shall be removed.
FR1.11. All header guards in C++ source filenames.
The old names shall be removed.
FR1.12. All debug symbols (as in
`DBUG_EXECUTE_IF("debug_symbol", ...)`).
The old names shall be removed.
FR1.13. The following names related to the two semisync libraries:
FR1.13.1. The library filenames.
FR1.13.2. The plugin names.
FR1.13.2. All system variables defined by the plugins.
FR1.13.2. All status variables defined by the plugins.
The libraries shall be duplicated so that old filenames
can be used with the old plugin names, and loading them
shall define the old-named system variables and status
variables.
Deprecation warnings shall be generated when loading the
old-named libraries.
An error shall be generated if the user loads both the
old-named and the new-named source libraries, or both the
old-named and the new-named replica libraries.
FR1.14. Compiled-in help texts for system variables and
command-line options:
- Any occurrence of renamed identifiers shall use the new
name.
- Any occurrence of master/slave/mts shall be replaced by
source/replica/mta.
- Any mistakes spotted while editing the text shall be
fixed.
- Any deprecated options shall be documented as
deprecated, with a reference to the new option.
SUMMARY OF THE APPROACH
=======================
Replace the old terms by the new terms in all the required places.
Where possible, maintain the old terms as aliases. This is possible
for configuration and other things that appear as identifiers in SQL,
but not for strings that appear in performance_schema tables, so we
break compatibility for them.
This needs to change more than 16000 files. Therefore, we perform the
replacements automatically. To make that practical, we use a tool
that reads a file containing replacement rules and applies each rule
to the full source tree. The tool needs to express things in
different ways than what standard tools like grep can, so we code a
new tool for this.
After applying the replacements, we code the aliases in a sequence of
follow-up steps.
CHANGES IN USER INTERFACE
=========================
Here we list the full set of replaced identifiers. Unless otherwise
specified, we uniformly replace "master" by "source", "slave" by
"replica", and "mts" by "mta".
* Changed system variables defined in the server:
For the variables below, the old name remains and is deprecated, and
the new name is added. Both names will exist in all contexts where
variables appear, i.e., --command-line-option, @@global.variable,
@@session.variable, SET PERSIST, SHOW VARIABLES,
performance_schema.global_variables,
performance_schema.persisted_variables,
performance_schema.variables_info. Deprecation warnings are emitted
when using the variable on the command line, in a SET statement, and
when accessing it using @@ notation. No warning is emitted when the
variable is accessed through a table or SHOW statement. The built-in
help for the old name states that it is deprecated.
master_verify_checksum
sync_master_info
init_slave
rpl_stop_slave_timeout
log_slow_slave_statements
slave_max_allowed_packet
slave_compressed_protocol
slave_exec_mode
slave_type_conversions
slave_sql_verify_checksum
slave_parallel_type
slave_preserve_commit_order
log_slave_updates
slave_allow_batching
slave_load_tmpdir
slave_net_timeout
sql_slave_skip_counter
slave_skip_errors
slave_checkpoint_period
slave_checkpoint_group
slave_transaction_retries
slave_parallel_workers
slave_pending_jobs_size_max
pseudo_slave_mode
skip_slave_start
The following variables are not changed:
- ndb_slave_conflict_role: We do not change NDB now.
- binlog_rotate_encryption_master_key_at_startup: This refers to the
"master key", which is a metaphor without offensive connotations.
This use of master is considered unproblematic and will not be
replaced.
- slave_rows_search_algorithms: Deprecated already
- master_info_repository: Deprecated already.
* Changed command-line options for mysqld:
For the following option, the old name remains and is deprecated, and
the new name is added. A deprecation warning is issued when the
command-line option is used. The built-in help for the old name
states that it is deprecated. (Most command-line options are also
system variables, and are included in the list in the preceding
section instead. The list here includes only the command-line options
which are not system variables.)
show-slave-auth-info
The following command-line options are not changed:
- abort-slave-event-count: Will be deprecated in WL#14526.
- disconnect-slave-event-count: Will be deprecated in WL#14526.
- master-info-file: Deprecated already.
- master-retry-count: Deprecated already.
* Changed command-line options for mysqlbinlog:
For the following option, the old name remains and is deprecated, and
the new name is added. A deprecation warning is issued when the
command-line option is used. The built-in help for the old name
states that it is deprecated.
read-from-remote-master
The following variable is not changed:
- stop-never-slave-server-id: Deprecated already.
* Changed command-line options for mysqldump:
For the following options, the old name remains and is deprecated, and
the new name is added. A deprecation warning is issued when the
command-line option is used. The built-in help for the old name
states that it is deprecated.
apply-slave-statements
delete-master-logs
dump-slave
include-master-host-port
master-data
* Changed command-line options for mysqladmin:
For the following options, the old name remains and is deprecated, and
the new name is added. A deprecation warning is issued when the
command-line option is used. The built-in help for the old name
states that it is deprecated.
start-slave
stop-slave
* Changed built-in SQL functions:
For the following function, the old name remains and is deprecated,
and the new name is added. A deprecation warning is issued when the
function is used.
MASTER_POS_WAIT
* Changed status variables:
Status variables are displayed with SHOW STATUS LIKE 'name' and
performance_schema.status_variables. The old name remains and the new
name is added. No deprecation warning is used.
Slave_open_temp_tables
Slave_rows_last_search_algorithm_used
Rpl_semi_sync_master_status
Rpl_semi_sync_master_clients
Rpl_semi_sync_master_yes_tx
Rpl_semi_sync_master_no_tx
Rpl_semi_sync_master_wait_sessions
Rpl_semi_sync_master_no_times
Rpl_semi_sync_master_timefunc_failures
Rpl_semi_sync_master_wait_pos_backtraverse
Rpl_semi_sync_master_tx_wait_time
Rpl_semi_sync_master_tx_waits
Rpl_semi_sync_master_tx_avg_wait_time
Rpl_semi_sync_master_net_wait_time
Rpl_semi_sync_master_net_waits
Rpl_semi_sync_master_net_avg_wait_time
Rpl_semi_sync_slave_status
All NDB-related status variables are skipped.
The following variables were already removed in 8.0:
Slave_running
Slave_retried_transactions
Slave_heartbeat_period
Slave_received_heartbeats
Slave_last_heartbeat
(There is some dead server code that makes it look like they exist, if
you read the code. Hence they were listed in an early version of this
worklog. We do not consider them in this worklog, and will remove
them in an independent bugfix.)
* Changed thread names in PERFORMANCE_SCHEMA:
Thread names are visible in the NAME column of
performance_schema.threads. The old name is replaced by the new name.
This is an incompatible change.
slave_io
slave_sql
slave_worker
* Changed thread commands in PERFORMANCE_SCHEMA:
Thread command is the value shown in the following places:
- The PROCESSLIST_COMMAND column of performance_schema.threads,
- the COMMAND column of performance_schema.processlist,
- the COMMAND column of SHOW PROCESSLIST,
- the COMMAND column of INFORMATION_SCHEMA.PROCESSLIST.
In the following case, the old name is replaced by the new name. This
is an incompatible change.
Register Slave
* Changed names of instrumented locks in PERFORMANCE_SCHEMA:
Mutex names are visible with a 'wait/synch/mutex/'
prefix, in the following places:
- the NAME column of performance_schema.mutex_instances
- the EVENT_NAME column of all performance_schema.events_waits_* tables
In the following cases, the old name is replaced by the new name. This
is an incompatible change.
Master_info::data_lock
Master_info::run_lock
Master_info::sleep_lock
Master_info::info_thd_lock
Master_info::rotate_lock
Slave_reporting_capability::err_lock
Relay_log_info::slave_worker_hash_lock
key_mts_temp_table_LOCK
key_mts_gaq_LOCK
LOCK_slave_list
* Changed names of instrumented condition variables in PERFORMANCE_SCHEMA:
Condition variable names are visible with a 'wait/synch/cond/' prefix
in the following places:
- the NAME column of performance_schema.cond_instances.
- the EVENT_NAME column of all performance_schema.events_waits_* tables
In the following cases, the old name is replaced by the new name. This
is an incompatible change.
Master_info::data_cond
Master_info::start_cond
Master_info::stop_cond
Master_info::sleep_cond
Master_info::rotate_cond
LOCK_slave_trans_dep_tracker
Relay_log_info::mts_gaq_cond
* Changed names of thread stages in PERFORMANCE_SCHEMA:
Thread stages are visible in the following places:
- the PROCESSLIST_STATE column of performance_schema.threads, and
- the EVENT_NAME column of all performance_schema.events_stages_* tables, and
- the STATE column of performance_schema.processlist, and
- the STATE column of SHOW PROCESSLIST, and
- the STATE column of INFORMATION_SCHEMA.PROCESSLIST.
In the following cases, the old name is replaced by the new name. This
is an incompatible change.
Checking master version
Connecting to master
Flushing relay log and master info repository.
Queueing master event to the relay log
Waiting until MASTER_DELAY seconds after master executed event
Waiting for master to send event
Waiting for master update
Killing slave
Master has sent all binlog to slave; waiting for more updates
Registering slave on master
Sending binlog event to slave
Slave has read all relay log; waiting for more updates
Waiting for slave workers to process their queues
Waiting for Slave Worker queue
Waiting for Slave Workers to free pending events
Waiting for Slave Worker to release partition
Waiting for slave mutex on exit
Waiting for slave thread to start
Waiting for the slave SQL thread to advance position
The following thread stages are defined in the old semi-sync plugin.
The new name is used in the new semi-sync plugin. This is a
*compatible* change, since users must enable the new library in order
to observe the new names.
Waiting for semi-sync ACK from slave
Waiting for semi-sync slave connection
Reading semi-sync ACK from slave
The following thread stages are changed in more complex way than the
scripted word substitution:
Waiting for the slave SQL thread to free enough relay log space
Waiting for the replica SQL thread to free relay log space
- The old text is exactly the maximum allowed length, and replacing
'slave' by 'replica' makes it one character too long. Hence removing
the redundant word 'enough'.
Changing master
Changing replication source
- This stage referred to the SQL statement CHANGE MASTER, which has
been renamed to CHANGE REPLICATION SOURCE, so the renamed stage
shall use the new SQL command name.
* Changed instrumented memory allocations:
Slave_job_group::group_relay_log_name
rpl_slave::check_temp_dir
SLAVE_INFO
show_slave_status_io_gtid_set
memory/sql/Relay_log_info::mts_coor
The old name is replaced by the new name. This is an incompatible
change.
* Changed C++ filenames:
rpl_slave.cc
rpl_slave.h
rpl_slave_until_options.cc
rpl_slave_until_options.h
rpl_slave_commit_order_manager.cc
rpl_slave_commit_order_manager.h
rpl_master.cc
rpl_master.h
rpl_mts_submode.cc
rpl_mts_submode.h
semisync_master.cc
semisync_master.h
semisync_master_ack_receiver.cc
semisync_master_ack_receiver.h
semisync_master_plugin.cc
semisync_master_socket_listener.h
semisync_slave.cc
semisync_slave.h
semisync_slave_plugin.cc
This is an internal change and does not change the API. Compatibility
is not an issue.
* Changed C++ header guards:
RPL_SLAVE_H
DEFINED_RPL_SLAVE_UNTIL_OPTIONS_H
RPL_SLAVE_COMMIT_ORDER_MANAGER
RPL_MASTER_H_INCLUDED
SEMISYNC_MASTER_H
SEMISYNC_MASTER_ACK_RECEIVER_DEFINED
SEMISYNC_MASTER_SOCKET_LISTENER
SEMISYNC_SLAVE_H
MTS_SUBMODE_H
This is an internal change and does not change the API. Compatibility
is not an issue.
* Changed C++ debug symbols:
after_locking_channel_map_in_start_slave
after_start_slave
begin_master_pos_wait
begin_start_slave
check_slave_debug_group
check_slave_master_info
dbug.before_get_MASTER_UUID
dbug.mts.force_clock_diff_eq_0
dbug.return_null_MASTER_UUID
debug_crash_slave_time_out
enable_mts_wokrer_failure_in_recovery_finalize
enable_mts_worker_failure_init
fail_generating_rotate_event_on_write_rotate_to_master_pos
fail_to_flush_master_info
fake_5_5_version_slave
fault_injection_get_slave_worker
fix_slave_net_timeout
force_rollback_in_slave_on_transactional_ddl_commit
FORCE_SLAVE_TO_RECONNECT_DUMP
FORCE_SLAVE_TO_RECONNECT_EVENT
FORCE_SLAVE_TO_RECONNECT_REG
get_master_server_id.
get_master_server_id.ER_NET_READ_INTERRUPTED
mts_checkpoint
mts_checkpoint_end
mts_checkpoint_start
mts_debug_concurrent_access
mts_debug_recovery_reset_fails
mts_debug_reset_workers_fails
mts_distribute_round_robin
mts_slave_worker_init_at_gaps_fails
mts_worker_thread_fails
mts_worker_thread_init_fails
ndb_slave_fail_marking_epoch_committed
old_row_based_repl_4_byte_map_id_master
query_log_event_mts_corrupt_db_names
remove_slave_load_file_before_write
request_master_log_pos_3
rotate_slave_debug_group
rpl_semisync_master_commit_trx_before_lock
rpl_semisync_simulate_add_slave_failure
rpl_slave_debug
rpl_slave_debug_point
signal.get_master_uuid
signal.got_stop_slave
simulate_hold_run_locks_on_stop_slave
simulate_io_slave_error_on_init
simulate_master_has_gtid_mode_off_something
simulate_master_has_gtid_mode_on_something
simulate_master_has_unknown_gtid_mode
simulate_register_slave_killed
simulate_slave_delay_at_terminate_bug38694
simulate_slave_unaware_checksum
simulate_slave_unaware_gtid
simulate_sql_slave_error_on_init
simulate_stop_when_mts_in_group
slave_acquired_backup_lock
slave_crash_after_commit
slave_crash_after_commit_no_atomic_ddl
slave_crash_before_commit
slave_reconnect_with_gtid_set_executed
slave_worker_ends_group_before_signal_lwm
stop_slave_dont_release_backup_lock
stop_slave_middle_group
stop_when_mts_in_group
undefined_algorithm_on_slave
uninitialized_master-info_structure
unrecognized_master_version
This is an internal change and does not change the API. Compatibility
is not an issue.
* Changed plugin library filenames:
semisync_slave.so
semisync_master.so
The old-named file remains and is deprecated, and the new-named file
is added. A deprecation warning is issued when loading the old-named
file.
* Changed plugin names:
rpl_semi_sync_slave
rpl_semi_sync_master
The old name exists only in the old-named file, and the new name
exists only in the new file. A deprecation warning is issued when
loading the old-named file.
* Changed system variables defined in the semisync plugins:
rpl_semi_sync_master_wait_for_slave_count
rpl_semi_sync_master_wait_no_slave
rpl_semi_sync_slave_enabled
rpl_semi_sync_slave_trace_level
rpl_semi_sync_master_wait_for_slave_count
rpl_semi_sync_master_wait_no_slave
rpl_semi_sync_master_enabled
rpl_semi_sync_master_timeout
rpl_semi_sync_master_trace_level
rpl_semi_sync_master_wait_point
The old name exists only in the old-named plugin, and the new name
exists only in the new-named plugin. A deprecation warning is issued
when loading the old-named plugin.
* Changed status variables defined in the semisync plugins:
Rpl_semi_sync_slave_enabled
Rpl_semi_sync_master_status
Rpl_semi_sync_master_clients
Rpl_semi_sync_master_yes_tx
Rpl_semi_sync_master_no_tx
Rpl_semi_sync_master_wait_sessions
Rpl_semi_sync_master_no_times
Rpl_semi_sync_master_timefunc_failures
Rpl_semi_sync_master_wait_pos_backtraverse
Rpl_semi_sync_master_tx_wait_time
Rpl_semi_sync_master_tx_waits
Rpl_semi_sync_master_tx_avg_wait_time
Rpl_semi_sync_master_net_wait_time
Rpl_semi_sync_master_net_waits
Rpl_semi_sync_master_net_avg_wait_time
The old name exists only in the old-named plugin, and the new name
exists only in the new-named plugin. A deprecation warning is issued
when loading the old-named plugin.
* Changed user variables recognized by dump threads:
(When replica connects to source, it sets some user variables on the
connection before issuing the COM_BINLOG_DUMP[_GTID] command.)
master_heartbeat_period
master_binlog_checksum
slave_uuid
rpl_semi_sync_slave
This is an internal change only and compatibility is not an isuse.
* Changed cloud configuration:
In internal/cloud/mysql-test/config/cloud.cnf, we use the new
library in the following line:
plugin-load-add="connection_control.so;semisync_master.so;thread_pool.so"
* Changed packaging definitions:
In all package definition files, we retain the existing semisync
libraries and add the new ones, so that they all get included in
packages.
* Changed help texts:
The help texts are too many to list here. Generally, we replace the
old terms by the new terms, and additionally rewrite the help texts
a bit where they can be improved. See the source patch for the full
list of changes.
DEPLOYMENT AND INSTALLATION
===========================
This adds two new files: semisync_source.so and semisync_replica.so.
They shall both be included in distributions. The existing
semisync_master.so and semisync_replica.so shall remain included in
distributions, too.
SECURITY
========
No security implications.
PROCEDURE TO MIGRATE APPLICATIONS TO NEW TERMINOLOGY
====================================================
If needed, we may publish the scripts that update the identifiers, to
aid in migration.
PROCEDURE TO MERGE UP FROM 5.7
==============================
After this worklog, patches written in 5.7 and merged to 8.0 may need
to rename identifiers. There are several cases:
1. New files, for example test, which use old identifiers. These can
be fixed by running a script after the merge.
2. Modified files, which add new lines that use old identifiers.
These can be fixed by running a script after the merge.
3. Modified files, which modify existing lines that use old
identifiers. These will generate merge conflicts. It seems hard
to use scripts to resolve the conflicts, so this will likely have
to be handled manually.
PROTOCOL
========
Existing behavior
-----------------
When an asynchronous replica connects to a source, it sets the
following user variables. These are user variables (not system
variables), which the dump thread interprets:
@master_heartbeat_period
@master_binlog_checksum
@slave_uuid
When a semi-synchronous replica connects, it checks if the source has
set the system variable @@global.rpl_semi_sync_source_enabled, and
also sets the user variable @rpl_semi_sync_slave, in addition to the
three user variables above.
New behavior
------------
We make the above logic more flexible, so that:
- For all the three user variables used in asynchronous replication,
the replica sets *both* the new-named variable and the old-named
variable when it connects.
- For the system variable checked in semi-synchronous replication, the
replica checks if *either* the new-named variable or the old-named
variable has been set.
- For the user variable set in semi-synchronous replication, the
replica sets *both* the new-named variable and the new-named
variable.
CROSS-VERSION REPLICATION
=========================
Asynchronous replication
------------------------
The changes in the protocol imply the following for *asynchronous*
cross-version replication. Here, OLD refers to a version before this
worklog. NEW refers to a version having this worklog. FUTURE refers
to a future version where the old terminology has been removed.
OLD->NEW: Works because NEW replica sets old-named variables.
NEW->OLD: Works because NEW source accepts old-named variables.
NEW->FUTURE: Works because NEW source accepts new-named variables.
FUTURE->NEW: Work because NEW replica sets new-named variables.
OLD->FUTURE: Unsupported. Does not work beacuse FUTURE sets only
new-named variables and OLD accepts only old-named variables.
FUTURE->OLD: Unsupported. Does not work beacuse FUTURE sets only
new-named variables and OLD accepts only old-named variables.
Semi-synchronous replication
----------------------------
The changes in the protocol, and the way we alias semisync libraries,
imply the following for *semi-synchronous* replication. In the
following, we assume all servers install a semisync library. On NEW,
there are two semisync libraries available, so we use NEW(old_semi) to
refer to a server having this worklog, on which the user has installed
the old-named semisync library, and we use NEW(new_semi) to refer to a
server having this worklog, on which the user has installed the
new-named semisync library. When a NEW server talks with an OLD
server, we require that NEW uses old_semi, and when NEW talks with
FUTURE, we requrie that NEW uses new_semi. The cases are as follows:
OLD->NEW(old_semi): Works because NEW(old_semi) recognizes old-named
system variable and sets new-named user variable.
OLD->NEW(new_semi): Unsupported: user should upgrade all servers
before using new_semi on NEW. Despite that, this works, because
NEW(new_semi) recognizes old-named system variable and sets
old-named user variable.
NEW(old_semi)->OLD: Works beacuse NEW(old_semi) exports old-named
system variable and recognizes old-named user variable.
NEW(new_semi)->OLD: Unsupported: user should upgrade all servers
before using new_semi. Does not work because OLD expects old-named
system variable rpl_semi_sync_source_enabled to be defined, which
NEW(new_semi) does not define.
NEW(old_semi)->NEW(new_semi): Works because NEW(new_semi) recognizes
old-named system variable and sets old-named user variable.
NEW(new_semi)->NEW(old_semi): Works because NEW(old_semi) recognizes
new-named system variable and sets new-named user variable.
NEW(old_semi)->FUTURE: Unsupported: user should enable new_semi before
upgrading to FUTURE. Does not work because FUTURE expects new-named
system variable, which NEW(old_semi) does not define.
NEW(new_semi)->FUTURE: Works because NEW(new_semi) defines new-named
system variables and accepts new-named user variables.
FUTURE->NEW(old_semi): Unsupported: user should enable new_semi before
upgrading to FUTURE. Despite that, this works because NEW(old_semi)
recognizes new system variable and sets new user variable.
FUTURE->NEW(new_semi): Works because NEW(new_semi) recognizes
new-named system variables and sets new-named user variables.
OLD->FUTURE: Unsupported (see asynchrounous replication case)
FUTURE->OLD: Unsupported (see asynchrounous replication case)
mysqlbinlog
-----------
When mysqlbinlog connects to a source server to receive binary logs,
it uses the same logic as an asynchronous replica. Thus, the same
cross-version compatibility constraints apply to the mysqlbinlog
connection as stated above for an asynchronous source-replica
connection.
mysqlbinlog prints one variable that is renamed in this worklog,
namely pseudo_slave_mode. In order for an old replica to process the
output from mysqlbinlog, in this worklog we retain the old variable
name in the output from mysqlbinlog. A possible future transition to
using only the new name is:
- OLD mysqlbinlog prints pseudo_slave_mode and recognizes only
pseudo_slave_mode.
- NEW mysqlbinlog prints pseudo_slave_mode and recognizes both
pseudo_slave_mode and pseudo_replica_mode.
- NEW+1 prints pseudo_replica_mode and recognizes both
pseudo_slave_mode and pseudo_replica_mode.
- NEW+2 prints pseudo_replica_mode and recognizes only
pseudo_replica_mode.
With this chain of changes, each version is able to interpret the
output both from a one version older mysqlbinlog and from a one
version newer mysqlbinlog.
APPLICATION COMPATIBILITY
=========================
Application changes required when upgrading from an older version to a
version having the feature:
- The changes in system variables, command-line options, status
variables, built-in SQL functions, and semisync plugins are fully
backward compatible, since we maintain the old names as aliases. An
application that uses them will work without change, but may observe
new deprecation warnings.
- The changes in peformance_schema instrumentation - thread names,
thread commands, locks, condition variables, thread stages, and
memory allocation - are incompatible. An application that depends
on the exact texts for these will have to be modified.
- The changes in C++ filenames, header guards, and debug symbols are
internal and not part of the user interface, so they do not count as
incompatible. However, future patches written in 5.7 will have to
deal with the renames while up-merging them.
Application changes required before upgrading from a version having
the feature to a future version where the old names are removed:
- The changes in system variables, command-line options, status
variables, built-in SQL functions, and semisync plugins require that
the user replaces the deprecated terminology by the new terminology
after upgrading to the version having the feature and before the
next upgrade.
Application changes on downgrade:
- An application that has upgraded to the version having the feature,
but not started to use the new names yet, can downgrade without
problem.
- An application that has upgraded to the version having the feature,
and started to adopt the new names in configuration files and SQL,
has to first revert to the old terminology before downgrading.
- An application that has upgraded to the version having the feature,
adopted the new names in configuration files and SQL, and upgraded
to the next version, can downgrade to the version where the feature
was introduced. From there, they can revert to the old terminology
and then downgrade to the version before the feature.
INTERNAL CODE CHANGES
=====================
We change other internal code identifiers when it is convenient, for
instance, when a C++ identifier matches a replacement pattern by
chance.
We implement this in several steps. We split the work in three sets
of patches:
1. Refactorings that simplify later steps, and scripts
======================================================
1.1.Various preparation steps
This contains miscelanneous preparation steps needed before the
automatic replacement:
- Eliminate multi-line string literals from the replication code, so
that string literals are easy to find by line-based search tools,
and so that the code formatting looks better. This was found by
searching for lines containing an odd number of double quotes. The
replacements were performed manually.
- Add a copyright header in a file that we are going to modify later,
so that git push won't complain. Since later steps are
auto-generated it's a bit easier to add the copyright header before
them than after them.
- Fix a command-line option that mixes dashes and
underscores (slave-preserve_commit-order), since the rename script
only searches for identifiers with only dashes or only underscores.
1.2.Fix mysqltest's escape function
* mysqltest's escape function should allow multiple escaped characters
mysqltest's escape function takes two arguments. The first one is a
character to escape and the second one is a string in which the
character will be escaped.
The usual use case for escape is to prepare a string for being used in
string context in an SQL statement. Therefore, both the quote
character (e.g. single quote) and the escape character (backslash)
need to be escaped. That requires two calls to escape, which is a bit
cumbersome.
Therefore, we extend the function to allow multiple escaped
characters. The reason we disallowed multiple escaped characters was
to allow any character including comma to be escaped, and still use
comma as a separator between the arguments. We make that work by
requiring that the user places any comma occurring among the escaped
characters first, and make the function look for the separator only
after the first character.
* Adapt test utilities to use the escape function
- assert.inc, eval.inc: These should have used 'escape' from the
beginning, so the caller wouldn't have to bother escaping strings.
Now we have too many tests using them and escaping strings, so
making it an option.
- escape_sql.inc: This was a workaround for not having proper 'escape'
function in the language. Removed it now.
- show_slave_status.inc: Updated it to use the escape function instead
of escape_sql.inc
1.3.Fix min/max values for sysvars with block_size!=1
Background:
System variables have a feature known as "block_size". Numeric system
variables can only be set to values that are multiples of block_size.
Most variables use a block size of 1. Numeric variables are
additionally limited to a maximum and a minimum value.
Problem:
In case the minimum and maximum values are not specified as multiples
of the block_size, the effective minimum and maximum are different
than the ones specified. This typically happens when using a
block_size that is a power of two, for instance 1024, and a maximum
that is a numeric limit such as MAXINT, which may have values such as
2^64-1, i.e., one-less than a multiple of the block_size.
This has no observable effect, other than the maximum value displayed
in performance_schema.system_variables_info being a value that is not
actually allowed to use for the variable.
This is a litte inconsistent, and also made it hard to write the
scripted tests in later steps of the worklog.
Also, the indentation was wrong all over Sys_var_integer.
Fix:
- Round the maximum down to the next smaller multiple of block_size.
- Round the minimum up to the next grater multiple of block_size.
- Fix the indentation.
1.4.Scripts
This contains the scripts used to execute the following steps. See
00README for an overview.
These scripts are only kept in the server branch during the
development of WL#14194. They will not be pushed to the main
repositories, but possibly published elsewhere in case it eases
migration for anyone.
2. Scripted replacements
========================
Perform all automated search-and-replace operations.
The following replacements are performed this way:
2.0. Replace "master" by "mistir" and "slave" by "slivi" in contexts
where we want to preserve the old terms. We replace this back
in the last search-and-replace operation. This hides it from
the other search-and-replace operations.
2.1. system variables
2.2. command-line options for mysqld
2.3. command-line options for mysqlbinlog
2.4. command-line options for mysqldump
2.5. command-line options for mysqladmin
2.6. SQL functions
2.7. status variables
2.8. thread commands
2.9. instrumented locks
2.10. instrumented condition variables
2.11. thread stages
2.12. thread names
2.13. instrumented memory allocations
2.13. C++ filenames
2.14. header guards in C++ header files
2.15. debug symbols
2.16. semisync
2.17. user variables used in the replication protocol handshake
2.99. Replace "mistir" by "master" and "slivi" by "slave"
Each of these categories is committed in a separate step.
This includes some C++ identifiers and mtr variables that contain the
replaced variables as substrings. This is only good.
It is necessary to rename filenames, since some .inc files in the test
suite contain e.g. system variable names in their names, so the place
where they are sourced will have their names changed.
3. Manual replacements, aliases, compatibility code, and tests
==============================================================
3.1.Fix tests
Fix test failures introduced by previous patch.
There were two tests that wrote variable names in alphabetic order to
their result files. So we re-record those result files.
3.2.Help texts
Remove the words "master/slave" from help texts for command line
options and system variables.
The search is automatic, finding occurrences of 'master' and 'slave'
within double-quoted strings, in sys_vars.cc mysqld.cc, and
semisync_*_plugin.cc (for server variables and command-line options),
and in mysqlbinlog.cc, mysqladmin.cc, mysqldump.cc (for client
command-line options).
However, the replacement is manual. Therefore we also clarify and
correct some texts. Since help texts rarely occur in the test suite,
this will not have too much unpredictable consequences: we just need
to update the result files for mysqld--help-[not]win.
We also fix some other texts occurring in these files, since it is
easier to do it now than to postpone it:
- Error messages in utilities. These usually don't occur in tests.
- Plugin description strings for the semisync plugins.
- Deprecation warning for --master-retry-count.
- Comments printed by mysqldump.
- Source code identifiers in mysqladmin.cc.
3.3.Aliases for command line options
For all renamed command-line options, we re-introduce the old name as
a deprecated alias. This includes command-line options in mysqladmin,
mysqlbinlog, mysqldump, and the server. For the server, it only
includes command-line options that are not defined through a system
variable.
3.4.Aliases for status variables
For all renamed status variables, we re-introduce the old name as an
alias. This does not include semisync status variables.
3.5.Aliases for semi-sync libraries and variables
* Server changes:
For the semisync libraries, we previously renamed both the filename,
the plugin name, the system variables defined in the libraries, and
the status variables defined in the libraries.
Now we create aliases for these as follows. We duplicate the
libraries so there exists one library having the old filename, the old
plugin name, the old-named system variables, and the old-named status
variables. We generate a deprecation warning only when the old
libraries are loaded, not when using the old-named variables.
Additionally, we generate an error when using the two duplicated
source-side libraries or the two duplicated replica-side libraries on
the same server.
* Package definition changes:
To make the packages aware of the new library, we update the following
files:
- mysql-test/include/plugin.defs
- packaging/**/*
* Test suite changes:
- Add a test utility to disable/restore binlogging in the current
session. Changed files:
- disable_binlog.inc
- restore_binlog.inc
- Add a test utility to use a JSON object as a key-value store and
lookup the value when using a mysqltest $variable as a key. New
file:
- json_lookup.inc
- Add test utilities to assert that some specific messages were
written to the error log.
New test utilities:
- save_error_log_position.inc: Store the current position in the log
in an internal variable.
- assert_error_log.inc: Assert that a given message (or sequence of
messages) was written to the log since the last saved position.
- show_error_log.inc: Show all messages generated since the last
saved position.
The above utilites need to respect the global error suppressions
defined in mtr_warnings.inc. To make use of that, we refactor
mtr_warnings.inc and related files. Changed files:
- check-warnings.test
- load_error_log.inc
- mtr_warnings.sql
A result file had to be updated, since the test printed all tables
in the mtr database to the result log, and we needed another one in
order to implement the utilities above:
- default_row_format_01.test
- Add test utility that simplifies the use of
mtr.add_suppression. This automates the suppression of messages on
all servers in a replication topology, and the disabling/re-enabling
of binary logging on the current session. Changed files:
- suppress_messages.inc
- Add test case for the feature: rpl_semi_sync_alias.
3.6.Cross-version-compatible connection user vars
When the replica connects to the source, it first connects, then sets
some user variables using regular SQL SET statements, and then starts
replication. We previously renamed some of those user variables.
Now we create aliases for those user variables in order to make
cross-version replication work, as follows:
- On the replica side, we set both the old-named variable and the
new-named variable.
- On the source side, we accept either the old-named variable or
the new-named variable.
3.7.Aliases for SQL functions
This creates the deprecated alias MASTER_POS_WAIT for the renamed SQL
function SOURCE_POS_WAIT.
3.8.Aliases for system variables
We previously renamed several system variables. Now we introduce a
framework for defining aliases for system variables, and define
deprecated aliases for all the renamed system variabes.
The framework consists of a subclass of Sys_var called
Sys_var_deprecated_alias. This class represents an alias for some
other Sys_var. It copies most member fields of that variable, so
setting one of the variables changes the value of both. We also make
it so that SET PERSIST writes the name of both variables.
When loading persisted variables, when either the alias or the real
name appears in the file, it loads both of them into the memory cache.
This ensures that both the real variable and the alias will be
persisted the next time the persisted variable file is written, which
ensures backward and forward compatibility in future upgrade/downgrade
scenarios.
In order to load the aliases while loading persisted variables, we
need to access the system variable framework (which contains the alias
definitions) while loading persisted variables. To do that, we have
to move the initialization of the system variable framework a bit
earlier - hence the change in mysqld.cc. This is only the
initialization of the framework, which makes it possible to access
metadata about the variable definitions; it is not the initialization
of system variable values. The system variable framework
initialization is self-contained and does not depend on other things,
so this change should not have any other side effects.
When using RESET PERSIST, we now reset both the variable and the
alias.
When handling command-line options, we make the old name generate a
deprecation warning. This is done in mysqld.cc:get_one_option.
We need changes to make the "variable_source" information accurate
when using an alias on the command line. (The variable_source is used
to populate columns of the performance_schema.variables_info table.)
When an alias is used on the command line, the command-line parsing
framework can only store the variable_source for the alias, since it
does not know that the alias is an alias. Therefore, we copy the
variable_source from the alias to the base variable from code that has
this information; namely, in mysqld.cc:get_one_option.
We remove the function clear_user_host_timestamp since it was unused,
and otherwise we would have to define it in Sys_var_alias.
We make sys_var::set_and_truncate return early if source and
destination are identical. Otherwise Valgrind complains. This happens
in the new call to sysvar->set_source_name from
mysqld.cc:mysqld_get_one_option.
3.9.Tests for system variable aliases
1. Test that the system variable aliases introduced in the previous
commit work as expected.
To make this less repetitive, introduce a framework to test system
variables using a declarative syntax. To make that framework
possible, we introduce another, lower-level framework, which
provides control flow based on JSON documents: in particular, it
allows test cases to iterate over JSON arrays, and unpack JSON
objects into a set of mtr variables. Relying on this JSON
framework, we can define properties of system variables using
JSON, and then iterate over all such definitions, unpack the
properties of the sysvars into mtr variables, and invoke the
higher-level framework to test the system variable.
The high-level framework is described in
mysql-test/suite/sys_vars/inc/control_for_sysvars_with_aliases.inc.
The JSON specifications are in
mysql-test/suite/sys_vars/inc/json_sysvar_specs.inc.
The JSON framework is in
mysql-test/include/create_json_iterator.inc,
mysql-test/include/create_json_unpacker.inc, and
mysql-test/include/create_unpacking_json_iterator.inc.
Persisted variables are tested in
mysql-test/suite/sys_vars/t/persisted_sysvars_with_aliases.inc
Also minor improvements in assert.inc and eval.inc.
Also added:
mysql-test/include/check_64bit_machine.inc,
mysql-test/include/check_64bit_ulong.inc,
mysql-test/include/have_32bit_ulong.inc, and
mysql-test/include/have_64bit_ulong.inc,
and updated:
mysql-test/include/have_32bit.inc and
mysql-test/include/have_64bit.inc
The new funcationliaty was necessary because some 64-bit windows
platforms use only 32 bits for ulong.
Also extended sys_vars.pseudo_replica_mode_basic. This is a
session variable, and therefore not covered by the new framework.
We could have written framework for session variables, but it's
too much work for just one variable.
2. Fix tests that broke when adding aliases:
binlog_nogtid.binlog_persist_only_variables:
- This test used wildcards like %master% and %slave% to select all
replication-related system variables from performance_schema
tables. The automatic replacements changed this to %source% and
%replica%, which made it match a few more variables, so the test
started to fail already in that step.
- After adding aliases, we need to make the match %master% and
%slave% again, in addition to %source% and %replica%, to include
all the aliases.
- We also need to modify the logic of the test when the test
executes RESET PERSIST. Before, it ran RESET PERSIST once for
each variable. However, when doing RESET PERSIST for a variable
having an alias, it will reset both the variable and the alias.
Then when the test issues RESET PERSIST for the alias, it has
already been reset and generates an error. Fixed by excluding
aliases from this part of the test.
persisted_variables_extended:
- This test used SET PERSIST replica_net_timeout and then
SELECT ... FROM performance_schema.persisted_variables ...
Therefore we update the result file to include the alias.
mysql-test/suite/binlog_nogtid/r/binlog_persist_variables.result
mysql-test/suite/binlog_nogtid/r/binlog_persist_variables.test
- This test counts the number of variales. Updated accordingly.
- The test lists replication variables, heuristically guessing
which variables are replication variables using a pattern.
Updated the pattern to include the aliases.
- Updated the result file to include the aliases and their
deprecation warnings
- The result file was auto-updated by the scripted renames.
It contains a list of variables in alphabetic order.
The rename broke the alphabetic order, so updated the
result file accordingly.
mysql-test/r/read_only_persisted_plugin_variables.result
mysql-test/r/read_only_persisted_variables.result
mysql-test/r/reset_persisted_variables.result
mysql-test/r/variables.result
mysql-test/suite/perfschema/r/persisted_variables.result
mysql-test/suite/perfschema/r/variables_info.result
mysql-test/suite/sys_vars/r/all_vars.result
mysql-test/t/all_persisted_variables.test
- These tests iterate over all variables, or some subset of
them, or assert that the number of variables is equal to some
constant. Updated result files to include the aliases.
mysql-test/r/mysqlcheck.result
mysql-test/r/partition_innodb_tablespace.result
mysql-test/suite/funcs_1/r/is_key_column_usage.result
mysql-test/suite/funcs_1/r/is_key_column_usage_ci.result
mysql-test/suite/funcs_1/r/is_statistics_ci.result
mysql-test/suite/funcs_1/r/is_statistics_cs.result
mysql-test/suite/funcs_1/r/is_table_constraints.result
mysql-test/suite/funcs_1/r/is_table_constraints_ci.result
mysql-test/suite/innodb/r/case_insensitive_fs.result
mysql-test/suite/innodb/r/choose_tbsp_location-discard.result
mysql-test/suite/innodb/r/create_tablespace.result
mysql-test/suite/innodb/r/create_tablespace_16k.result
mysql-test/suite/innodb/r/create_tablespace_32k.result
mysql-test/suite/innodb/r/create_tablespace_4k.result
mysql-test/suite/innodb/r/create_tablespace_64k.result
mysql-test/suite/innodb/r/create_tablespace_8k.result
mysql-test/suite/innodb/r/create_tablespace_debug.result
mysql-test/suite/innodb/r/create_tablespace_replication.result
mysql-test/suite/innodb/r/default_row_format.result
mysql-test/suite/innodb/r/default_row_format_compatibility.result
mysql-test/suite/innodb/r/default_row_format_tablespace.result
mysql-test/suite/innodb/r/discard_tablespace.result
mysql-test/suite/innodb/r/hidden_directory_dotfile.result
mysql-test/suite/innodb/r/hidden_directory_win.result
mysql-test/suite/innodb/r/innodb-index-online-fk.result
mysql-test/suite/innodb/r/innodb_tablespace.result
mysql-test/suite/innodb/r/known_dir_unique_undo.result
mysql-test/suite/innodb/r/known_directory.result
mysql-test/suite/innodb/r/partition_autoinc.result
mysql-test/suite/innodb/r/portability_tablespace.result
mysql-test/suite/innodb/r/rename_table.result
mysql-test/suite/innodb/r/tablespace_per_table.result
mysql-test/suite/innodb/r/unknown_directory.result
mysql-test/suite/innodb_fts/r/tablespace_location.result
mysql-test/suite/innodb_fts/r/tablespace_location_error.result
mysql-test/suite/innodb_zip/r/16k.result
mysql-test/suite/innodb_zip/r/4k.result
mysql-test/suite/innodb_zip/r/8k.result
mysql-test/suite/innodb_zip/r/restart.result
mysql-test/suite/innodb_zip/r/zip.result
mysql-test/suite/parts/r/partition_reorganize_innodb.result
- These tests count or list all tables defined. We added two
tables in the framework for assert_error_log.inc in step 3.5,
so updated the result file.
mysql-test/include/mysqld--help.inc
- This has a list of variables whose default is nondeterministic,
and one of the aliases had to be added there.
mysql-test/r/non_table_atomic_ddl.result
mysql-test/r/trigger_mdl.result
mysql-test/suite/binlog/r/binlog_truncate_kill.result
- The removal of begin_include_file.inc from filter_file.inc
also got rid of a 'connection ' printout from
the result log in tests using show_binlog_events.inc.
Updated result files.
3.10.Justify use of old term in mysqlbinlog
Make mysqlbinlog print a comment to clarify why we are still using the
old terminology.
Copyright (c) 2000, 2023, Oracle Corporation and/or its affiliates. All rights reserved.