WL#1697: Multi-source replication
Affects: Server-5.7
—
Status: Complete
Executive Summary
==================
Multi source replication, will enable a replication slave to have multiple
masters. This feature does not do any additional conflict detection and resolution.
This feature will be useful when the user/DBA wants to commission
a Fan in slave for data aggregation. After implementation of this feature
it will be possible to configure all possible topologies using MySQL replication.
NOTE: This feature differs from Multi-Master Replication, which is an
update Everywhere architecture, where all the nodes act as master and replicate
to all other nodes in the cluster.
FULL DESCRIPTION
=================
A Slave server in MySQL replication: (till 5.6)
1. Receives transactions (in the form of binary log events) from a single
server instance (called Master) via a single connection.
2. Applies these received transactions.
Multisource replication(MSR) feature for a slave server aims to:
1. Receive transactions (in the form of binary log events) from several
MySQL servers simultaneously via several connections.
2. Apply these received transactions whilst not doing any additional conflict
detection or conflict resolution.
Example: Consider the case of replication from three sources to a single slave
below(The downward arrow is the data flow through a TCP/IP connection)
+--------+ +--------+ +--------+
| mysqld | | mysqld | | mysqld |
+----v---+ +----v---+ +----v---+
v v v
v v v
v v v
v v v
+-v---------------v--------------v--+
| mysqld Slave |
+-----------------------------------+
USES
=====
1. For the aggregation of disjoint data from many servers.
a) Mainly used for backup purposes.
b) Merging of table shards: Multiple shards of table can be joinly replicated
to form a single table in a slave. If all the shards are replicated to
form a single global table, it is easier for the application to replace
the complex cross shard joins, since all the data is located in one
place only. Mainly used for BI and data analytics purposes.
2. To make it "easier" to deprecate Federated storage engine, as MSR could be
used to transfer the data from several servers.
3. Decrease in latency in circular replication. Consider the case of circular
replication M1->M2->M3->M4->M1. Latency could be decreased if all these
four servers form a cluster. For example: Updates from M2 to M1 could be
reached directly from M2 without intermediate servers M3 & M4.
Terms and Definitions
----------------------
* Source: Source for a transaction T for a particular server, is classified
into following types:
** A server where transaction T is generated : This is a server where a
transaction T was originally executed by the user. In other words, this
is a source where the GTID for a transaction T was first generated.
** A server where transaction T is relayed : In a hierarchical setup
consisting of M1->M2->S, the server M2 relays the transactions generated
on M1 to S.
** A server where transaction T1 is relayed and transaction T2 is generated.
Combination of above both types.
For ex: in the setup M1->M2->M3->S, on M2, new transactions are generated
and transactions from M1 are relayed.
* Multisource replication through hierarchical setup: MySQL 5.6, provides a way
to aggregate data from several sources through hierarchical setup i.e
M1->M2->...Mn->S. In this case, the slave would have the
transactions generated from several sources {M1, M2...Mn} but except
for M1, all other servers acted as relayed servers.
* Master: Traditionally, Master has been used to refer to any type of sources
mentioned above.Current replication (5.6) can support only one master
per slave and all the replication commands act on that "master". So,
redefining master is prone to confusion at this point.In this document,
master and source mean the same.But we have to make distinction between
Multi Source Replication and Multi Master replication.
* Multi Master replication: Implicity, this means Multi Master Update Everywhere
replication setup. In this setup all the N servers form a cluster
(i.e each server is connected to other N-1 servers). Updates could be run
on any of the server and these are replicated to other remaining servers.
This setup should maintain ordering of the transactions,
do transaction conflict detection, transaction conflict resolution
(or rollback)
* Connection: A connection between a slave and it's immediate source.
* Channel: When a slave has a single source (till mysql 5.6), the commands by
default act on that source only.But when a slave has several sources
(through WL#1697), we need to have an abstraction for
source>IOthread->SQLthread so the commands can act/control each source.
This abstraction is called a Channel. In short, a channel name (i.e
string) acts as an abstraction for a source-receiver-applier path.
According to this definition, connection is a *part* of the channel.
+-----------+ +-------------+ +-------------+
|dump thread|>>>>>>>>| IO thread |>>Relay log>>| SQL thread |
| | | | | |
+-----------+ +-------------+ +-------------+
<-connection->
<------------------------channel-------------->
DESIGN CONSIDERATIONS
=====================
The following are the "sub systems" that are considered for design.
1. Event Receiver System (ER)
2. Event Queueing System (EQ)
3. Event Application System (EA)
4. User interfaces.
4.a Startup options
4.b commands through mysql client.
5. Status reporting and Monitoring through P_S replication tables.
See 'decision_design.txt' attached to this worklog, to find out the design
chosen for this worklog. The design chosen is described in the points 1, 2 & 3
below
1.Event Receiver System (ER)
=========================
a) Spawn an IO thread for each channel
2.Event Queueing System
=====================
b) Every IO thread writes the events it received to it's own relay log.
i.e a channel has it's own relay log.
3.Event Application System
=========================
slave_parallel_workers=0
-------------------------
a) Spawn an SQL thread for each channel, which applies the events read from the
relay log belonging to that channel only.
slave_parallel_workers>1
------------------------
b) Each channel will have it's own coordinator.
c) Each coordinator will have it's own sets of workers. Transitively,
this implies that there are dedicated set of workers for each channel.
Diagrammatically
----------------`
slave_parallel_workers= 0
-------------------------
+-----------------------------------+---------------------------+
| | |
+-----+ | +-----------------------+ | |
| | | | Relay log per | | +------------------+ |
| M1 |>>>>>>>| | source |>>>>>>>>>>>| SQL thread | |
+-----+ | +-----------------------+ | +------------------+ |
| | |
+---------------------------------------------------------------+
| | |
| | |
+-----+ | +----------------------+ | +-----------------+
| | | | Relay log |>>>>>>>>>>>>| SQL thread | |
| M2 |>>>>>>>| | | | +------------------ |
+-----+ | | source | | |
| +----------------------+ | |
| | |
| | |
+---------------------------------------------------------------+
+-----+ | | |
| M | | +----------------------+ | |
| |>>>>>>>| | Relay log | | +----------------+ |
+-----+ | | per source |>>>>>>>>>>>>| SQL thread | |
| +----------------------+ | +----------------+ |
| | |
| | |
+-----------------------------------+--------------------------+
slave_parallel_workers > 0
--------------------------
+-------------------------+-------------------------------------+
| | +------+ |
| | | Wi | |
| | +------+ |
+-----+ | +-------------+ | +------+ +------+ |
| M1 | | | Relay Log | | | | | Wj | |
| |>>>>>>>>>>>| Per |>>>>>>>>>>| C | -------+ |
+-----+ | | Source | | +------+ +--------+ |
| +-------------+ | | Wk | |
| | +--------+ |
+---------------------------------------------------------------+
| | Coordinator +--------+ |
| | and set of | Wi | |
| | workers per +--------+ |
+-----+ | +------------+ | Source |
| M2 | | | Relay Lg | | +------+ +-----+ |
| |>>>>>>>>>>>>| Per |>>>>>>>>>>>| C | | wj | |
+-----+ | | Source | | | | +-----+ |
| +------------+ | +------+ +------+ |
| | | Wk | |
| | -------+ |
+---------------------------------------------------------------+
| +------------+ | +-------+ |
+-----+ | | Relay log | | | Wi | |
| M3 | | | Per | | +-------+ +-------+ |
| |>>>>>>>>>>>>| Source |>>>>>>>>>>>| C | |
+-----+ | +------------+ | | | +-----+ |
| | +-------+ | Wj | |
| | +-------+ +------+ |
| | | Wk | |
| | +-------+ |
| | |
+-------------------------+-------------------------------------+
4.User interfaces
================
See High level specification for details.
5. Slave status and Monitoring support
======================================
Capabilities for reporting channel configuration and execution status
will be done through performance schema tables for replication by adding a
channel column for each row
FUNCTIONAL REQUIREMENTS
=======================
It is a requirement:
R1: To implement Fan-in replication from N servers.
See BUG#1697: REQUEST TO IMPLEMENT MULTI-SOURCE REPLICATION
R1.a: A corollary, of this requirement is a setup of any
replication topology shall be possible.
R2: That there shall be no deviation from the current replication
behaviour(till mysql 5.6), when the user does not use multisource
replication feature
R2.a: Backward compatibility of existing interfaces will be maintained.
R2.b: Not to break external utitiles (MySQL utilities) and MySQL cluster
(ndb_mi.cc)
R3: To show the replication status for each channel in Performance_Schema
replication tables.
R4: To work when slave_parallel_workers == 0 and > 0 (MTS mode)
R5: To stop replication only for a channel, on which a conflict has occured.
R6: To support GTIDs as well as binary log positions
LIMITATIONS
===========
L1: This feature works only when the replication repositories are of type TABLE.
NON FUNCTIONAL REQUIREMENTS
===========================
NR1: No performance drop for a single channel replication
NR2: Slave throughput shall increase with the increase in number of sources.
NON-REQUIREMENTS
===========================
It is NOT needed :
NE1: To add any extra conflict detection support
(i.e use whatever is already present)
NE2: To do any conflict resolution if a conflict occurs.
NE3: To add an extra binlog event on a source's binary log or change binary
log format.
It follows from both the above 3 points that,the application developer or
a DBA should partition the data so that it is disjoint for a successful
fan in replication.
WL#2271: Remove my.cnf replication options (5.1)
WL#2385: Multi-source testing
WL#2775: System tables for master.info, relay_log.info
WL#7817: RPL Monitoring: Move status variables to replication P_S tables
WL#2385: Multi-source testing
WL#2775: System tables for master.info, relay_log.info
WL#7817: RPL Monitoring: Move status variables to replication P_S tables
This section is organized into
A) Startup options
B) Naming conventions.
a) channel name
b) file names
C) MySQL client commands
D) System variables
E) Slave status variables
F) Error Messages, warnings and notes
G) Compatibility
a) backward compatibility
b) Upgrade from 5.5 and 5.6 slave to 5.7 slave with MSR
H) Replication Filters
I) Semisync replication
J) Slave status
a) SHOW SLAVE STATUS
b) Performance schema tables
Default channel
----------------
Before diving into the specification, the definition of
default channel is to be cleared. Multi source replication
introduces creation of new channels to replicate from several sources.
To deal with backward compatibility (where the user does not want to use
this feature and want to replicate from a single source just like till 5.6),
this WL creates the notion of a default channel.
This default channel is an empty string, i.e ''. Here are the properties of a
default channel:
- Unlike other channels,(which can be created and destroyed by the user)
the default channel always exists: It is created when the server starts
and it cannot be removed by the user.
- So, if a user wants to use replication commands till 5.6,(i.e user completely
ignores about multisource replication) all his commands would act on a default
channel. This clearly supports backward compatability.
- However, if multiple channels are created by the user on this slave, behaviour
of commmands varies i.e a user can no longer expect all his/her till 5.6 commands
would act only on a default channel.
To completely know about behaviour of commands, see command MySQL client command
section.
In short after the implementation of this feature.
-> A replication command (till 5.6) will act on a default channel if no
extra channels exist. This presevers backward compatablity.
-> A default channel is implicitly an empty string.
Maximum Number of channels on a replication slave
-------------------------------------------------
Currently, the maximum number of channels is set to 256.
It is hard coded to avoid unnecessary start up options.
A)Startup options
=================
For this WL, there are no startup options. However, there are startup options
which are modified. They are described in the next section.
* options that are applicable to *ALL* channels
-------------------------------------------------
Following are the startup options with a brief descrption of how
they would fit in this feature. The ones which does not have
the description below, have the usual meaning applied to all
channels.
# --log-slave-updates
This option means that *ALL* the transactions that are being received
by the slave from several sources shall be logged in the binary log.
# --log-slow-slave_statement
# --log-warnings
# --relay-log-purge
When set, all channels purge their own relay logs automatically
# --relay-log-recovery
# --report-host
# --report-user
# --report-password
# --slave-compressed-protocol
# --relay-log-info-repository
This has to be TABLE for MSR. Otherwise, addition of second channel
fails with --ER_SLAVE_NEW_CHANNEL_WRONG_REPOSITORY
# --master-log-info-repository
This has to be TABLE for MSR. Otherwise, addition of second channel
fails with --ER_SLAVE_NEW_CHANNEL_WRONG_REPOSITORY
# --sync_relay_log
# --slave_transaction_retries
This is yet to be implemented for slave_parallel_workers >= 1
# --relay-log-recovery
# --skip-slave-start
A slave for all channels does not start.
# --slave-load-tmpdir
# --slave-sql-verify-checksum
# --slave_type_conversions
# --slave-skip-errors
This means execution continues and errors are skipped for *ALL* channels.
# --slave-exec-mode
* options that are applicable per channel
----------------------------------------------
Since the following options are set up as startup options, all the channels
will have the same value even though these options act on a single channel.
In the future, an interface could be built (probably through CHANGE MASTER)
so that the user can change these values per channel.
For example: consider this option,
--relay-log-space-limit: If this option acts on *all* the
channels, the meaning would have been, "the upper limit
on the total size of relay logs of *all* channels".
If this option acts on a single channel, the meaning is
"the upper limit on the total size of relay log per channel"
# --max-relay-log-size=
Denotes the maximum size of relay log file per channel.
# --relay-log-space-limit=
Denotes the upper limit of relay log size per channel
# --slave-parallel-workers=
The number of slave parallel workers per channel.
# --slave-checkpoint-group=#
# --slave-pending-jobs-max-size=#
# --slave-checkpoint-period=#
# --slave-net-timeout=
Denotes the waiting time by an IO thread for each source.
# --relay-log-index=filename
Denotes the base name of relay log index per channel
See, Naming conventions section below
# --relay-log=filename
Denotes the base name of relay log per channel
See, Naming conventions section below
# --slave_net-timeout=N
This value is set per channel i.e each channel will wait for N seconds
to check for a broken connection
# --slave-skip-counter= N
This value is set per channel i.e each channel will skip N events
from its master.
* options that act if and only if empty stringed defaul channel exists.
otherwise there is an error generated.
---------------------------------------------------
# --master-info-file=
This is required for FILE based repository and MSR requires
TABLE
# --relay-log-info-file=
This is required for FILE based repository and MSR requires
TABLE
Above options mean that, if there are multiple channels (ofcourse
in TABLE type repo, then the above options do not make convert
to FILE type repo. Instead there will be an error.)
B) Naming conventions
=====================
* channel name
---------------
# User shall specify in the commands of a particular channel.
It is a string with a maximum allowed of 64 characters.
# The constraints are same as DB names in lower_case_table_names=1 systems.
This means that channel names are case insenstive. (For ex: CHANNEL1
&& channel1 are same).
# since channel names are also part of slave tables, we have to choose
the char set for channel names (as they exist as column names in the
mysql.slave_master_info, mysql.slave_relay_log_info, mysql.slave_worker_info
tables)
This means that system_charset or server_charset has virtually no role in
deciding the channel_name charsets.
# For the above reason, the hardcoded channel name is chosen to be utf8 with
COLLATION as utf8_general_ci (to support case insensitivity)
# default channel is an empty string.
* File names
---------------
The files that are assosiated with the slave replication are
info files, relay-log index file and relay log files. The following describes
the naming of files per channel and how start up options effect the naming of
files.(There is no change in the binlog name behaviour for a multisourced slave)
To distinguish relay-log files and indices of a particular channel, a
distinguisher -
which is a function of channel name - will be a part of these files.
Till 5.6, these files are named as BASE_NAME.index (or HOSTNAME-relay-bin.NNNNN)
and BASE_NAME-relay-bin.NNNNN (or HOSTNAME-relay-bin.NNNNN)
BASE_NAME is specified by relay-log-index or/and relay-log startup options.
For this WL, an extension '-channel_name' is added to the
relay log files, where 'chanenl_name' lower case of the channel name
provided by the user. For example: If channel name ShannonNoisyChannel then,
# relay log index file: BASE_NAME-shannonnoisychannel.index
# relay log files: BASE_NAME-shannonnoisychannel.00000x
where BASE_NAME follows the same semantics of mysql 5.6
In the case of default channel (i.e an empty string), there is NO
EXTENSION added to the relay log files.
# In case if channel has special characters like (-,$,? etc) these
characters are converted to a filesystem charset to get support from
the Operating system to create and open the relaylog files.
When File names and channel names are considered together, then the
net result is:
# channel_names are case insensitive
# channel names can have special characters. However, these special characters
would get converted to file system charset for relay log file names.
# default channel ("") does not have any extension to support backward
compatability.
B). MySQL client commands
=======================
General rules for the commands
==============================
1. Add FOR CHANNEL '' to all the replication commands that
support channel specificity. In this case, this particular command
acts only on that channel specified.
2. For any command, if a specified channel does not exist to act on the command,
following error is displayed on the client and *server log*
--ER_SLAVE_CHANNEL_DOES_NOT_EXISTS
"Slave channel "" does not exist on the slave"
3. To support backward compaitbility and also for the users who are not
interested to use this feature (i.e if FOR CHANNEL ''
is not given by the user) the command behaviour depends on the type
of command.
a) There are commands for which without specifying a channel would act on
*ALL* channels. For example: START SLAVE/STOP SLAVE/SHOW SLAVE STATUS/
RESET SLAVE/FLUSH RELAYLOGS
b) There are commads for which without specifying a channel does not make
sense if there are multiple channels.
Example: CHANGE MASTER/ SHOW RELAYLOG EVENTS/ master_pos_wait(),
gtid_set_wait() etc. Remember that default channel always exists.
so, if a user has created a new replciation channel, this category of
command needs Channel name specified to them always.
Based on the above reasoning, we categorize them in to following two
types.
Commands that can act only on single channel if the clause
FOR CHANNEL '' is not specified.
----------------------------------------------------------------------
C1: CHANGE MASTER TO master_def.... FOR CHANNEL
i) Updates the members of the master_info object for the channel_name if
the channel already exists on the slave.
ii) If the channel does not exist, a Master_info and Relay_log_info object
for that channel is created and flushed to the repository.
Notice that, --ER_SLAVE_CHANNEL_DOES_NOT_EXISTS error is not thrown for
this command.
iii) Following both commands shall be possible:
* CHANGE MASTER TO MASTER_NAME=, MASTER_PORT=
FOR CHANNEL
* CHANGE MASTER TO MASTER_NAME=, MASTER_PORT=
FOR CHANNEL
i.e one could change the host and/or port for the same channel.
iv) Following both commands shall not be possible:
* CHANGE MASTER TO MASTER_NAME=, MASTER_PORT=
FOR CHANNEL
* CHANGE MASTER TO MASTER_NAME=, MASTER_PORT=
FOR CHANNEL
The second command will fail with the following error:
--ER_SLAVE_MULTIPLE_CHANNELS_HOST_PORT
"Multiple slave channels for the same host and port combination"
v) If the FOR CHANNEL is not specified, and if there are more then one
channel on the slave, then the command errors out with
--ER_MULTIPLE_CHANNELS_CMD "Multiple channels exist. Cannot choose a
channel to act. Use FOR CHANNEL clause to
choose a channel'
C2: SHOW RELAY LOG EVENTS FOR CHANNEL ''
i) If FOR CHANNEL is not specified and if there channels>=2, we exit with
--ER_MULTIPLE_CHANNELS_CMD
C3: SELECT MASTER_POS_WAIT(('master_log_file', master_log_pos
[, timeout] [,channel_name])
i) if channel_name is not specified and multiple channels exists, then
--ER_MULTIPLE_CHANNELS_CMD is issued.
C4: SELECT WAIT_UNTIL_SQL_THREAD_AFTER_GTIDS('gtid_set',[time_out [, channel_name])
i) if channel_name is not specified and multiple channels exists, then
--ER_MULTIPLE_CHANNELS_CMD is issued.
NOTE: We create default channel all the time.
Commands that act on *ALL* channels if FOR CHANNEL <'channel_name'> is not
provided
--------------------------------------------------------------------------
C5: STOP SLAVE [thread_types]
i) This will stop slave for all channels.
ii) STOP SLAVE [thread_types] FOR CHANNEL <'channel_name'> will stop
slave only for that channel.
C6: START SLAVE [thread_types]
i) Starts slave for all channels.
ii) START SLAVE [thread_types] [until_option] [connection_options] FOR
CHANNEL <'channel_name'>
will start slave for <'channel_name'>
iii) NOTE:START SLAVE [thread_types] [until_option] [connection_options]
will act only on a single channel.
But if there are multiple channels, this commmand errors out with:
--ER_SLAVE_MULTIPLE_CHANNELS_CMD "Multiple channels exist.
Cannot choose a channel to act.
Use FOR CHANNEL clause to choose a
channel'
C7: RESET SLAVE [ALL]
i) Reset the slave for all channels. All the relay log files and index files
per named channels are also deleted.
ii) [ALL] clause deletes the master info structures of all channels too
except default channel. This means, SHOW SLAVE STATUS after a
RESET SLAVE ALL should be empty.
iii) RESET SLAVE [ALL] FOR CHANNEL <'channel_name'> will act only on a
particular channel.
C8: SHOW SLAVE STATUS
i) Shows the slave status for all channels
ii) SHOW SLAVE STATUS FOR CHANNEL <'channel_name'> will act only on that
channel_name
Removed command
=================
C9: SHOW SLAVE STATUS NONBLOCKING
i) This command has no meaning for multisource replication as now
{mi,rli} pair i.e
slave info objects can be deleted. So, the way this command has
implemented could create server crashes. This command will be removed.
(See LLD for the reason).
However, WL#??? will make SHOW SLAVE STATUS non blocking implicitly.
D) System Variables
========================
These are the options for a replication slave server are dynamic.
(change at run time using SET GLOBAL
# master_info_repository or relay_log_info_repository
i) Not allowed to be set as FILE when the number of channels > 1
ii) Not allowed to be set as FILE when the number of channels == 1
and channel name != ''
iii) In the above both cases, an error --ER_SLAVE_WRONG_REPOSITORY is put
in the thread diag area
# Rest of the variables shall be applicable to all channels.
E) slave status variables
========================
All these status variables act on default channel. To know, the status of other
channels, the user has to lookinto performance schema tables.
Show status like '%Slave_running%'
Show status like '%Slave_retried_transactions%'
Show status like '%last_received_heartbeat%'
Show status like '%received_heartbeat%'
show status like '%heartbeat_period%'
F) Error messages
===================
Error messages should indicate the channel on which a particular thread, failed.
* The following are the error variables introduced.
i) --ER_SLAVE_CHANNEL_DOES_NOT_EXISTS "Slave channel '%s' does not exist"
ii) --ER_SLAVE_WRONG_REPOSITORYL "To have multiple channels,
repository cannot be of type FILE"
* The following are the error messages modified with channel name mentioned
in them:
All of these error message would have the string 'FOR CHANNEL '
added to them: This means the error messages in the sql/errmsg.txt are
modified by adding
a string format specifier %s and this is filled with FOR CHANNEL
for a non default channel.
i) --ER_SLAVE_MUST_STOP "This operation cannot be performed.
Run STOP SLAVE"%s' first"
ii) --ER_SLAVE_NOT_RUNNING "This operation requires a running slave.
Configure slave and do START SLAVE'%s' first"
iii) --ER_MASTER_INFO "Coudn't initialize master info structure'%s'.
More info in the error log"
iv) --ER_SLAVE_RUNNING "Slave is already running'%s'"
v) --ER_SLAVE_WAS_NOT_RUNNING "Slave is already stopped'%s'"
vi) --ER_SLAVE_MI_INIT_REPOSITORY "Slave failed to initialize master info
from the repository%s'"
vii) --ER_SLAVE_RLI_INIT_REPOSITORY "Slave failed to initialize relay log
infor from the repository'%s'"
* Notes
i) CHANGE MASTER execution results in printing of a channel name in the
error log
ii) Messages related to IO/SQL threadsa are written to the mysqld error log.
G) Compatibility
==============
i) The concept of a default channel which is '' will take care of backward
compatibility i.e 5.7 with multi-source feature shall work for the user
scripts based on 5.5 and 5.6 without any hiccups.
ii) To upgrade from 5.6 to 5.7 with multi-source, users *have* to run the
upgrade scripts which would add a channel_name column in the slave
repository tables.
H) Slave Status
=============
a) SHOW SLAVE STATUS command as mentioned in C7
b) replication performance schema tables
i) Every performance schema replication table would have channel_name as
the *first* column
ii) *NOTE* that this is in contrast to SHOW SLAVE STATUS and slave
repository tables, where the Channel_name is the last column.
The main idea of design should involve:
1) creation of new Master_info and Relay_log_info objects for each channel
2) Datastructure to store Master_infos (since mi->rli and rli->mi,
Master_info's are sufficient)
3) Reading and writing these objects to the slave repositories
4) Any structures that are global, should be made made structural
(i.e wrapped in an object) if required.
5) Remove explicit dependencies on active_mi and use the master info per channel
6) create a special datastructure for replication performance schema tables
7) remove unwarranted code (because Multisource replication, gives the user
a chance to delete slave info objects, which was not possible early).
After the above broad idea, the following are the main changes that are done in
the code.
A) Creation of repositories per channel.
=====================================
MSR supports only TABLE type repository. So, we need to add channel name
to the slave info tables based on the conventions specified in HLS.
@file:scrips/mysql_system_table.sql
+++++++++++++++++++++++++++++++++++
a) mysql.slave_master_info
-------------------------
- PRIMARY KEY(Id)) DEFAULT CHARSET=utf8 STATS_PERSISTENT=0 COMMENT 'Relay Log
Information'";
+ Channel_name CHAR(64) CHARACTER SET utf8 COLLATE utf8_general_ci NOT NULL
COMMENT 'The channel on which the slave is connected to a source. Used in
Multisource Replication',
+ PRIMARY KEY(Channel_name)) DEFAULT CHARSET=utf8 STATS_PERSISTENT=0 COMMENT
'Relay Log Information'";
b) mysql.slave_relay_log_info
-----------------------------
- PRIMARY KEY(Host, Port)) DEFAULT CHARSET=utf8 STATS_PERSISTENT=0 COMMENT
'Master Information'";
+ Channel_name CHAR(64) CHARACTER SET utf8 COLLATE utf8_general_ci NOT NULL
COMMENT 'The channel on which the slave is connected to a source. Used in
Multisource Replication',
+ PRIMARY KEY(Channel_name)) DEFAULT CHARSET=utf8 STATS_PERSISTENT=0 COMMENT
'Master Information'";
c) mysql.slave_worker_info
-------------------------
- PRIMARY KEY(Id)) DEFAULT CHARSET=utf8 STATS_PERSISTENT=0 COMMENT 'Worker
Information'";
+ Channel_name CHAR(64) CHARACTER SET utf8 COLLATE utf8_general_ci NOT NULL
COMMENT 'The channel on which the slave is connected to a source. Used in
Multisource Replication',
+ PRIMARY KEY(Channel_name, Id)) DEFAULT CHARSET=utf8 STATS_PERSISTENT=0
COMMENT 'Worker Information'";
B) Storing the repository information in memory for modification and
retrieving ie. creation and reading the slave info objects per channel.
=======================================================================
This is done by creating a msr_map structure that holds the master infos
of all channels.
@Added file: rpl_msr.h and rpl_msr.cc
--------------------------------------
/**
Maps a channel name to it's Master_info.
*/
typedef std::map mi_map;
/**
Class to store all the Master_info objects of a slave
to access them in the replication code base or performance
schema replication tables.
In a Multisourced replication setup, a slave connects
to several masters (also called as sources). This class
stores the Master_infos where each Master_info belongs
to a slave.
The important objects for a slave are the following:
i) Master_info and Relay_log_info (slave_parallel_workers == 0)
ii) Master_info, Relay_log_info and Slave_worker(slave_parallel_workers >0 )
Master_info is always assosiated with a Relay_log_info per channel.
So, it is enough to store Master_infos and call the corresponding
Relay_log_info by mi->rli;
This class is not yet thread safe. Any part of replication code that
calls this class member function should always Lock the mutex LOCK_msr_map.
Only a single global object for a server instance should be created.
The two important data structures in this class are
i) c++ std map to store the Master_info pointers with channel name as a key.
ii) An array of Master_info pointers to access from performance schema
tables. This array is specifically is implemented in a way to make
a) pfs indices simple i.e a simple integer counter
b) To avoid recalibration of data structure if master info is deleted.
* Consider the following high level implementation of a pfs table
to make a row.
highlevel_pfs_funciton()
{
while(replication_table_xxxx.rnd_next())
{
do stuff;
}
}
However, we lock LOCK_msr_map for every rnd_next(); There is a gap
where an addition/deletion of a channel would rearrange the map
making the integer indices of the pfs table point to a wrong value.
Either missing a row or duplicating a row.
We solve this problem, by using an array exclusively to use in
replciation pfs tables, by marking a master_info defeated as 0
(i.e NULL). A new master info is added to this array at the
first NULL always.
*/
class Multisource_info
{
private:
/* Maximum number of channels per slave */
static const unsigned int MAX_CHANNELS= 256;
/* A Map that maps, a channel name to a Master_info */
mi_map channel_to_mi;
/* Array for performance schema related tables */
Master_info *pfs_mi[MAX_CHANNELS];
/* Number of master_infos at the moment*/
uint current_mi_count;
/**
Default_channel for this instance, currently is predefined
and cannot be modified.
*/
static const char* default_channel;
/**
Get the index of the master info correposponding to channel name
from the pfs_mi array.
*/
int get_index_from_pfs_mi(const char* channel_name);
public:
/* Constructor for this class.*/
Multisource_info()
{
current_mi_count= 0;
for (uint i= 0; i < MAX_CHANNELS; i++)
pfs_mi[i]= 0;
}
/**
Adds the Master_info object to both channel_to_mi and multisource_mi
@param[in] channel channel name
@param[mi] mi pointer to master info corresponding
to this channel
@return
@retval FALSE succesfully added to the map
@retval TRUE ok.
*/
bool add_mi(const char* channel_name, Master_info* mi);
/**
Find the master_info object corresponding to a channel explicitly
from channel_to_mi;
Return if it exists, otherwise return 0
@param[in] channel channel name for the master info object.
@retval pointer to the master info object if exists
in the map. Otherwise, NULL;
*/
Master_info* get_mi(const char* channel_name);
/**
Remove the entry corresponding to the channel, from the channel_to_mi
and sets index in the multisource_mi to 0;
And also delete the {mi, rli} pair corresponding to this channel
@param[in] channel_name Name of the channel for a Master_info object
@return
@retval false succesfully deleted.
@retval true not ok
*/
bool delete_mi(const char* channel_name);
/**
Used only by replication performance schema indices to get the master_info
at the position 'pos' from the multisource_mi array.
@param[in] pos the index in the pfs_mi array
@retval pointer to the master info object at pos 'pos';
*/
Master_info* get_mi_at_pos(uint pos);
/**
Return a channel name from the map having the same host and port.
@param[in] host host of the new channel.
@param[in] port port of the new channel.
@return channel_name channel in the map with same host and port.
If no such channel, exists, return 0
*/
const char* get_channel_with_host_port(char* host, uint port);
/**
Get the default channel for this multisourced_slave;
*/
inline const char* get_default_channel()
{
return default_channel;
}
/**
Get the number of instances of Master_info in the map.
*/
inline uint get_num_instances()
{
return channel_to_mi.size();
}
/**
Get max channels allowed for this map.
*/
inline uint get_max_channels()
{
return MAX_CHANNELS;
}
/**
Returns true if the current number of channels in this slave
is less than the MAX_CHANNLES
*/
inline bool is_valid_channel_count()
{
return (current_mi_count < MAX_CHANNELS);
}
/**
Forward iterators to initiate traversing of a map.
@todo: Not to expose iterators. But instead to return
only Master_infos or create generators when
c++11 is introduced.
*/
mi_map::iterator begin()
{
return channel_to_mi.begin();
}
mi_map::iterator end()
{
return channel_to_mi.end();
}
};
C) Command specification by the user using FOR CHANNEL
===================================================
@file: sql_lex.h
----------------
Add the following to lexer and parser.
TO lex->mi:
const char* channel; // name of the channel supplied by the user
bool for_channel;
bool slave_until
@file: sql/lex.h
----------------
Add the terminal/keyword CHANNEL
@file: sql/sql_yacc.yy
----------------------
Add the non terminal opt_channel with the following rule:
opt_channel:
/*empty */
{
Lex->mi.channel= "";
Lex->mi.for_channel= false;
}
| FOR_SYM CHANNEL_SYM TEXT_STRING_sys_nonewline
{
Lex->mi.channel= $3.str;
Lex->mi.for_channel= true;
}
;
D) Entry functions for different functions
================================================
All the code from sql/sql_parse.cc is moved to rpl_slave.cc
to handle differnet commands.
@file: rpl_slave.h
------------------
/**
Entry point for the STOP SLAVE command. This function stops replication
threads for all channels or a single channel based on the command
options supplied.
@param[in] thd the client thread.
@return
@retval false ok
@retval true not ok.
*/
bool start_slave_cmd(THD* thd);
/**
Entry point to the START SLAVE command. The function
decides to start replication threads on several channels
or a single given channel.
@param[in] thd the client thread carrying the command.
@return
@retval false ok
@retval true not ok.
*/
bool stop_slave_cmd(THD* thd);
/**
Entry point for the CHANGE MASTER command. Function
decides to create a new channel or create an existing one.
Errors out with ER_SLAVE_MULTIPLE_CHANNELS_CMD if channel
is not provided to the command.
@param[in] thd the client thread that issued the command.
@return
@retval true fail
@retval false success.
*/
bool change_master_cmd(THD *thd);
/**
Entry function for RESET SLAVE command. Function either resets
the slave for all channels or for a single channel.
When RESET SLAVE ALL is given, the slave_info_objects (mi, rli & workers)
are destroyed.
@param[in] thd the client thread with the command.
@return
@retval false OK
@retval true not OK
*/
bool reset_slave_cmd(THD *thd);
/**
Entry point for SHOW SLAVE STATUS command. Function displayes
the slave status for all channels or for a single channel
based on the FOR CHANNEL clause. This function doesn't error out.
If a channel is not present, then it displays empty result.
(just like previously, when the server not configured as a slave.
@param[in] thd the client thread.
@return
@retval false ok
@retval true not ok
*/
bool show_slave_status_cmd(THD *thd);
/**
Entry point for FLUSH RELAYLOGS command or to flush relaylogs for
the FLUSH LOGS command.
FLUSH LOGS or FLUSH RELAYLOGS needs to flush the relaylogs of all
the replciaiton channels in multisource replication.
FLUSH RELAYLOGS FOR CHANNEL flushes only the relaylogs pertaining to
a channel.
@param[in] thd the client thread carrying the command.
@return
@retval true fail
@retval false success
*/
bool flush_relay_logs_cmd(THD *thd);
E) Creation of slave_info_objects per channel and to read/write repositories
============================================================================
@file: rpl_info_factory.[h,cc]
----------------------------
/**
This function should be called from init_slave() only.
During the server start, read all the slave repositories
on disk (either in FILE or TABLE form) and create corresponding
slave info objects. Each thus created master_info object is
added to pmsr_map.
Multisource replication is supported by only TABLE based
repositories. Based on this fact, the following table shows
the supported cases considering the repository type and
multiple channels of a slave.
Each <---> represents a channel with a name on top of it.
'' is an empty stringed channel (or default channel).
'N' indicates some name for a channel.
+-----------------------------+------------------+-----------+
| channels | Supported? FILE | TABLE |
+-----------------------------+------------------+-----------+
| '' | | |
| A) Master<------->Slave | YES | YES |
| | | |
| | | |
| 'N' | | |
| B) Master<------->Slave | NO | YES |
| | | |
| '' | | |
| Master0<------------+ | | |
| 'N' v | NO | YES |
| C) Master1<----------->Slave| | |
| 'N' ^ | | |
| Mastern<------------+ | | |
| | | |
| | | |
| 'N' | | |
| Master1<------------+ | | |
| 'N' v | NO | YES |
| D) Master2<----------->Slave| | |
| 'N' ^ | | |
| Mastern<------------+ | | |
| | | |
| | | |
+-----------------------------+------------------+-----------+
In a new server, A) shown above is created by default.
If there are multiple 'named' channels, but and if a default_channel
is not created, it is created.
Some points to note from the above table
=========================================
From the table it also follows that conversion of repositories
is possible *ONLY* in the case of A) i.e for ex: if B) type repository
(i.e a named slave channel) was found during server starup but the user
repository option is INFO_REPOSITORY_FILE, then we exit the function.
@note: only for type A) i.e default channel, it is permissable to
have different repo types for Master_info and Relay_log_info
(Ex: FILE for mi and TABLE for rli)
@note: The above restrictions break factory pattern in the code
which has been followed throughout before.
@note: All the repository conversion(or live migration) functions
(ex: decide_repository()) take Rpl_info::internal_id as an
identifier which is always 1 for the case of Master_info and
Relay_log_info. So, in the case of multisource replication,
the decision to convert the repositories shall be made even before
invoking decide_repository(). In other words, if a channel is not a
default channel('') we shall not invoke decide_repository().
@note: In general, the algorithm in creation of slave info object is:
l1: new slave_info;
l2: Initialize the repository handlers
l3: if (default_channel)
check and convert repositories
else
// TABLE type repository
set the value of PK in the TABLE handler.
@note: Update from 5.6 to 5.7(which has Channel_Name in slave_info_tables)
is handled in the upgrade script as usual.
@param[in] mi_option the user provided master_info_repository
@param[in] rli_option the user provided relay_log_info_repository
@param[in] thread_mask thread mask
@param[in] pmsr_map the pointer to the multi source map
(see, rpl_msr.h)
@return
@retval false success
@retval true fail
*/
bool Rpl_info_factory::create_slave_info_objects(uint mi_option,
uint rli_option,
int thread_mask,
Multisource_info *pmsr_map);
/**
Create Master_info and Relay_log_info objects for a new channel.
Also, set cross dependencies between these objects used all over
the code.
Both master_info and relay_log_info repositories should be of the type
TABLE. We do a check for this here as well.
@param[in] mi_option master info repository
@param[in] rli_option relay log info repository
@param[in] channel the channel for which these objects
should be created.
@param[in] pmsr_map a pointer to msr_map
@param[in] to_decide_repo For this channel, check if repositories
are allowed to conver from one type
to other.
@return Pointer pointer to the created Master_info
@return NULL when creation fails
*/
Master_info* Rpl_info_factory::create_slave_per_channel(uint mi_option,
uint rli_option,
const char* channel,
bool to_decide_repo,
Multisource_info* pmsr_map);
/**
Make a list of all the channels if existed on the previos slave run.
@param[in,out] channel_list the names of all channels that exists
on this slave.
@param[in] mi_instances number of master_info repositories
@param[in] mi_repository Found master_info repository
@param[in] default_channel pointer to default channel.
@return
@retval true fail
@retval false success
*/
bool Rpl_info_factory::create_channel_list(std::vector&channel_list,
uint mi_instances, uint mi_repository,
const char* default_channel);
/**
In a multisourced slave, during init_slave(), the repositories
are read to initialize the slave info objects. To initialize
the slave info objects, we need the number of channels the slave
was connected to previously. The following function, finds the
number of channels in the master info repository.
Later, this chanenl list is used to create a pair of {mi, rli}
objects required for IO and SQL threads respectively.
@param [in, out] channel_list A reference to the channel list
This will be filled after reading the
master info table, row by row.
*/
bool
Rpl_info_factory::create_channel_list_from_mi_table(std::vector
&channel_list);
F) Replacing active_mi
=======================
Depending on the placement of the code, active_mi could be replaced with the
following entities
i) msr_map.get_mi(char* channel)
ii) thd->rli_slave->mi
iii) my_pthread_getspecific_ptr(Master_info*, RPL_MASTER_INFO);
G) Replacing LOCK_active_mi
=========================
LOCK_active_mi will be replaced with LOCK_msr_map. Instead of RW which might
improve throughput for some of the command, we go with this single mutex to
keep the design simple for current WL.
A single mutex with a different name (only to avoid confusion later) is used
through out the code to access msr_map
H) REMOVAL of SHOW SLAVE STATUS NONBLOCKING
==========================================
Show slave status nonblocking was implemented without locking
LOCK_active_mi previously. this was because active_mi always
existed. Now this WL supports deletion of channels. So,
SSS NONBLOCKING might crash because another session might
delete the master_info. To prevent a crash, we need to lock
LOCK_msr_map which would make SSS NONBLOCKING eqvivalent
to SSS. So, for this WL SSS NONBLOCKING will be removed.
Copyright (c) 2000, 2024, Oracle Corporation and/or its affiliates. All rights reserved.