WL#1697: Multi-source replication

Affects: Server-5.7   —   Status: Complete

Executive Summary
==================
Multi source replication, will enable a replication slave to have multiple
masters. This feature does not do any additional conflict detection and resolution.
This feature will be useful when the user/DBA wants to commission
a Fan in slave for data aggregation. After implementation of this feature
it will be possible to configure all possible topologies using MySQL replication.

NOTE: This feature differs from Multi-Master Replication, which is an
update Everywhere architecture, where all the nodes act as master and replicate
to all other nodes in the cluster.

FULL DESCRIPTION
=================
A Slave server in MySQL replication: (till 5.6)
 1. Receives transactions (in the form of binary log events) from a single
    server instance (called Master) via a single connection.
 2. Applies these received transactions.

Multisource replication(MSR) feature for a slave server aims to:
 1. Receive transactions (in the form of binary log events) from several
    MySQL servers simultaneously via several connections.
 2. Apply these received transactions whilst not doing any additional conflict
    detection or conflict resolution.

Example: Consider the case of replication from three sources to a single slave
 below(The downward arrow is the data flow through a TCP/IP connection)



       +--------+      +--------+     +--------+
       | mysqld |      | mysqld |     | mysqld |
       +----v---+      +----v---+     +----v---+
            v               v              v
            v               v              v
            v               v              v
            v               v              v
          +-v---------------v--------------v--+
          |              mysqld Slave         |
          +-----------------------------------+

 USES
 =====
 1. For the aggregation of disjoint data from many servers.
   a) Mainly used for backup purposes.
   b) Merging of table shards: Multiple shards of table can be joinly replicated
      to form a single table in a slave. If all the shards are replicated to
      form  a single global table, it is easier for the application to replace
      the complex cross shard joins, since all the data is located in one
      place only. Mainly used for BI and data analytics purposes.
 2. To make it "easier" to deprecate Federated storage engine, as MSR could be
      used to transfer the data from several servers.
 3.  Decrease in latency in circular replication. Consider the case of circular
      replication M1->M2->M3->M4->M1. Latency could be decreased if all these
      four servers form a cluster. For example: Updates from M2 to M1 could be
      reached directly from M2 without intermediate servers M3 & M4.

 Terms and Definitions
 ----------------------
 * Source: Source for a transaction T for a particular server, is classified
   into following types:
 ** A server where transaction T is generated : This is a server where a
    transaction T was originally executed by the user. In other words, this
    is a source where the GTID for a transaction T was first generated.
 ** A server where transaction T is relayed  : In a hierarchical setup 
    consisting of M1->M2->S, the server M2 relays the transactions generated
    on M1 to S.
 ** A server where transaction T1 is relayed and transaction T2 is generated.
    Combination of above both types.
    For ex: in the setup M1->M2->M3->S, on M2, new transactions are generated
    and transactions from M1 are relayed.

 * Multisource replication through hierarchical setup: MySQL 5.6, provides a way
  to aggregate data from several sources through hierarchical setup i.e
  M1->M2->...Mn->S. In this case, the slave would have the
  transactions generated from several sources {M1, M2...Mn} but except
  for M1, all other servers acted as relayed servers.

  * Master: Traditionally, Master has been used to refer to any type of sources
   mentioned above.Current replication (5.6) can support only one master
   per slave and all the replication commands act on that "master". So, 
   redefining master is prone to confusion at this point.In this document,
   master and source mean the same.But we have to make distinction between
   Multi Source Replication and Multi Master replication.

  * Multi Master replication: Implicity, this means Multi Master Update Everywhere
    replication setup. In this setup all the N servers form a cluster
    (i.e each server is connected to other N-1 servers). Updates could be run
    on any of the server and these are replicated to other remaining servers.
    This setup should maintain ordering of the transactions,
    do transaction conflict detection, transaction conflict resolution
    (or rollback)

 * Connection:  A connection between a slave and it's immediate source.
 * Channel: When a slave has a single source (till mysql 5.6), the commands by
    default act on that source only.But when a slave has several  sources
    (through WL#1697), we need to have an abstraction for
    source>IOthread->SQLthread so the commands can act/control each source.
    This abstraction is called a Channel. In short, a  channel name (i.e
    string) acts as an abstraction for a source-receiver-applier path.
    According to this definition, connection is a *part* of the channel.

        +-----------+        +-------------+             +-------------+
        |dump thread|>>>>>>>>|  IO thread  |>>Relay log>>| SQL thread  |
        |           |        |             |             |             |
        +-----------+        +-------------+             +-------------+
                  <-connection->
                  <------------------------channel-------------->

 DESIGN CONSIDERATIONS
 =====================
 The following are the "sub systems" that are considered for design.
 1. Event Receiver System (ER)
 2. Event Queueing System (EQ)
 3. Event Application System (EA)
 4. User interfaces.
    4.a  Startup options
    4.b  commands through mysql client.
 5. Status reporting and Monitoring through P_S replication tables.

  See 'decision_design.txt' attached to this worklog, to find out the design
  chosen for this worklog. The design chosen is described in the points 1, 2 & 3
below

 1.Event Receiver System (ER)
   =========================
   a) Spawn an IO thread for each channel

 2.Event Queueing System
  =====================
  b) Every IO thread writes the events it received to it's own relay log.
     i.e a channel has it's own relay log.

 3.Event Application System
  =========================
   slave_parallel_workers=0
   -------------------------
  a) Spawn an SQL thread for each channel, which applies the events read from the
     relay log belonging to that channel only.

   slave_parallel_workers>1
   ------------------------
  b) Each channel will have it's own coordinator.
  c) Each coordinator will have it's own sets of workers. Transitively,
     this implies that there are dedicated set of workers for each channel.

 Diagrammatically
 ----------------`
 slave_parallel_workers= 0
 -------------------------
              +-----------------------------------+---------------------------+
              |                                   |                           |
+-----+       |     +-----------------------+     |                           |
|     |       |     |  Relay log per        |     |     +------------------+  |
|  M1 |>>>>>>>|     |     source            |>>>>>>>>>>>|  SQL thread      |  |
+-----+       |     +-----------------------+     |     +------------------+  |
              |                                   |                           |
              +---------------------------------------------------------------+
              |                                   |                           |
              |                                   |                           |
+-----+       |       +----------------------+    |       +-----------------+
|     |       |       |   Relay log          |>>>>>>>>>>>>|     SQL thread  | |
|  M2 |>>>>>>>|       |                      |    |       +------------------ |
+-----+       |       |    source            |    |                           |
              |       +----------------------+    |                           |
              |                                   |                           |
              |                                   |                           |
              +---------------------------------------------------------------+
+-----+       |                                   |                           |
|  M  |       |      +----------------------+     |                           |
|     |>>>>>>>|      | Relay log            |     |      +----------------+   |
+-----+       |      |     per source       |>>>>>>>>>>>>|     SQL thread |   |
              |      +----------------------+     |      +----------------+   |
              |                                   |                           |
              |                                   |                           |
              +-----------------------------------+--------------------------+


 slave_parallel_workers > 0
 --------------------------


            +-------------------------+-------------------------------------+
            |                         |                +------+             | 
            |                         |                |   Wi |             | 
            |                         |                +------+             | 
+-----+     |     +-------------+     |    +------+              +------+   | 
|  M1 |     |     |  Relay Log  |     |    |      |              |  Wj  |   |
|     |>>>>>>>>>>>|     Per     |>>>>>>>>>>|  C   |              -------+   | 
+-----+     |     |   Source    |     |    +------+  +--------+             | 
            |     +-------------+     |              |   Wk   |             | 
            |                         |              +--------+             | 
            +---------------------------------------------------------------+
            |                         |  Coordinator   +--------+           |
            |                         |   and set of   |   Wi   |           |
            |                         |   workers per  +--------+           |
+-----+     |      +------------+     |   Source                            |
| M2  |     |      |   Relay Lg |     |     +------+            +-----+     |
|     |>>>>>>>>>>>>|     Per    |>>>>>>>>>>>|  C   |            |  wj |     |
+-----+     |      |   Source   |     |     |      |            +-----+     |
            |      +------------+     |     +------+   +------+             |
            |                         |                | Wk   |             |
            |                         |                -------+             |
            +---------------------------------------------------------------+
            |      +------------+     |                +-------+            |
+-----+     |      | Relay log  |     |                |    Wi |            |
| M3  |     |      |    Per     |     |     +-------+  +-------+            |
|     |>>>>>>>>>>>>|   Source   |>>>>>>>>>>>|   C   |                       |
+-----+     |      +------------+     |     |       |            +-----+    |
            |                         |     +-------+            | Wj  |    |
            |                         |              +-------+   +------+   |
            |                         |              |    Wk |              |
            |                         |              +-------+              |
            |                         |                                     |
            +-------------------------+-------------------------------------+


 4.User interfaces
   ================
   See High level specification for details.


 5. Slave status and Monitoring support
  ======================================
 Capabilities for reporting channel configuration and execution status
 will be done through performance schema tables for replication by adding a
 channel column for each row





FUNCTIONAL REQUIREMENTS
=======================
It is a requirement:
R1: To implement Fan-in replication from N servers.
    See BUG#1697: REQUEST TO IMPLEMENT MULTI-SOURCE REPLICATION
  R1.a: A corollary, of this requirement is a setup of any
        replication topology shall be possible.
R2: That there shall be no deviation from the current replication
    behaviour(till mysql 5.6), when the user does not use multisource
    replication feature
    R2.a: Backward compatibility of existing interfaces will be maintained.
    R2.b: Not to break external utitiles (MySQL utilities) and MySQL cluster
          (ndb_mi.cc)
R3: To show the replication status for each channel in Performance_Schema
    replication tables.
R4: To work when slave_parallel_workers == 0 and > 0 (MTS mode)
R5: To stop replication only for a channel, on which a conflict has occured.
R6: To support GTIDs as well as binary log positions

LIMITATIONS
===========
L1: This feature works only when the replication repositories are of type TABLE.


NON FUNCTIONAL REQUIREMENTS
===========================
NR1: No performance drop for a single channel replication
NR2: Slave throughput shall increase with the increase in number of sources.


NON-REQUIREMENTS
===========================
It is NOT needed :
NE1: To add any extra conflict detection support
     (i.e use whatever is already present)
NE2: To do any conflict resolution if a conflict occurs.
NE3: To add an extra binlog event on a source's binary log or change binary
     log  format.
     It follows from both the above 3 points that,the application developer or
     a DBA should partition the data so that it is disjoint for a successful
     fan in replication.
This section is organized into
A) Startup options
B) Naming conventions.
   a) channel name
   b) file names
C) MySQL client commands
D) System variables
E) Slave status variables
F) Error Messages, warnings and notes
G) Compatibility
   a) backward compatibility
   b) Upgrade from 5.5 and 5.6 slave to 5.7 slave with MSR
H) Replication Filters
I) Semisync replication
J) Slave status
   a) SHOW SLAVE STATUS
   b) Performance schema tables

Default channel
----------------
Before diving into the specification, the definition of
default channel is to be cleared. Multi source replication
introduces creation of new channels to replicate from several sources.
To deal with backward compatibility (where the user does not want to use
this feature and want to replicate from a single source just like till 5.6),
this WL creates the notion of a default channel.
This default channel is an empty string, i.e ''. Here are the properties of a
default channel:
- Unlike other channels,(which can be created and destroyed by the user)
  the default channel always exists: It is created when the server starts
  and it cannot be removed by the user.
- So, if a user wants to use replication commands till 5.6,(i.e user completely
  ignores about multisource replication) all his commands would act on a default
  channel. This clearly supports backward compatability.
- However, if multiple channels are created by the user on this slave, behaviour
  of commmands varies i.e a user can no longer expect all his/her till 5.6 commands
  would act only on a default channel.
  To completely know about behaviour of commands, see command MySQL client command
  section.

In short after the implementation of this feature.
-> A replication command (till 5.6) will act on a default channel if no
   extra channels exist. This presevers backward compatablity.
-> A default channel is implicitly an empty string.

Maximum Number of channels on a replication slave
-------------------------------------------------
Currently, the maximum number of channels is set to 256.
It is hard coded to avoid unnecessary start up options.


A)Startup options
=================
 For this WL, there are no startup options. However, there are startup options
 which are modified. They are described in the next section.


* options that are applicable to *ALL* channels
-------------------------------------------------
Following are the startup options with a brief descrption of how
they would fit in this feature. The ones which does not have
the description below, have the usual meaning applied to all
channels.

# --log-slave-updates
   This option means that *ALL* the transactions that are being received
    by the slave from several sources shall be logged in the binary log.
# --log-slow-slave_statement
# --log-warnings
# --relay-log-purge
   When set, all channels purge their own relay logs automatically
# --relay-log-recovery
# --report-host
# --report-user
# --report-password
# --slave-compressed-protocol
# --relay-log-info-repository
    This has to be TABLE for MSR. Otherwise, addition of second channel
    fails with --ER_SLAVE_NEW_CHANNEL_WRONG_REPOSITORY
# --master-log-info-repository
    This has to be TABLE for MSR. Otherwise, addition of second channel
    fails with --ER_SLAVE_NEW_CHANNEL_WRONG_REPOSITORY
# --sync_relay_log
# --slave_transaction_retries
    This is yet to be implemented for slave_parallel_workers >= 1
# --relay-log-recovery
# --skip-slave-start
    A slave for all channels does not start.
# --slave-load-tmpdir
# --slave-sql-verify-checksum
# --slave_type_conversions
# --slave-skip-errors
    This means execution continues and errors are skipped for *ALL* channels.
# --slave-exec-mode

* options that are applicable per channel
----------------------------------------------
 Since the following options are set up as startup options, all the channels
 will have the same value even though these options act on a single channel.
 In the future, an interface could be built (probably through CHANGE MASTER)
 so that the user can change these values per channel.

 For example: consider this option,
 --relay-log-space-limit: If this option acts on *all* the
 channels, the meaning would have been, "the upper limit
 on the total size of relay logs of *all* channels".

 If this option acts on a single channel, the meaning is
 "the upper limit on the total size of relay log per channel"

# --max-relay-log-size=
   Denotes the maximum size of relay log file per channel.
# --relay-log-space-limit=
   Denotes the upper limit of relay log size per channel
# --slave-parallel-workers=
    The number of slave parallel workers per channel.
# --slave-checkpoint-group=#
# --slave-pending-jobs-max-size=#
# --slave-checkpoint-period=#
# --slave-net-timeout=
    Denotes the waiting time by an IO thread for each source.
# --relay-log-index=filename
    Denotes the base name of relay log index per channel
    See, Naming conventions section below
# --relay-log=filename
    Denotes the base name of relay log per channel
    See, Naming conventions section below
# --slave_net-timeout=N
    This value is set per channel i.e each channel will wait for N seconds
    to check for a broken connection
# --slave-skip-counter= N
    This value is set per channel i.e each channel will skip N events
    from its master.


* options that act if and only if empty stringed defaul channel exists.
  otherwise there is an error generated.
---------------------------------------------------
# --master-info-file=
    This is required for FILE based repository and MSR requires
    TABLE
# --relay-log-info-file=
    This is required for FILE based repository and MSR requires
    TABLE

    Above options mean that, if there are multiple channels (ofcourse
    in TABLE type repo, then the above options do not make convert
    to FILE type repo. Instead there will be an error.)


B) Naming conventions
=====================
* channel name
---------------
 # User shall specify  in the commands of a particular channel.
   It is a string with a maximum allowed of 64 characters.
 # The constraints are same as DB names in lower_case_table_names=1 systems.
   This means that channel names are case insenstive. (For ex: CHANNEL1
    && channel1 are same).
 # since channel names are also part of slave tables, we have to choose
   the char set  for channel names (as they exist as column names in the
   mysql.slave_master_info, mysql.slave_relay_log_info, mysql.slave_worker_info
   tables)
   This means that system_charset or server_charset has virtually no role in
   deciding the channel_name charsets.
 # For the above reason, the hardcoded channel name is chosen to be utf8 with
   COLLATION as utf8_general_ci (to support case insensitivity)
 # default channel is an empty string.


* File names
---------------
The files that are assosiated with the slave replication are
info files, relay-log index file and relay log files. The following describes
the naming of files per channel and how start up options effect the naming of
files.(There is no change in the binlog name behaviour for a multisourced slave)

To distinguish relay-log files and indices of a particular channel, a
distinguisher -
which is a function of channel name - will be a part of these files.
Till 5.6, these files are named as BASE_NAME.index (or HOSTNAME-relay-bin.NNNNN)
and BASE_NAME-relay-bin.NNNNN (or HOSTNAME-relay-bin.NNNNN)
BASE_NAME is specified by relay-log-index or/and relay-log startup options.

For this WL, an extension '-channel_name' is added to the
relay log files, where 'chanenl_name' lower case of the channel name
provided by the user.  For example: If channel name ShannonNoisyChannel then,
 # relay log index file: BASE_NAME-shannonnoisychannel.index
 # relay log files:      BASE_NAME-shannonnoisychannel.00000x
 where BASE_NAME follows the same semantics of mysql 5.6

 In the case of default channel (i.e an empty string), there is NO
 EXTENSION added to the relay log files.

 # In case if channel has special characters like (-,$,? etc) these
   characters are converted to a filesystem charset to get support from
   the Operating system to create and open the relaylog files.

 When File names and channel names are considered together, then the
 net result is:
 # channel_names are case insensitive
 # channel names can have special characters. However, these special characters
   would  get converted to file system charset for relay log file names.
 # default channel ("") does not have any extension to support backward
compatability.


B). MySQL client commands
=======================
General rules for the commands
==============================
1. Add FOR CHANNEL '' to all the replication commands that
   support channel specificity. In this case, this particular command
   acts only on that channel specified.

2. For any command, if a specified channel does not exist to act on the command,
   following error is displayed on the client and *server log*
   --ER_SLAVE_CHANNEL_DOES_NOT_EXISTS
           "Slave channel "" does not exist on the slave"

3. To support backward compaitbility and also for the users who are not
   interested to use this feature (i.e if FOR CHANNEL  ''
   is not given by the user) the command behaviour depends on the type
   of command.

   a) There are commands for which without specifying a channel would act on
      *ALL* channels. For example: START SLAVE/STOP SLAVE/SHOW SLAVE STATUS/
      RESET SLAVE/FLUSH RELAYLOGS
   b) There are commads for which without specifying a channel does not make
      sense if there are multiple channels.
      Example: CHANGE MASTER/ SHOW RELAYLOG EVENTS/ master_pos_wait(), 
      gtid_set_wait() etc. Remember that default channel always exists.
      so, if a user has created a new replciation channel, this category of
      command needs Channel name specified to them always.

      Based on the above reasoning, we categorize them in to following two
      types.

Commands that can act only on single channel if the clause
FOR CHANNEL ''  is not specified.
----------------------------------------------------------------------

C1: CHANGE MASTER TO master_def.... FOR CHANNEL 

  i) Updates the members of the master_info object for the channel_name if
     the channel already exists on the slave.
 ii) If the channel does not exist, a Master_info and Relay_log_info object
     for that channel is created and flushed to the repository.
     Notice that, --ER_SLAVE_CHANNEL_DOES_NOT_EXISTS error is not thrown for
     this command.

 iii) Following both commands shall be possible:
      * CHANGE MASTER TO MASTER_NAME=, MASTER_PORT=
        FOR CHANNEL 
      * CHANGE MASTER TO MASTER_NAME=, MASTER_PORT=
        FOR CHANNEL  
     i.e one could change the host and/or port for the same channel.

  iv) Following both commands shall not be possible:
      * CHANGE MASTER TO MASTER_NAME=, MASTER_PORT=
        FOR CHANNEL  
      * CHANGE MASTER TO MASTER_NAME=, MASTER_PORT=
         FOR CHANNEL  

     The second command will fail with the following error:
       --ER_SLAVE_MULTIPLE_CHANNELS_HOST_PORT
             "Multiple slave channels for the same host and port combination"

  v) If the FOR CHANNEL is not specified, and if there are more then one
     channel on the slave, then the command errors out with
      --ER_MULTIPLE_CHANNELS_CMD "Multiple channels exist. Cannot choose a
                                  channel to act. Use FOR CHANNEL clause to
                                  choose a channel'


C2: SHOW RELAY LOG EVENTS FOR CHANNEL ''
    i) If FOR CHANNEL is not specified and if there channels>=2, we exit with
       --ER_MULTIPLE_CHANNELS_CMD

C3: SELECT MASTER_POS_WAIT(('master_log_file', master_log_pos
                            [, timeout] [,channel_name])
    i) if channel_name is not specified and multiple channels exists, then
        --ER_MULTIPLE_CHANNELS_CMD is issued.

C4: SELECT WAIT_UNTIL_SQL_THREAD_AFTER_GTIDS('gtid_set',[time_out [, channel_name])

    i) if channel_name is not specified and multiple channels exists, then
        --ER_MULTIPLE_CHANNELS_CMD is issued.

  NOTE: We create default channel all the time.

 Commands that act on *ALL* channels if FOR CHANNEL <'channel_name'> is not
 provided
 --------------------------------------------------------------------------

C5: STOP SLAVE [thread_types]
    i) This will stop slave for all channels.
    ii) STOP SLAVE [thread_types]  FOR CHANNEL <'channel_name'> will stop
        slave  only for that channel.

C6: START SLAVE [thread_types]
    i) Starts slave for all channels.
    ii) START SLAVE [thread_types] [until_option] [connection_options] FOR
        CHANNEL <'channel_name'>
        will start slave for <'channel_name'>
    iii) NOTE:START SLAVE [thread_types] [until_option] [connection_options] 
         will act only on a single channel.
         But if there are multiple channels, this commmand errors out with:
         --ER_SLAVE_MULTIPLE_CHANNELS_CMD  "Multiple channels exist.
                                            Cannot choose a channel to act.
                                            Use FOR CHANNEL clause to choose a
                                            channel'

C7: RESET SLAVE [ALL]
    i) Reset the slave for all channels. All the relay log files and index files
        per named channels are also deleted.
    ii) [ALL] clause deletes the master info structures of all channels too
        except default channel. This means, SHOW SLAVE STATUS after a 
         RESET SLAVE ALL should be empty.
    iii) RESET SLAVE [ALL] FOR CHANNEL <'channel_name'> will act only on a
         particular channel.


C8: SHOW SLAVE STATUS
     i) Shows the slave status for all channels
     ii) SHOW SLAVE STATUS FOR CHANNEL <'channel_name'> will act only on that
          channel_name

Removed command
=================
C9: SHOW SLAVE STATUS NONBLOCKING
     i) This command has no meaning for multisource replication as now
        {mi,rli} pair i.e
        slave info objects  can be deleted. So, the way this command has
        implemented could create server crashes. This command will be removed. 
       (See LLD for the reason).
        However, WL#??? will make SHOW SLAVE STATUS non blocking implicitly.



D) System Variables
========================
These are the options for a replication slave server are dynamic.
(change at run time using SET GLOBAL 

# master_info_repository or relay_log_info_repository
  i)  Not allowed to be set as FILE when the number of channels > 1
  ii) Not allowed to be set as FILE when the number of channels == 1
       and channel name != ''
  iii) In the above both cases, an error --ER_SLAVE_WRONG_REPOSITORY is put
        in the thread diag area

#  Rest of the variables shall be applicable to all channels.


E) slave status variables
========================
All these status variables act on default channel. To know, the status of other
channels, the user has to lookinto performance schema tables.

Show status like '%Slave_running%'
Show status like '%Slave_retried_transactions%'
Show status like '%last_received_heartbeat%'
Show status like '%received_heartbeat%'
show status like '%heartbeat_period%'


F) Error messages
===================
 Error messages should indicate the channel on which a particular thread, failed.
* The following are the error variables introduced.
 i)  --ER_SLAVE_CHANNEL_DOES_NOT_EXISTS  "Slave channel '%s' does not exist"
 ii) --ER_SLAVE_WRONG_REPOSITORYL "To have multiple channels,
                                    repository cannot be of type FILE"

* The following are the error messages modified with channel name mentioned
  in them:
  All of these error message would have the string 'FOR CHANNEL '
  added to them: This means the error messages in the sql/errmsg.txt are
   modified by adding
  a string format specifier %s and this is filled with FOR CHANNEL
 for a non default channel.
  i) --ER_SLAVE_MUST_STOP  "This operation cannot be performed.
                                Run STOP SLAVE"%s' first"
  ii) --ER_SLAVE_NOT_RUNNING "This operation requires a running slave.
                     Configure slave and do START SLAVE'%s' first"
  iii) --ER_MASTER_INFO  "Coudn't initialize master info structure'%s'. 
                          More info in the error log"
  iv)  --ER_SLAVE_RUNNING "Slave is already running'%s'"
   v)  --ER_SLAVE_WAS_NOT_RUNNING "Slave is already stopped'%s'"
  vi)  --ER_SLAVE_MI_INIT_REPOSITORY "Slave failed to initialize master info
                                         from the repository%s'"
  vii) --ER_SLAVE_RLI_INIT_REPOSITORY "Slave failed to initialize relay log
                                     infor from the repository'%s'"

* Notes
  i) CHANGE MASTER execution results in printing of a channel name in the
      error log
  ii) Messages related to IO/SQL threadsa are written to the mysqld error log.

 G) Compatibility
  ==============
  i) The concept of a default channel which is '' will take care of backward
      compatibility i.e 5.7 with multi-source feature shall work for the user
      scripts based on 5.5 and 5.6 without any hiccups.
  ii) To upgrade from 5.6 to 5.7 with multi-source, users *have* to run the 
     upgrade scripts which would add a channel_name column in the slave 
     repository tables.

H) Slave Status
   =============
   a) SHOW SLAVE STATUS command as mentioned in C7

   b) replication performance schema tables
    i) Every performance  schema replication table would have channel_name as
        the *first* column
    ii) *NOTE* that this is in contrast to SHOW SLAVE STATUS and slave
        repository tables, where the Channel_name is the last column.
 The main idea of design should involve:
1) creation of new Master_info and Relay_log_info objects for each channel
2) Datastructure to store Master_infos (since mi->rli and rli->mi,
   Master_info's are sufficient)
3) Reading and writing these objects to the slave repositories
4) Any structures that are global, should be made made structural 
   (i.e wrapped in an object) if required.
5) Remove explicit dependencies on active_mi and use the master info per channel
6) create a special datastructure for replication performance schema tables
7) remove unwarranted code (because Multisource replication, gives the user
   a chance to delete slave info objects, which was not possible early).

After the above broad idea, the following are the main changes that are done in
the code.


A) Creation of repositories per channel.
  =====================================
  MSR supports only TABLE type repository. So, we need to add channel name
  to the slave info tables based on the conventions specified in HLS.

  @file:scrips/mysql_system_table.sql
  +++++++++++++++++++++++++++++++++++
   a) mysql.slave_master_info
     -------------------------
-  PRIMARY KEY(Id)) DEFAULT CHARSET=utf8 STATS_PERSISTENT=0 COMMENT 'Relay Log
Information'";
+  Channel_name CHAR(64) CHARACTER SET utf8 COLLATE utf8_general_ci NOT NULL
COMMENT 'The channel on which the slave is connected to a source. Used in
Multisource Replication',
+  PRIMARY KEY(Channel_name)) DEFAULT CHARSET=utf8 STATS_PERSISTENT=0 COMMENT
'Relay Log Information'";

   b) mysql.slave_relay_log_info
     -----------------------------
-  PRIMARY KEY(Host, Port)) DEFAULT CHARSET=utf8 STATS_PERSISTENT=0 COMMENT
'Master Information'";
+  Channel_name CHAR(64) CHARACTER SET utf8 COLLATE utf8_general_ci NOT NULL
COMMENT 'The channel on which the slave is connected to a source. Used in
Multisource Replication',
+  PRIMARY KEY(Channel_name)) DEFAULT CHARSET=utf8 STATS_PERSISTENT=0 COMMENT
'Master Information'";

   c) mysql.slave_worker_info
     -------------------------

-  PRIMARY KEY(Id)) DEFAULT CHARSET=utf8 STATS_PERSISTENT=0 COMMENT 'Worker
Information'";
+  Channel_name CHAR(64) CHARACTER SET utf8 COLLATE utf8_general_ci NOT NULL
COMMENT 'The channel on which the slave is connected to a source. Used in
Multisource Replication',
+  PRIMARY KEY(Channel_name, Id)) DEFAULT CHARSET=utf8 STATS_PERSISTENT=0
COMMENT 'Worker Information'";


B) Storing the repository information in memory for modification and 
   retrieving ie. creation and reading the slave info objects per channel.
   =======================================================================
   This is done by creating a msr_map structure that holds the master infos
    of all channels.

   @Added file: rpl_msr.h and rpl_msr.cc
   --------------------------------------
 /**
  Maps a channel name to it's Master_info.
 */


typedef std::map mi_map;

/**
  Class to store all the Master_info objects of a slave
  to access them in the replication code base or performance
  schema replication tables.

  In a Multisourced replication setup, a slave connects
  to several masters (also called as sources). This class
  stores the Master_infos where each Master_info belongs
  to a slave.

  The important objects for a slave are the following:
  i) Master_info and Relay_log_info (slave_parallel_workers == 0)
  ii) Master_info, Relay_log_info and Slave_worker(slave_parallel_workers >0 )

  Master_info is always assosiated with a Relay_log_info per channel.
  So, it is enough to store Master_infos and call the corresponding
  Relay_log_info by mi->rli;

  This class is not yet thread safe. Any part of replication code that
  calls this class member function should always Lock the mutex LOCK_msr_map.

  Only a single global object for a server instance should be created.

  The two important data structures in this class are
  i) c++ std map to store the Master_info pointers with channel name as a key.
  ii) An array of Master_info pointers to access from performance schema
     tables. This array is specifically is implemented in a way to make
      a) pfs indices simple i.e a simple integer counter
      b) To avoid recalibration of data structure if master info is deleted.
         * Consider the following high level implementation of a pfs table
            to make a row.
          
          highlevel_pfs_funciton()
          {
           while(replication_table_xxxx.rnd_next())
           {
             do stuff;
             }
          }
         
         However, we lock LOCK_msr_map for every rnd_next(); There is a gap
         where an addition/deletion of a channel would rearrange the map
         making the integer indices of the pfs table point to a wrong value.
         Either missing a row or duplicating a row.

         We solve this problem, by using an array exclusively to use in
         replciation pfs tables, by marking a master_info defeated as 0
         (i.e NULL). A new master info is added to this array at the
         first NULL always.

*/
class Multisource_info
{

private:
 /* Maximum number of channels per slave */
  static const unsigned int MAX_CHANNELS= 256;

 /* A Map that maps, a channel name to a Master_info */
  mi_map channel_to_mi;

  /* Array for  performance schema related tables */
  Master_info *pfs_mi[MAX_CHANNELS];

  /* Number of master_infos at the moment*/
  uint current_mi_count;

  /**
    Default_channel for this instance, currently is predefined
    and cannot be modified.
  */
  static const char* default_channel;

  /**
     Get the index of the master info correposponding to channel name
     from the pfs_mi array.
  */
  int get_index_from_pfs_mi(const char* channel_name);

public:

  /* Constructor for this class.*/

  Multisource_info()
  {
    current_mi_count= 0;
    for (uint i= 0; i < MAX_CHANNELS; i++)
      pfs_mi[i]= 0;
  }

  /**
    Adds the Master_info object to both channel_to_mi and multisource_mi

    @param[in]   channel     channel name
    @param[mi]   mi          pointer to master info corresponding
                             to this channel

    @return
      @retval      FALSE       succesfully added to the map
      @retval      TRUE        ok.
  */
  bool add_mi(const char* channel_name, Master_info* mi);

  /**
    Find the master_info object corresponding to a channel explicitly
    from channel_to_mi;
    Return if it exists, otherwise return 0

    @param[in]  channel       channel name for the master info object.

    @retval                   pointer to the master info object if exists
                              in the map. Otherwise, NULL;
  */
  Master_info* get_mi(const char* channel_name);

  /**
    Remove the entry corresponding to the channel, from the channel_to_mi
    and sets index in the  multisource_mi to 0;
    And also delete the {mi, rli} pair corresponding to this channel

    @param[in]    channel_name     Name of the channel for a Master_info object
    @return
      @retval     false            succesfully deleted.
      @retval     true             not ok
  */
  bool delete_mi(const char* channel_name);

  /**
    Used only by replication performance schema indices to get the master_info
    at the position 'pos' from the multisource_mi array.

    @param[in]   pos   the index in the pfs_mi array
    @retval           pointer to the master info object at pos 'pos';
  */
  Master_info* get_mi_at_pos(uint pos);

  /**
     Return a channel name from the map having the same host and port.

    @param[in]         host         host of the new channel.
    @param[in]         port         port of the new channel.

    @return            channel_name  channel in the map with same host and port.
                                     If no such channel, exists, return 0
 */
  const char* get_channel_with_host_port(char* host, uint port);

  /**
    Get the default channel for this multisourced_slave;
  */
  inline const char* get_default_channel()
  {
    return default_channel;
  }

  /**
     Get the number of instances of Master_info in the map.
  */
  inline uint get_num_instances()
  {
    return channel_to_mi.size();
  }

  /**
    Get max channels allowed for this map.
  */
  inline uint get_max_channels()
  {
    return MAX_CHANNELS;
  }

  /**
    Returns true if the current number of channels in this slave
    is less than the MAX_CHANNLES
  */
  inline bool is_valid_channel_count()
  {
    return (current_mi_count < MAX_CHANNELS);
  }

  /**
     Forward iterators to initiate traversing of a map.

     @todo: Not to expose iterators. But instead to return
            only Master_infos or create generators when
            c++11 is introduced.
  */
  mi_map::iterator begin()
  {
    return channel_to_mi.begin();
  }

  mi_map::iterator end()
  {
    return channel_to_mi.end();
  }

};


C) Command specification by the user using FOR CHANNEL
   ===================================================
   @file: sql_lex.h
   ----------------
   Add the following to lexer and parser.
   TO lex->mi:
       const char* channel;  // name of the channel supplied by the user
       bool for_channel;
       bool slave_until

   @file: sql/lex.h
   ----------------
    Add the terminal/keyword CHANNEL

   @file: sql/sql_yacc.yy
   ----------------------
   Add the non terminal opt_channel with the following rule:

   opt_channel:
         /*empty */
       {
         Lex->mi.channel= "";
         Lex->mi.for_channel= false;
       }
     | FOR_SYM CHANNEL_SYM TEXT_STRING_sys_nonewline
       {
         Lex->mi.channel= $3.str;
         Lex->mi.for_channel= true;
       }
    ;

D) Entry functions for different functions
   ================================================
   All the code from sql/sql_parse.cc is moved to rpl_slave.cc
   to handle differnet commands.

   @file: rpl_slave.h
   ------------------
   /**
    Entry point for the STOP SLAVE command. This function stops replication
    threads for all channels or a single channel based on the  command
    options supplied.

    @param[in]     thd         the client thread.

   @return
     @retval       false            ok
     @retval       true             not ok.
  */
   bool start_slave_cmd(THD* thd);

   /**
     Entry point to the START SLAVE command. The function
     decides to start replication threads on several channels
     or a single given channel.

    @param[in]   thd        the client thread carrying the command.

     @return
      @retval      false      ok
      @retval      true       not ok.
  */
   bool stop_slave_cmd(THD* thd);

   /**
    Entry point for the CHANGE MASTER command. Function
    decides to create a new channel or create an existing one.
    Errors out with ER_SLAVE_MULTIPLE_CHANNELS_CMD if channel
    is not provided to the command.

    @param[in]        thd        the client thread that issued the command.

    @return
      @retval         true        fail
      @retval         false       success.
  */
   bool change_master_cmd(THD *thd);

   /**
     Entry function for RESET SLAVE command. Function either resets
     the slave for all channels or for a single channel.
     When RESET SLAVE ALL is given, the slave_info_objects (mi, rli & workers)
     are destroyed.

    @param[in]           thd          the client thread with the command.

    @return
      @retval            false            OK
      @retval            true            not OK
  */
   bool reset_slave_cmd(THD *thd);

   /**
     Entry point for SHOW SLAVE STATUS command. Function displayes
     the slave status for all channels or for a single channel
     based on the FOR CHANNEL  clause. This function doesn't error out.
     If a channel is not present, then it displays empty result.
     (just like previously, when the server not configured as a slave.

    @param[in]       thd          the client thread.

    @return
      @retval        false          ok
      @retval        true          not ok
  */
   bool show_slave_status_cmd(THD *thd);

   /**
     Entry point for FLUSH RELAYLOGS command or to flush relaylogs for
     the FLUSH LOGS command.
     FLUSH LOGS or FLUSH RELAYLOGS needs to flush the relaylogs of all
     the replciaiton channels in multisource replication.
     FLUSH RELAYLOGS FOR CHANNEL flushes only the relaylogs pertaining to
     a channel.

   @param[in]         thd              the client thread carrying the command.

   @return
     @retval           true                fail
     @retval           false              success
  */
   bool flush_relay_logs_cmd(THD *thd);



E) Creation of slave_info_objects per channel and to read/write repositories
============================================================================
 @file:  rpl_info_factory.[h,cc]
----------------------------
/**
  This function should be called from init_slave() only.

  During the server start, read all the slave repositories
  on disk (either in FILE or TABLE form) and create corresponding
  slave info objects. Each thus created master_info object is
  added to pmsr_map.

  Multisource replication is supported by only TABLE based
  repositories. Based on this fact, the following table shows
  the supported cases considering the repository type and
  multiple channels of a slave.
  Each <---> represents a channel with a name on top of it.
  '' is an empty stringed channel (or default channel).
  'N' indicates some name for a channel.

 +-----------------------------+------------------+-----------+
 | channels                    | Supported? FILE  |  TABLE    |
 +-----------------------------+------------------+-----------+
 |              ''             |                  |           |
 | A)  Master<------->Slave    |  YES             |  YES      |
 |                             |                  |           |
 |                             |                  |           |
 |              'N'            |                  |           |
 | B) Master<------->Slave     |  NO              |  YES      |
 |                             |                  |           |
 |              ''             |                  |           |
 |    Master0<------------+    |                  |           |
 |              'N'       v    |  NO              |  YES      |
 | C) Master1<----------->Slave|                  |           |
 |              'N'       ^    |                  |           |
 |    Mastern<------------+    |                  |           |
 |                             |                  |           |
 |                             |                  |           |
 |              'N'            |                  |           |
 |    Master1<------------+    |                  |           |
 |              'N'       v    |   NO             |   YES     |
 | D) Master2<----------->Slave|                  |           |
 |              'N'       ^    |                  |           |
 |    Mastern<------------+    |                  |           |
 |                             |                  |           |
 |                             |                  |           |
 +-----------------------------+------------------+-----------+

 In a new server, A) shown above is created by default.
 If there are multiple 'named' channels, but and if a default_channel
 is not created, it is created.

 Some points to note from the above table
 =========================================

 From the table it also follows that conversion of repositories
 is possible *ONLY* in the case of A) i.e for ex: if B) type repository
 (i.e a named slave channel) was found during server starup but the user
 repository option is INFO_REPOSITORY_FILE, then we exit the function.

 @note: only for type A) i.e default channel, it is permissable to
        have different repo types for Master_info and Relay_log_info
        (Ex: FILE for mi and TABLE for rli)

 @note: The above restrictions break factory pattern in the code
        which has been followed throughout before.

 @note: All the repository conversion(or live migration) functions
       (ex: decide_repository()) take Rpl_info::internal_id as an
       identifier which is always 1 for the case of Master_info and
       Relay_log_info. So, in the case of multisource replication,
       the decision to convert the repositories shall be made even before
       invoking decide_repository(). In other words, if a channel is not a
       default channel('') we shall not invoke decide_repository().

 @note:  In general, the algorithm in creation of slave info object is:
          l1: new slave_info;
          l2: Initialize the repository handlers
          l3: if (default_channel)
                 check and convert repositories
              else
                   // TABLE type repository
                  set the value of PK in the TABLE handler.

 @note: Update from 5.6 to 5.7(which has Channel_Name in slave_info_tables)
        is handled in the upgrade script as usual.


  @param[in]        mi_option        the user provided master_info_repository
  @param[in]       rli_option        the user provided relay_log_info_repository
  @param[in]       thread_mask       thread mask
  @param[in]       pmsr_map          the pointer to the multi source map
                                     (see, rpl_msr.h)

  @return
   @retval         false              success
   @retval         true               fail

*/
bool Rpl_info_factory::create_slave_info_objects(uint mi_option,
                                               uint rli_option,
                                               int thread_mask,
                                               Multisource_info *pmsr_map);


/**
   Create Master_info and Relay_log_info objects for a new channel.
   Also, set cross dependencies between these objects used all over
   the code.

   Both master_info and relay_log_info repositories should be of the type
   TABLE. We do a check for this here as well.

   @param[in]    mi_option      master info repository
   @param[in]    rli_option     relay log info repository
   @param[in]    channel        the channel for which these objects
                                should be created.
   @param[in]    pmsr_map       a pointer to msr_map
   @param[in]   to_decide_repo  For this channel, check if repositories
                                are allowed to conver from one type
                                to other.

   @return      Pointer         pointer to the created Master_info
   @return      NULL            when creation fails

*/

Master_info* Rpl_info_factory::create_slave_per_channel(uint mi_option,
                                                        uint rli_option,
                                                        const char* channel,
                                                        bool to_decide_repo,
                                                        Multisource_info* pmsr_map);


/**
   Make a list of all the channels if existed on the previos slave run.

   @param[in,out]  channel_list   the names of all channels that exists
                                  on this slave.
   @param[in]       mi_instances  number of master_info repositories
   @param[in]       mi_repository  Found master_info repository
   @param[in]       default_channel  pointer to default channel.

   @return
     @retval        true             fail
     @retval        false            success

*/

bool Rpl_info_factory::create_channel_list(std::vector&channel_list,
                         uint mi_instances,  uint mi_repository,
                         const char* default_channel);



/**
  In a multisourced slave, during init_slave(), the repositories
  are read to initialize the slave info objects. To initialize
  the slave info objects, we need the number of channels the slave
  was connected to previously. The following function, finds the
  number of channels in the master info repository.
  Later, this chanenl list is used to create a pair of {mi, rli}
  objects required for IO and SQL threads respectively.

  @param [in, out]      channel_list A reference to the channel list
                                     This will be filled after reading the
                                     master info table, row by row.

*/
bool
Rpl_info_factory::create_channel_list_from_mi_table(std::vector
&channel_list);


F) Replacing active_mi
  =======================
  Depending on the placement of the code, active_mi could be replaced with the
  following entities
 i)  msr_map.get_mi(char* channel)
 ii) thd->rli_slave->mi
 iii) my_pthread_getspecific_ptr(Master_info*, RPL_MASTER_INFO);

G) Replacing LOCK_active_mi
  =========================
 LOCK_active_mi will be replaced with LOCK_msr_map. Instead of RW which might
 improve throughput for some of the  command, we go with this single mutex to
 keep the design simple for current WL.
 A single mutex with a different name (only to avoid confusion later) is used
 through out the code to access msr_map

H) REMOVAL of SHOW SLAVE STATUS NONBLOCKING
  ==========================================
  Show slave status nonblocking was implemented without locking
  LOCK_active_mi previously. this was because active_mi always
  existed. Now this WL supports deletion of channels. So,
  SSS NONBLOCKING might crash because another session might
  delete the master_info. To prevent a crash, we need to lock
  LOCK_msr_map which would make SSS NONBLOCKING eqvivalent
  to SSS. So, for this WL SSS NONBLOCKING will be removed.