There are a number of techniques when using MySQL Replication with Global Transaction Identifiers (GTIDs) for provisioning a new replica which can then be used for scaleout, being promoted to source as necessary for failover. This section describes the following techniques:
Global transaction identifiers were added to MySQL Replication for the purpose of simplifying in general management of the replication data flow and of failover activities in particular. Each identifier uniquely identifies a set of binary log events that together make up a transaction. GTIDs play a key role in applying changes to the database: the server automatically skips any transaction having an identifier which the server recognizes as one that it has processed before. This behavior is critical for automatic replication positioning and correct failover.
The mapping between identifiers and sets of events comprising a given transaction is captured in the binary log. This poses some challenges when provisioning a new server with data from another existing server. To reproduce the identifier set on the new server, it is necessary to copy the identifiers from the old server to the new one, and to preserve the relationship between the identifiers and the actual events. This is neccessary for restoring a replica that is immediately available as a candidate to become a new source on failover or switchover.
Simple replication. The easiest way to reproduce all identifiers and transactions on a new server is to make the new server into the replica of a source that has the entire execution history, and enable global transaction identifiers on both servers. See Section 2.3.4, “Setting Up Replication Using GTIDs”, for more information.
Once replication is started, the new server copies the entire binary log from the source and thus obtains all information about all GTIDs.
This method is simple and effective, but requires the replica to read the binary log from the source; it can sometimes take a comparatively long time for the new replica to catch up with the source, so this method is not suitable for fast failover or restoring from backup. This section explains how to avoid fetching all of the execution history from the source by copying binary log files to the new server.
Copying data and transactions to the replica. Executing the entire transaction history can be time-consuming when the source server has processed a large number of transactions previously, and this can represent a major bottleneck when setting up a new replica. To eliminate this requirement, a snapshot of the data set, the binary logs and the global transaction information the source server contains can be imported to the new replica. The server where the snapshot is taken can be either the source or one of its replicas, but you must ensure that the server has processed all required transactions before copying the data.
There are several variants of this method, the difference being in the manner in which data dumps and transactions from binary logs are transfered to the replica, as outlined here:
- Data Set
Create a dump file using mysqldump on the source server. Set the mysqldump option
--master-data(with the default value of 1) to include a
CHANGE REPLICATION SOURCE TO|
CHANGE MASTER TOstatement with binary logging information. Set the
AUTO(the default) or
ON, to include information about executed transactions in the dump. Then use the mysql client to import the dump file on the target server.
Alternatively, create a data snapshot of the source server using raw data files, then copy these files to the target server, following the instructions in Section 2.2.5, “Choosing a Method for Data Snapshots”. If you use
InnoDBtables, you can use the mysqlbackup command from the MySQL Enterprise Backup component to produce a consistent snapshot. This command records the log name and offset corresponding to the snapshot to be used on the replica. MySQL Enterprise Backup is a commercial product that is included as part of a MySQL Enterprise subscription. See MySQL Enterprise Backup Overview for detailed information.
Alternatively, stop both the source and target servers, copy the contents of the source's data directory to the new replica's data directory, then restart the replica. If you use this method, the replica must be configured for GTID-based replication, in other words with
gtid_mode=ON. For instructions and important information for this method, see Section 2.2.8, “Adding Replicas to a Replication Environment”.
- Transaction History
If the source server has a complete transaction history in its binary logs (that is, the GTID set
@@GLOBAL.gtid_purgedis empty), you can use these methods.
Alternatively, copy the source server's binary log files to the replica. You can make copies from the replica using mysqlbinlog with the
--rawoptions. These can be read into the replica by using mysqlbinlog
--rawoption) to export the binary log files to SQL files, then passing these files to the mysql client for processing. Ensure that all of the binary log files are processed using a single mysql process, rather than multiple connections. For example:
$> mysqlbinlog copied-binlog.000001 copied-binlog.000002 | mysql -u root -p
For more information, see Using mysqlbinlog to Back Up Binary Log Files.
This method has the advantage that a new server is available almost immediately; only those transactions that were committed while the snapshot or dump file was being replayed still need to be obtained from the existing source. This means that the replica's availability is not instantanteous, but only a relatively short amount of time should be required for the replica to catch up with these few remaining transactions.
Copying over binary logs to the target server in advance is usually faster than reading the entire transaction execution history from the source in real time. However, it may not always be feasible to move these files to the target when required, due to size or other considerations. The two remaining methods for provisioning a new replica discussed in this section use other means to transfer information about transactions to the new replica.
Injecting empty transactions.
The source's global
gtid_executed variable contains
the set of all transactions executed on the source. Rather than
copy the binary logs when taking a snapshot to provision a new
server, you can instead note the content of
gtid_executed on the server from which the
snapshot was taken. Before adding the new server to the
replication chain, simply commit an empty transaction on the new
server for each transaction identifier contained in the
gtid_executed, like this:
SET GTID_NEXT='aaa-bbb-ccc-ddd:N'; BEGIN; COMMIT; SET GTID_NEXT='AUTOMATIC';
Once all transaction identifiers have been reinstated in this way
using empty transactions, you must flush and purge the
replica's binary logs, as shown here, where
N is the nonzero suffix of the current
binary log file name:
FLUSH LOGS; PURGE BINARY LOGS TO 'source-bin.00000N';
You should do this to prevent this server from flooding the
replication stream with false transactions in the event that it is
later promoted to the source. (The
LOGS statement forces the creation of a new binary log
PURGE BINARY LOGS purges the
empty transactions, but retains their identifiers.)
This method creates a server that is essentially a snapshot, but in time is able to become a source as its binary log history converges with that of the replication stream (that is, as it catches up with the source or sources). This outcome is similar in effect to that obtained using the remaining provisioning method, which we discuss in the next few paragraphs.
Excluding transactions with gtid_purged.
The source's global
gtid_purged variable contains
the set of all transactions that have been purged from the
source's binary log. As with the method discussed
Injecting empty transactions), you can
record the value of
gtid_executed on the server
from which the snapshot was taken (in place of copying the
binary logs to the new server). Unlike the previous method,
there is no need to commit empty transactions (or to issue
PURGE BINARY LOGS); instead, you
gtid_purged on the
replica directly, based on the value of
gtid_executed on the server
from which the backup or snapshot was taken.
As with the method using empty transactions, this method creates a server that is functionally a snapshot, but in time is able to become a source as its binary log history converges with that of the source and other replicas.
Restoring GTID mode replicas. When restoring a replica in a GTID based replication setup that has encountered an error, injecting an empty transaction may not solve the problem because an event does not have a GTID.
Use mysqlbinlog to find the next transaction,
which is probably the first transaction in the next log file after
the event. Copy everything up to the
that transaction, being sure to include the
@@SESSION.gtid_next. Even if you are not using row-based
replication, you can still run binary log row events in the
command line client.
Stop the replica and run the transaction you copied. The
mysqlbinlog output sets the delimiter to
/*!*/;, so set it back:
mysql> DELIMITER ;
Restart replication from the correct position automatically:
mysql> SET GTID_NEXT=automatic; mysql> RESET SLAVE; mysql> START SLAVE; Or from MySQL 8.0.22: mysql> SET GTID_NEXT=automatic; mysql> RESET REPLICA; mysql> START REPLICA;