When a joining member connects to an online existing member for
state transfer during distributed recovery, the joining member
acts as a client on the connection and the existing member acts
as a server. When state transfer from the donor's binary log is
in progress over this connection (using the asynchronous
group_replication_recovery), the joining
member acts as the replica and the existing member acts as the
source. When a remote cloning operation is in progress over this
connection, the joining member acts as a recipient and the
existing member acts as a donor. Configuration settings that
apply to those roles outside the Group Replication context can
apply for Group Replication also, unless they are overridden by
a Group Replication-specific configuration setting or behavior.
The connection that an existing member offers to a joining member for distributed recovery is not the same connection that is used by Group Replication for communication between online members of the group.
The connection used by the group communication engine for Group Replication (XCom, a Paxos variant) for TCP communication between remote XCom instances is specified by the
group_replication_local_addresssystem variable. This connection is used for TCP/IP messages between online members. Communication with the local instance is over an input channel using shared memory.
For distributed recovery, up to MySQL 8.0.20, group members offer their standard SQL client connection to joining members, as specified by MySQL Server's
portsystem variables. If an alternative port number is specified by the
report_portsystem variable, that one is used instead.
From MySQL 8.0.21, group members may advertise an alternative list of distributed recovery endpoints as dedicated client connections for joining members, allowing you to control distributed recovery traffic separately from connections by regular client users of the member. You specify this list using the
group_replication_advertise_recovery_endpointssystem variable, and the member transmits their list of distributed recovery endpoints to the group when they join the group. The default is that the member continues to offer the standard SQL client connection as in earlier releases.
Distributed recovery can fail if a joining member cannot
correctly identify the other members using the host name as
defined by MySQL Server's
hostname system variable. It
is recommended that operating systems running MySQL have a
properly configured unique host name, either using DNS or
local settings. The host name that the server is using for SQL
client connections can be verified in the
Member_host column of the Performance
multiple group members externalize a default host name set by
the operating system, there is a chance of the joining member
not resolving it to the correct member address and not being
able to connect for distributed recovery. In this situation
you can use MySQL Server's
report_host system variable
to configure a unique host name to be externalized by each of
The steps for a joining member to establish a connection for distributed recovery are as follows:
When the member joins the group, it connects with one of the seed members included in the list in its
group_replication_group_seedssystem variable, initially using the
group_replication_local_addressconnection as specified in that list. The seed members might be a subset of the group.
Over this connection, the seed member uses Group Replication's membership service to provide the joining member with a list of all the members that are online in the group, in the form of a view. The membership information includes the details of the distributed recovery endpoints or standard SQL client connection offered by each member for distributed recovery.
The joining member selects a suitable group member from this list to be its donor for distributed recovery, following the behaviors described in Section 126.96.36.199, “Fault Tolerance for Distributed Recovery”.
The joining member then attempts to connect to the donor using the donor's advertised distributed recovery endpoints, trying each in turn in the order they are specified in the list. If the donor provides no endpoints, the joining member attempts to connect using the donor's standard SQL client connection. The SSL requirements for the connection are as specified by the
group_replication_recovery_ssl_*options described in Section 188.8.131.52.4, “SSL and Authentication for Distributed Recovery”.
If the joining member is not able to connect to the selected donor, it retries with other suitable donors, following the behaviors described in Section 184.108.40.206, “Fault Tolerance for Distributed Recovery”. Note that if the joining member exhausts the list of advertised endpoints without making a connection, it does not fall back to the donor's standard SQL client connection, but switches to another donor.
When the joining member establishes a distributed recovery connection with a donor, it uses that connection for state transfer as described in Section 18.4.3, “Distributed Recovery”. The host and port for the connection that is used are shown in the joining member's log. Note that if a remote cloning operation is used, when the joining member has restarted at the end of the operation, it establishes a connection with a new donor for state transfer from the binary log. This might be a connection to a different member from the original donor used for the remote cloning operation, or it might be a different connection to the original donor. In any case, the distributed recovery process continues in the same way as it would have with the original donor.
IP addresses supplied by the
system variable as distributed recovery endpoints do not have
to be configured for MySQL Server (that is, they do not have
to be specified by the
admin_address system variable
or in the list for the
variable). They do have to be assigned to the server. Any host
names used must resolve to a local IP address. IPv4 and IPv6
addresses can be used.
The ports supplied for the distributed recovery endpoints do
have to be configured for MySQL Server, so they must be
specified by the
admin_port system variable.
The server must listen for TCP/IP connections on these ports.
If you specify the
admin_port, the replication
user for distributed recovery needs the
privilege to connect. Selecting the
admin_port keeps distributed
recovery connections separate from regular MySQL client
Joining members try each of the endpoints in turn in the order
they are specified on the list. If
is set to
DEFAULT rather than a list of
endpoints, the standard SQL client connection is offered. Note
that the standard SQL client connection is not automatically
included on a list of distributed recovery endpoints, and is
not offered as a fallback if the donor's list of endpoints is
exhausted without a connection. If you want to offer the
standard SQL client connection as one of a number of
distributed recovery endpoints, you must include it explicitly
in the list specified by
You can put it in the last place so that it acts as a last
resort for connection.
A group member's distributed recovery endpoints (or standard
SQL client connection if endpoints are not provided) do not
need to be added to the Group Replication allowlist specified
system variable. The allowlist is only for the address
for each member. A joining member must have its initial
connection to the group permitted by the allowlist in order to
retrieve the address or addresses for distributed recovery.
The distributed recovery endpoints that you list are validated
when the system variable is set and when a
statement has been issued. If the list cannot be parsed
correctly, or if any of the endpoints cannot be accessed on
the host because the server is not listening on them, Group
Replication logs an error and does not start.
From MySQL 8.0.18, you can optionally configure compression
for distributed recovery by the method of state transfer from
a donor's binary log. Compression can benefit distributed
recovery where network bandwidth is limited and the donor has
to transfer many transactions to the joining member. The
system variables configure permitted compression algorithms,
zstd compression level, used when
carrying out state transfer from a donor's binary log. For
more information, see
Section 4.2.8, “Connection Compression Control”.
Note that these compression settings do not apply for remote
cloning operations. When a remote cloning operation is used
for distributed recovery, the clone plugin's
Distributed recovery requires a replication user that has the correct permissions so that Group Replication can establish direct member-to-member replication channels. The replication user must also have the correct permissions to act as the clone user on the donor for a remote cloning operation. The same replication user must be used for distributed recovery on every group member. For instructions to set up this replication user, see Section 220.127.116.11, “User Credentials For Distributed Recovery”. For instructions to secure the replication user credentials, see Section 18.104.22.168, “Secure User Credentials for Distributed Recovery”.
SSL for distributed recovery is configured separately from SSL
for normal group communications, which is determined by the
server's SSL settings and the
system variable. For distributed recovery connections,
dedicated Group Replication distributed recovery SSL system
variables are available to configure the use of certificates
and ciphers specifically for distributed recovery.
By default, SSL is not used for distributed recovery
connections. To activate it, set
and configure the Group Replication distributed recovery SSL
system variables as described in
Section 18.5.3, “Securing Distributed Recovery Connections”.
You need a replication user that is set up to use SSL.
When distributed recovery is configured to use SSL, Group
Replication applies this setting for remote cloning
operations, as well as for state transfer from a donor's
binary log. Group Replication automatically configures the
settings for the clone SSL options
clone_ssl_key) to match your
settings for the corresponding Group Replication distributed
If you are not using SSL for distributed recovery (so
is set to
OFF), and the replication user
account for Group Replication authenticates with the
caching_sha2_password plugin (which is the
default in MySQL 8.0) or the
sha256_password plugin, RSA key-pairs are
used for password exchange. In this case, either use the
system variable to specify the RSA public key file, or use the
system variable to request the public key from the source, as
Section 22.214.171.124.1, “Replication User With The Caching SHA-2 Authentication Plugin”.