WL#13769: Classify important GR log messages as system messages

Affects: Server-8.0 — Status: Complete

Description
Requirements
High Level Architecture
Low Level Design

EXECUTIVE SUMMARY

This worklog reclassifies some Group Replication log messages as system messages. System messages are always logged, independently of the server log level.

After this work is done, operator can always do a post observation of the main events on group lifetime.

RATIONALE

Currently the default MySQL log level[1] is 2, that is, only log messages from levels SYSTEM, ERROR and WARNING are logged, while INFORMATION level log messages are skipped.

Some Group Replication status messages are logged with INFORMATION ([Note]) level, which means that these messages are not logged by default.

One of these examples is:

[Note] [MY-011503] [Repl] Plugin group_replication reported: \
'Group membership changed to 10.0.0.1:33061, 10.0.0.2:33061, \
10.0.0.3:33061 on view 15760832593961450:3.'

This worklog will reclassify all important INFORMATION log messages to SYSTEM[2] log level, making them always present on the server error log.

[1] https://dev.mysql.com/doc/refman/8.0/en/server-system-variables.html#sysvar_log_error_verbosity

[2] https://dev.mysql.com/doc/dev/mysql-server/latest/my__loglevel_8h.html

USER STORIES

As a MySQL operator I want to always be capable to see a group main events on the server error log.

Functional requirements

FR1: Important group events must always be logged on the server error log with SYSTEM log level[1].

[1] https://dev.mysql.com/doc/dev/mysql-server/latest/my__loglevel_8h.html

Non-functional requirements

None.

SUMMARY OF THE APPROACH

Currently the default MySQL log level[1] is 2, that is, only log messages from levels SYSTEM, ERROR and WARNING are logged, while INFORMATION level log messages are skipped.

Some Group Replication status messages are logged with INFORMATION ([Note]) level, which means that these messages are not logged by default.

One of these examples is:

[Note] [MY-011503] [Repl] Plugin group_replication reported: \
'Group membership changed to 10.0.0.1:33061, 10.0.0.2:33061, \
10.0.0.3:33061 on view 15760832593961450:3.'

This worklog will reclassify all important INFORMATION log messages to SYSTEM[2] log level, making them always present on the server error log.

[1] https://dev.mysql.com/doc/refman/8.0/en/server-system-variables.html#sysvar_log_error_verbosity

[2] https://dev.mysql.com/doc/dev/mysql-server/latest/my__loglevel_8h.html

USER INTERFACE

There are no repercussions.

SECURITY CONTEXT

There are no repercussions.

UPGRADE/DOWNGRADE AND CROSS-VERSION REPLICATION

There are no repercussions.

OBSERVABILITY

The following log messages will move from INFORMATION to SYSTEM log level.

Group Replication start

Code: MY-013587

Name: ER_GRP_RPL_IS_STARTING

Input message: Plugin 'group_replication' is starting.

Materialized message: 2020-01-15T11:29:27.464556Z 11 [System] [MY-013587] [Repl] Plugin group_replication reported: 'Plugin 'group_replication' is starting.'

Rationale: Log that Group Replication is starting.

Group Replication is stopped

Code: MY-011651

Name: ER_GRP_RPL_IS_STOPPED

Input message: Plugin 'group_replication' has been stopped.

Materialized message: 2020-01-15T11:29:42.179200Z 11 [System] [MY-011651] [Repl] Plugin group_replication reported: 'Plugin 'group_replication' has been stopped.'

Rationale: Log that Group Replication stopped.

Member is ONLINE

Code: MY-011490

Name: ER_GRP_RPL_SRV_ONLINE

Input message: This server was declared online within the replication group.

Materialized message: 2020-01-15T11:29:31.296731Z 0 [System] [MY-011490] [Repl] Plugin group_replication reported: 'This server was declared online within thebreplication group.'

Rationale: Log that this member is ONLINE in the group. It is only logged on the member that became ONLINE.

Code: MY-011492

Name: ER_GRP_RPL_MEM_ONLINE

Input message: The member with address %s:%u was declared online within the replication group.

Materialized message: 2020-01-15T11:29:31.296927Z 0 [System] [MY-011492] [Repl] Plugin group_replication reported: 'The member with address 127.0.0.1:13000 was declared online within the replication group.'

Rationale: Log that a given member, not the local one, became ONLINE. It is logged on all members, except on the one that became ONLINE. On the one that became ONLINE, we log MY-011490.

Member read mode

Code: MY-011565

Name: ER_GRP_RPL_SUPER_READ_ON

Input message: Setting super_read_only=ON.

Materialized message: 2020-01-15T11:56:27.318727Z 42 [System] [MY-011565] [Repl] Plugin group_replication reported: 'Setting super_read_only=ON.'

Rationale: Log that the local member enabled super_read_only.

Code: MY-011566

Name: ER_GRP_RPL_SUPER_READ_OFF

Input message: Setting super_read_only=OFF.

Materialized message: 2020-01-15T11:56:17.428430Z 39 [System] [MY-011566] [Repl] Plugin group_replication reported: 'Setting super_read_only=OFF.'

Rationale: Log that the local member disabled super_read_only.

Membership change

Code: MY-011503

Name: ER_GRP_RPL_MEMBER_CHANGE

Input message: Group membership changed to %s on view %s.

Materialized message: 2020-01-15T11:29:30.998526Z 0 [System] [MY-011503] [Repl] Plugin group_replication reported: 'Group membership changed to 127.0.0.1:13001, 127.0.0.1:13000 on view 15790877649964184:2.'

Rationale: Log all membership changes in the group.

Code: MY-011722

Name: ER_GRP_RPL_FORCE_MEMBER_VALUE_SET

Input message: The group_replication_force_members value '%s' was set in the group communication interfaces

Materialized message: 2020-02-12T16:06:57.259465Z 34 [System] [MY-011722] [Repl] Plugin group_replication reported: 'The group_replication_force_members value 'localhost:13010,localhost:13011' was set in the group communication interfaces'

Rationale: Log a forced membership change in the group.

Member left the group

Code: MY-011504

Name: ER_GRP_RPL_MEMBER_LEFT_GRP

Input message: Group membership changed: This member has left the group.

Materialized message: 2020-01-15T11:29:42.169464Z 0 [System] [MY-011504] [Repl] Plugin group_replication reported: 'Group membership changed: This member has left the group.'

Rationale: Log that this member left the group. It is only logged on the member that left.

Primary election

Code: MY-011500

Name: ER_GRP_RPL_PRIMARY_MEMBER_LEFT_GRP

Input message: Primary server with address %s left the group. Electing new Primary.

Materialized message: 2020-01-15T11:55:54.635836Z 0 [System] [MY-011500] [Repl] Plugin group_replication reported: 'Primary server with address 127.0.0.1:13000 left the group. Electing new Primary.'

Rationale: Log that the primary left the group, a new election was triggered.

Code: MY-011507

Name: ER_GRP_RPL_NEW_PRIMARY_ELECTED

Input message: A new primary with address %s:%u was elected. %s

Materialized message: 2020-01-15T11:55:55.638364Z 0 [System] [MY-011507] [Repl] Plugin group_replication reported: 'A new primary with address 127.0.0.1:13001 was elected. The new primary will execute all previous group transactions before allowing writes.'

Rationale: Log which member was elected to be the primary, it is also logged on the first group member.

Code: MY-013214

Name: ER_GRP_RPL_CONFIGURATION_ACTION_START

Input message: Starting group operation local execution: %s

Materialized message: 2020-01-15T14:51:35.928075Z 36 [Note] [MY-013214] [Repl] Plugin group_replication reported: 'Starting group operation local execution: Primary election change'

Rationale: Log that a group wide configuration is starting, in this case a manual primary change.

Code: MY-013213

Name: ER_GRP_RPL_CONFIGURATION_ACTION_LOCAL_TERMINATION

Input message: Configuration operation '%s' terminated. %s

Materialized message: 2020-01-15T12:23:15.537902Z 0 [System] [MY-013213] [Repl] Plugin group_replication reported: 'Configuration operation 'Primary election change' terminated. Primary server switched to: c6b58ec0-3791-11ea-931b-0010e0734796'

Rationale: Log that a group wide configuration is terminated, in this case a manual primary change.

Member role

Code: MY-011510

Name: ER_GRP_RPL_SRV_PRIMARY_MEM

Input message: This server is working as primary member

Materialized message: 2020-01-15T11:56:17.431854Z 38 [System] [MY-011510] [Repl] Plugin group_replication reported: 'This server is working as primary member.'

Rationale: Log that this member is the primary.

Code: MY-011511

Name: ER_GRP_RPL_SRV_SECONDARY_MEM

Input message: This server is working as secondary member with primary member address %s:%u.

Materialized message: 2020-01-15T11:56:17.431854Z 38 [System] [MY-011510] [Repl] Plugin group_replication reported: 'This server is working as secondary member with primary member address 127.0.0.1:13001.'

Rationale: Log that this member is a secondary.

Recovery

Code: MY-013471

Name: ER_GRP_RPL_RECOVERY_STRAT_CHOICE

Input message: Distributed recovery will transfer data using: %s

Materialized message: 2020-01-15T12:23:13.856388Z 0 [System] [MY-013471] [Repl] Plugin group_replication reported: 'Distributed recovery will transfer data using: Incremental recovery from a group donor' 2020-01-15T12:23:13.856388Z 0 [System] [MY-013471] [Repl] Plugin group_replication reported: 'Distributed recovery will transfer data using: Cloning from a group donor'

Rationale: Log which recovery strategy will be used.

OBSERVABILITY

The following log messages will move from INFORMATION to ERROR log level.

GCS

Code: MY-011735

Name: ER_GRP_RPL_GCS_GR_ERROR_MSG

Input message: "Unable to bind to %s:%d (socket=%d, errno=%d)!", "INADDR_ANY", port, fd.val, err)

Materialized message: 2020-01-27T14:21:11.884550Z 0 [ERROR] [MY-011735] [Repl] Plugin group_replication reported: '[GCS] Unable to bind to INADDR_ANY:13011 (socket=42, errno=98)!'

Rationale: Log socket bind errors on group communication layer.

DEPLOYMENT AND INSTALLATION

There are no repercussions.

PROTOCOL

There are no repercussions.

FAILURE MODEL SPECIFICATION

There are no repercussions.

EXISTENT WARNING LEVEL MESSAGES

The following messages log level will not change, they have WARNING level. They are being listed here for context.

Code: MY-011493

Name: ER_GRP_RPL_MEM_UNREACHABLE

Input message: Member with address %s:%u has become unreachable.

Materialized message: 2020-01-15T12:57:06.182981Z 0 [Warning] [MY-011493] [Repl] Plugin group_replication reported: 'Member with address 127.0.0.1:13000 has become unreachable.'

Code: MY-011494

Name: ER_GRP_RPL_MEM_REACHABLE

Input message: Member with address %s:%u is reachable again.

Materialized message: 2020-01-15T12:57:06.182981Z 0 [Warning] [MY-011493] [Repl] Plugin group_replication reported: 'Member with address 127.0.0.1:13000 has become reachable again.'

Code: MY-013472

Name: ER_GRP_RPL_RECOVERY_STRAT_FALLBACK

Input message: Due to some issue on the previous step distributed recovery is now executing: %s

Materialized message: 2020-01-15T12:56:56.192597Z 0 [Warning] [MY-013472] [Repl] Plugin group_replication reported: 'Due to some issue on the previous step distributed recovery is now executing: Incremental Recovery.'

Code: MY-013469

Name: ER_GRP_RPL_RECOVERY_STRAT_CLONE_THRESHOLD

Input message: This member will start distributed recovery using clone. It is due to the number of missing transactions being higher than the configured threshold of %llu.

Materialized message: 2020-01-15T13:05:34.532571Z 0 [Warning] [MY-013469] [Repl] Plugin group_replication reported: 'This member will start distributed recovery using clone. It is due to the number of missing transactions being higher than the configured threshold of 1.'

Code: MY-013470

Name: ER_GRP_RPL_RECOVERY_STRAT_CLONE_PURGED

Input message: This member will start distributed recovery using clone. It is due to no ONLINE member has the missing data for recovering in its binary logs.

Materialized message: 2020-01-15T13:05:34.532571Z 0 [Warning] [MY-013469] [Repl] Plugin group_replication reported: 'This member will start distributed recovery using clone. It is due to no ONLINE member has the missing data for recovering in its binary logs.'

EXAMPLES

This is the log output with MySQL log level set to 2 after the changes proposed by this worklog are in place.

Group bootstrap in single-primary mode

2020-01-15T15:42:46.745705Z 17 [System] [MY-011565] [Repl] Plugin group_replication reported: 'Setting super_read_only=ON.'
2020-01-15T15:42:46.761861Z 13 [System] [MY-013587] [Repl] Plugin group_replication reported: 'Plugin 'group_replication' is starting.'
2020-01-15T15:42:46.783753Z 18 [System] [MY-010597] [Repl] 'CHANGE MASTER TO FOR CHANNEL 'group_replication_applier' executed'. Previous state master_host='', master_port= 3306, master_log_file='', master_log_pos= 4, master_bind=''. New state master_host='<NULL>', master_port= 0, master_log_file='', master_log_pos= 4, master_bind=''.
2020-01-15T15:42:50.029480Z 0 [System] [MY-013471] [Repl] Plugin group_replication reported: 'Distributed recovery will transfer data using: Incremental recovery from a group donor'
2020-01-15T15:42:51.033075Z 0 [System] [MY-011503] [Repl] Plugin group_replication reported: 'Group membership changed to 127.0.0.1:13000 on view 15791029690268059:1.'
2020-01-15T15:42:51.033366Z 0 [System] [MY-011490] [Repl] Plugin group_replication reported: 'This server was declared online within the replication group.'
2020-01-15T15:42:52.035644Z 0 [System] [MY-011507] [Repl] Plugin group_replication reported: 'A new primary with address 127.0.0.1:13000 was elected. The new primary will execute all previous group transactions before allowing writes.'
2020-01-15T15:42:52.038129Z 32 [System] [MY-011566] [Repl] Plugin group_replication reported: 'Setting super_read_only=OFF.'
2020-01-15T15:42:52.039011Z 32 [System] [MY-011510] [Repl] Plugin group_replication reported: 'This server is working as primary member.'

Member join in single-primary mode

2020-01-15T15:42:52.230083Z 16 [System] [MY-011565] [Repl] Plugin group_replication reported: 'Setting super_read_only=ON.'
2020-01-15T15:42:52.246425Z 12 [System] [MY-013587] [Repl] Plugin group_replication reported: 'Plugin 'group_replication' is starting.'
2020-01-15T15:42:52.268200Z 17 [System] [MY-010597] [Repl] 'CHANGE MASTER TO FOR CHANNEL 'group_replication_applier' executed'. Previous state master_host='', master_port= 3306, master_log_file='', master_log_pos= 4, master_bind=''. New state master_host='<NULL>', master_port= 0, master_log_file='', master_log_pos= 4, master_bind=''.
2020-01-15T15:42:52.501095Z 0 [ERROR] [MY-011735] [Repl] Plugin group_replication reported: '[GCS] Error on opening a connection to localhost:13012 on local port: 13011.'
2020-01-15T15:42:55.027048Z 12 [System] [MY-011511] [Repl] Plugin group_replication reported: 'This server is working as secondary member with primary member address 127.0.0.1:13000.'
2020-01-15T15:42:56.029095Z 0 [System] [MY-013471] [Repl] Plugin group_replication reported: 'Distributed recovery will transfer data using: Incremental recovery from a group donor'
2020-01-15T15:42:56.030223Z 0 [System] [MY-011503] [Repl] Plugin group_replication reported: 'Group membership changed to 127.0.0.1:13001, 127.0.0.1:13000 on view 15791029690268059:2.'
2020-01-15T15:42:56.062424Z 28 [System] [MY-010597] [Repl] 'CHANGE MASTER TO FOR CHANNEL 'group_replication_recovery' executed'. Previous state master_host='', master_port= 3306, master_log_file='', master_log_pos= 4, master_bind=''. New state master_host='127.0.0.1', master_port= 13000, master_log_file='', master_log_pos= 4, master_bind=''.
2020-01-15T15:42:56.077444Z 29 [Warning] [MY-010897] [Repl] Storing MySQL user name or password information in the master info repository is not secure and is therefore not recommended. Please consider using the USER and PASSWORD connection options for START SLAVE; see the 'START SLAVE Syntax' in the MySQL Manual for more information.
2020-01-15T15:42:56.078827Z 29 [System] [MY-010562] [Repl] Slave I/O thread for channel 'group_replication_recovery': connected to master 'root@127.0.0.1:13000',replication started in log 'FIRST' at position 4
2020-01-15T15:42:56.380348Z 28 [System] [MY-010597] [Repl] 'CHANGE MASTER TO FOR CHANNEL 'group_replication_recovery' executed'. Previous state master_host='127.0.0.1', master_port= 13000, master_log_file='', master_log_pos= 4, master_bind=''. New state master_host='<NULL>', master_port= 0, master_log_file='', master_log_pos= 4, master_bind=''.

2020-01-15T15:42:56.393148Z 0 [System] [MY-011490] [Repl] Plugin group_replication reported: 'This server was declared online within the replication group.'

Primary failover

2019-12-19T12:31:30.078766Z 0 [Warning] [MY-011493] [Repl] Plugin group_replication reported: 'Member with address 127.0.0.1:13000 has become unreachable.'
2019-12-19T12:31:32.082929Z 0 [Warning] [MY-011499] [Repl] Plugin group_replication reported: 'Members removed from the group: 127.0.0.1:13000'
2019-12-19T12:31:32.082960Z 0 [System] [MY-011500] [Repl] Plugin group_replication reported: 'Primary server with address 127.0.0.1:13000 left the group. Electing new Primary.'
2019-12-19T12:31:33.085475Z 0 [System] [MY-011507] [Repl] Plugin group_replication reported: 'A new primary with address 127.0.0.1:13001 was elected. The new primary will execute all previous group transactions before allowing writes.'
2019-12-19T12:31:54.673986Z 12 [ERROR] [MY-013317] [Repl] Plugin group_replication reported: 'Error waiting for group executed transactions commit on session '12'.'
2019-12-19T12:31:54.674725Z 0 [System] [MY-011503] [Repl] Plugin group_replication reported: 'Group membership changed to 127.0.0.1:13001, 127.0.0.1:13002 on view 15767586730861839:4.'
2019-12-19T12:31:54.874627Z 39 [System] [MY-011566] [Repl] Plugin group_replication reported: 'Setting super_read_only=OFF.'
2019-12-19T12:31:54.876002Z 39 [System] [MY-011510] [Repl] Plugin group_replication reported: 'This server is working as primary member.'

Manual primary change

2020-01-15T15:15:00.470451Z 36 [System] [MY-013214] [Repl] Plugin group_replication reported: 'Starting group operation local execution: Primary election change'
2020-01-15T15:15:01.474298Z 0 [System] [MY-011507] [Repl] Plugin group_replication reported: 'A new primary with address 127.0.0.1:13011 was elected. The new primary will execute all previous group transactions before allowing writes. Enabling conflict detection until the new primary applies all relay logs.'
2020-01-15T15:15:01.648157Z 38 [System] [MY-011566] [Repl] Plugin group_replication reported: 'Setting super_read_only=OFF.'
2020-01-15T15:15:01.650837Z 38 [System] [MY-011510] [Repl] Plugin group_replication reported: 'This server is working as primary member.'
2020-01-15T15:15:01.652104Z 0 [System] [MY-013213] [Repl] Plugin group_replication reported: 'Configuration operation 'Primary election change' terminated. Primary server switched to: c5a5c801-37a9-11ea-9e54-0010e0734796'

Member join with clone (which includes a server restart)

2020-01-15T16:01:34.783803Z 24 [System] [MY-011565] [Repl] Plugin group_replication reported: 'Setting super_read_only=ON.'
2020-01-15T16:01:34.798898Z 7 [System] [MY-013587] [Repl] Plugin group_replication reported: 'Plugin 'group_replication' is starting.'
2020-01-15T16:01:34.837001Z 25 [System] [MY-010597] [Repl] 'CHANGE MASTER TO FOR CHANNEL 'group_replication_applier' executed'. Previous state master_host='<NULL>', master_port= 0, master_log_file='', master_log_pos= 4, master_bind=''. New state master_host='<NULL>', master_port= 0, master_log_file='', master_log_pos= 4, master_bind=''.
2020-01-15T16:01:39.011267Z 0 [Warning] [MY-013469] [Repl] Plugin group_replication reported: 'This member will start distributed recovery using clone. It is due to the number of missing transactions being higher than the configured threshold of 1.'
2020-01-15T16:01:40.021762Z 0 [System] [MY-013471] [Repl] Plugin group_replication reported: 'Distributed recovery will transfer data using: Cloning from a remote group donor.'
2020-01-15T16:01:40.022492Z 0 [System] [MY-011503] [Repl] Plugin group_replication reported: 'Group membership changed to 127.0.0.1:13000, 127.0.0.1:13001 on view 15791040750193367:4.'
2020-01-15T16:01:40.024218Z 38 [System] [MY-011566] [Repl] Plugin group_replication reported: 'Setting super_read_only=OFF.'
2020-01-15T16:01:40.037718Z 38 [Warning] [MY-013460] [InnoDB] Clone removing all user data for provisioning: Started
2020-01-15T16:01:40.981419Z 38 [Warning] [MY-013460] [InnoDB] Clone removing all user data for provisioning: Finished
2020-01-15T16:01:44.612847Z 0 [Warning] [MY-010909] [Server] .../mysqld: Forcing close of thread 10  user: 'root'.
2020-01-15T16:01:44.612977Z 0 [Warning] [MY-010909] [Server] .../mysqld: Forcing close of thread 11  user: 'root'.
2020-01-15T16:01:44.613086Z 0 [Warning] [MY-010909] [Server] .../mysqld: Forcing close of thread 12  user: 'root'.
2020-01-15T16:01:48.159819Z 0 [System] [MY-011504] [Repl] Plugin group_replication reported: 'Group membership changed: This member has left the group.'
2020-01-15T16:01:48.162757Z 0 [System] [MY-011651] [Repl] Plugin group_replication reported: 'Plugin 'group_replication' has been stopped.'
2020-01-15T16:01:49.004835Z 0 [System] [MY-010910] [Server] .../mysqld: Shutdown complete (mysqld 8.0.20-tr-debug)  Source distribution.
2020-01-15T16:01:49.349021Z 0 [Warning] [MY-011068] [Server] The syntax '--master-retry-count' is deprecated and will be removed in a future release. Please use 'CHANGE MASTER TO master_retry_count = <num>' instead.
2020-01-15T16:01:49.349305Z 0 [Warning] [MY-010097] [Server] Insecure configuration for --secure-file-priv: Current value does not restrict location of generated files. Consider setting it to a valid, non-empty path.
2020-01-15T16:01:49.349401Z 0 [System] [MY-010116] [Server] .../mysqld (mysqld 8.0.20-tr-debug) starting as process 35369
2020-01-15T16:01:52.491553Z 0 [Warning] [MY-010068] [Server] CA certificate ca.pem is self signed.
2020-01-15T16:01:52.501836Z 0 [Warning] [MY-011810] [Server] Insecure configuration for --pid-file: Location '/export/home2/tmp' in the path is accessible to all OS users. Consider choosing a different directory.
2020-01-15T16:01:52.690301Z 0 [System] [MY-010931] [Server] .../mysqld: ready for connections. Version: '8.0.20-tr-debug'  socket: '/tmp/6yV80Bnmli/mysqld.2.sock'  port: 13001  Source distribution.
2020-01-15T16:01:52.735009Z 2 [System] [MY-010597] [Repl] 'CHANGE MASTER TO FOR CHANNEL 'group_replication_recovery' executed'. Previous state master_host='', master_port= 3306, master_log_file='', master_log_pos= 4, master_bind=''. New state master_host='<NULL>', master_port= 0, master_log_file='', master_log_pos= 4, master_bind=''.
2020-01-15T16:01:52.746740Z 2 [System] [MY-011565] [Repl] Plugin group_replication reported: 'Setting super_read_only=ON.'
2020-01-15T16:01:52.760301Z 2 [System] [MY-013587] [Repl] Plugin group_replication reported: 'Plugin 'group_replication' is starting.'
2020-01-15T16:01:52.778852Z 10 [System] [MY-010597] [Repl] 'CHANGE MASTER TO FOR CHANNEL 'group_replication_applier' executed'. Previous state master_host='<NULL>', master_port= 0, master_log_file='', master_log_pos= 4, master_bind=''. New state master_host='<NULL>', master_port= 0, master_log_file='', master_log_pos= 4, master_bind=''.
2020-01-15T16:01:52.841013Z 0 [System] [MY-011323] [Server] X Plugin ready for connections. Socket: '/tmp/6yV80Bnmli/mysqlx.2.sock' bind-address: '::' port: 13021
2020-01-15T16:01:57.001177Z 0 [System] [MY-013471] [Repl] Plugin group_replication reported: 'Distributed recovery will transfer data using: Incremental recovery from a group donor'
2020-01-15T16:01:57.002102Z 0 [System] [MY-011503] [Repl] Plugin group_replication reported: 'Group membership changed to 127.0.0.1:13000, 127.0.0.1:13001 on view 15791040750193367:6.'
2020-01-15T16:01:57.038572Z 26 [System] [MY-010597] [Repl] 'CHANGE MASTER TO FOR CHANNEL 'group_replication_recovery' executed'. Previous state master_host='<NULL>', master_port= 0, master_log_file='', master_log_pos= 4, master_bind=''. New state master_host='127.0.0.1', master_port= 13000, master_log_file='', master_log_pos= 4, master_bind=''.
2020-01-15T16:01:57.052970Z 27 [Warning] [MY-010897] [Repl] Storing MySQL user name or password information in the master info repository is not secure and is therefore not recommended. Please consider using the USER and PASSWORD connection options for START SLAVE; see the 'START SLAVE Syntax' in the MySQL Manual for more information.
2020-01-15T16:01:57.054298Z 27 [System] [MY-010562] [Repl] Slave I/O thread for channel 'group_replication_recovery': connected to master 'root@127.0.0.1:13000',replication started in log 'FIRST' at position 4
2020-01-15T16:01:57.184906Z 26 [System] [MY-010597] [Repl] 'CHANGE MASTER TO FOR CHANNEL 'group_replication_recovery' executed'. Previous state master_host='127.0.0.1', master_port= 13000, master_log_file='', master_log_pos= 4, master_bind=''. New state master_host='<NULL>', master_port= 0, master_log_file='', master_log_pos= 4, master_bind=''.
2020-01-15T16:01:57.199405Z 0 [System] [MY-011490] [Repl] Plugin group_replication reported: 'This server was declared online within the replication group.'
2020-01-15T16:01:57.201731Z 34 [System] [MY-011566] [Repl] Plugin group_replication reported: 'Setting super_read_only=OFF.'

Member join failing both on clone and incremental recovery

2020-01-15T16:13:46.636324Z 15 [System] [MY-011565] [Repl] Plugin group_replication reported: 'Setting super_read_only=ON.'
2020-01-15T16:13:46.648522Z 12 [System] [MY-013587] [Repl] Plugin group_replication reported: 'Plugin 'group_replication' is starting.'
2020-01-15T16:13:46.669809Z 16 [System] [MY-010597] [Repl] 'CHANGE MASTER TO FOR CHANNEL 'group_replication_applier' executed'. Previous state master_host='', master_port= 3306, master_log_file='', master_log_pos= 4, master_bind=''. New state master_host='<NULL>', master_port= 0, master_log_file='', master_log_pos= 4, master_bind=''.
2020-01-15T16:13:51.041737Z 0 [Warning] [MY-013469] [Repl] Plugin group_replication reported: 'This member will start distributed recovery using clone. It is due to the number of missing transactions being higher than the configured threshold of 1.'
2020-01-15T16:13:52.052223Z 0 [System] [MY-013471] [Repl] Plugin group_replication reported: 'Distributed recovery will transfer data using: Cloning from a remote group donor.'
2020-01-15T16:13:52.053077Z 0 [System] [MY-011503] [Repl] Plugin group_replication reported: 'Group membership changed to 127.0.0.1:13000, 127.0.0.1:13001 on view 15791048240388124:2.'
2020-01-15T16:13:52.054994Z 29 [System] [MY-011566] [Repl] Plugin group_replication reported: 'Setting super_read_only=OFF.'
2020-01-15T16:13:52.055122Z 28 [ERROR] [MY-013464] [Repl] Plugin group_replication reported: 'There was an issue when configuring the remote cloning process: Could not extract the access credentials for the clone process'
2020-01-15T16:13:52.055946Z 29 [System] [MY-011565] [Repl] Plugin group_replication reported: 'Setting super_read_only=ON.'
2020-01-15T16:13:52.056231Z 28 [Warning] [MY-013472] [Repl] Plugin group_replication reported: 'Due to some issue on the previous step distributed recovery is now executing: Incremental Recovery.'
2020-01-15T16:13:52.077350Z 30 [System] [MY-010597] [Repl] 'CHANGE MASTER TO FOR CHANNEL 'group_replication_recovery' executed'. Previous state master_host='', master_port= 3306, master_log_file='', master_log_pos= 4, master_bind=''. New state master_host='127.0.0.1', master_port= 13000, master_log_file='', master_log_pos= 4, master_bind=''.
2020-01-15T16:13:52.091428Z 31 [Warning] [MY-010897] [Repl] Storing MySQL user name or password information in the master info repository is not secure and is therefore not recommended. Please consider using the USER and PASSWORD connection options for START SLAVE; see the 'START SLAVE Syntax' in the MySQL Manual for more information.
2020-01-15T16:13:52.091492Z 31 [ERROR] [MY-013117] [Repl] Slave I/O for channel 'group_replication_recovery': Fatal error: Invalid (empty) username when attempting to connect to the master server. Connection attempt terminated. Error_code: MY-013117
2020-01-15T16:13:52.247705Z 30 [ERROR] [MY-011582] [Repl] Plugin group_replication reported: 'There was an error when connecting to the donor server. Please check that group_replication_recovery channel credentials and all MEMBER_HOST column values of performance_schema.replication_group_members table are correct and DNS resolvable.'
2020-01-15T16:13:52.247744Z 30 [ERROR] [MY-011583] [Repl] Plugin group_replication reported: 'For details please check performance_schema.replication_connection_status table and error log messages of Slave I/O for channel group_replication_recovery.'
2020-01-15T16:13:52.247954Z 30 [ERROR] [MY-011574] [Repl] Plugin group_replication reported: 'Maximum number of retries when trying to connect to a donor reached. Aborting group replication incremental recovery.'
2020-01-15T16:13:52.248037Z 30 [ERROR] [MY-011620] [Repl] Plugin group_replication reported: 'Fatal error during the incremental recovery process of Group Replication. The server will leave the group.'
2020-01-15T16:13:52.248215Z 30 [ERROR] [MY-011712] [Repl] Plugin group_replication reported: 'The server was automatically set into read only mode after an error was detected.'
2020-01-15T16:13:52.288121Z 12 [Warning] [MY-011645] [Repl] Plugin group_replication reported: 'Skipping leave operation: concurrent attempt to leave the group is on-going.'
2020-01-15T16:13:55.900306Z 0 [System] [MY-011504] [Repl] Plugin group_replication reported: 'Group membership changed: This member has left the group.'

Group of 5 members, 3 crashed, forced membership to 2 members

2020-02-12T16:06:57.102485Z 0 [Warning] [MY-011493] [Repl] Plugin group_replication reported: 'Member with address 127.0.0.1:13004 has become unreachable.'
2020-02-12T16:06:57.102552Z 0 [Warning] [MY-011493] [Repl] Plugin group_replication reported: 'Member with address 127.0.0.1:13003 has become unreachable.'
2020-02-12T16:06:57.102580Z 0 [Warning] [MY-011493] [Repl] Plugin group_replication reported: 'Member with address 127.0.0.1:13002 has become unreachable.'
2020-02-12T16:06:57.102611Z 0 [ERROR] [MY-011495] [Repl] Plugin group_replication reported: 'This server is not able to reach a majority of members in the group. This server will now block all updates. The server will remain blocked until contact with the majority is restored. It is possible to use group_replication_force_members to force a new group membership.'
2020-02-12T16:06:57.259465Z 34 [System] [MY-011722] [Repl] Plugin group_replication reported: 'The group_replication_force_members value 'localhost:13010,localhost:13011' was set in the group communication interfaces'
2020-02-12T16:07:01.102213Z 0 [Warning] [MY-011498] [Repl] Plugin group_replication reported: 'The member has resumed contact with a majority of the members in the group. Regular operation is restored and transactions are unblocked.'
2020-02-12T16:07:02.106636Z 0 [Warning] [MY-011499] [Repl] Plugin group_replication reported: 'Members removed from the group: 127.0.0.1:13002, 127.0.0.1:13004, 127.0.0.1:13003'
2020-02-12T16:07:02.107122Z 0 [System] [MY-011503] [Repl] Plugin group_replication reported: 'Group membership changed to 127.0.0.1:13001, 127.0.0.1:13000 on view 15815235821141496:6.'