WL#12826: Group Replication: cross-version policies
Affects: Server-8.0
—
Status: Complete
EXECUTIVE SUMMARY AND GOALS =========================== To support fast and agile development, MySQL 8.0 patch releases may introduce incompatible changes that Replication - and Group Replication in particular - must support. Group Replication does support different MySQL versions for a purpose like allowing a new member to join the group and selecting new primary etc. Group replication needs to make the necessary modifications to adapt to fast and agile development. GOALS ===== This worklog defines the behaviour and functional changes need to make Group Replication resilient to MySQL 8.0 fast and agile development. User stories ============ As a MySQL DBA, I want Group replication to maintain replication safety during group reconfigurations.
Functional requirements
=======================
FR 1: Primary election algorithm must choose PRIMARY member based on the
following order:
1. the lowest member version of the group
1.a. if all of the group members have a version greater than 8.0.16, then
patch level of members will also be considered to find the lowest member
version of the group
1.b. if any of the group member has a version less than or equal to 8.0.16
or has a member with 5.7 release level, then patch level of the member
must not be considered to find the lowest member version of the group
2. higher member weight within the lowest members of the group as per step-1
3. lexical order of server uuid within lowest members of the group with same
member weight
FR 2.1: Joiner member with a higher version than the existent lowest version
member (considering the patch version on 8.0) on a group must set read-only
when in multi-primary mode.
FR 2.2: On multi-primary mode, when a member leaves the group will make the
lowest version (considering the patch version on 8.0) member(s) writable.
FR 2.3: On multi-primary mode, when a 8.0 member with patch version less then
or equal to 8.0.16 joins the group, existing writable member(s) of the group
shall continue to be writable.
FR 2.4: On multi-primary mode, when a lower member version joins the group
using option group_replication_allow_local_lower_version_join, existing
writable member(s) of the group shall continue to be writable.
FR 3.1: Joiner member with a lower version than the existent lowest version
member (considering the patch version on 8.0) on a group shall not join the
group.
FR 3.2: Member with version less then or equal to 8.0.16 shall join the group
consisting of 8.0 member(s)(at any patch level), ignoring the patch version
of the group.
FR 3.3: A lower member version may join the group using option
group_replication_allow_local_lower_version_join.
FR 4.1: Recovery donor shall have a lower or equal version
(considering the patch version on 8.0) than the joiner.
FR 4.2: If the option group_replication_allow_local_lower_version_join is set,
rule that a donor should have equal or lower version than the joiner will be
ignored.
Ideally group should have members with same version including patch level.
However during the upgrade the versions of group members will differ.
SUMMARY
=======
PRIMARY MEMBER SELECTION
------------------------
If all the members of the group have a version greater than 8.0.16 then the
Primary member election algorithm will also consider the patch level of the
group members for selecting a new primary.
If any of the member in the group has a version lower than or equal to 8.0.16
then the Primary member election algorithm will not consider the patch
level of the group members for selecting a new primary.
If primary member is changed using UDF listed below
group_replication_set_as_primary(server_uuid) OR
group_replication_switch_to_single_primary_mode([server_uuid]) then:
1. If any of the member in the group has version 5.7 or version less than
8.0.13, function will not make any primary member changes.
2. If any of the member in the group has version between 8.0.13 and 8.0.16
(including) and no other member in the group has version lower than 8.0.13
then:
2.a: If server_uuid is provided with major version 8, that server will be
made primary member else error will be thrown.
2.b: If server_uuid is not provided then election will happen considering
only major release of the group members version.
3. If all the members of the group have a version greater than 8.0.16 then:
3.a: If server_uuid is provided, that server will be made primary member,
if its version is lowest (considering the patch level) in the group,
else an error will be thrown.
3.b: If server_uuid is not provided then election will happen considering
the group members version till patch level.
NOTE: For UDF group_replication_set_as_primary server_uuid is
mandatory. UDF will fail if server_uuid is not provided.
Examples
E1: Group members after the previous primary left:
5.7.22,
8.0.20,
8.0.20
The server with 5.7.22 will be elected as the primary since its lowest version
of the group.
E2: Group members after the previous primary left:
8.0.19,
8.0.20,
8.0.20
The server with 8.0.19 will be elected as the primary since its lowest version.
E3: Group members after the previous primary left:
8.0.19 weight: 90,
8.0.20 weight 50,
8.0.20 weight 90,
8.0.20 weight 95
The server with 8.0.19 weight: 90 will be elected as the primary since its
lowest version of the group.
E4: Group members after the previous primary left:
8.0.19 weight 90 uuid: 5a5d0f6e-6ad1-11e7-9aee-f48c5048ab0c,
8.0.19 weight 90 uuid: 5a67adc9-6ad1-11e7-9b1f-f48c5048ab0c,
8.0.19 weight 50 uuid: 5a6e5078-6ad1-11e7-9bce-f48c5048ab0c
The server with 8.0.19 weight: 90 uuid: 5a5d0f6e-6ad1-11e7-9aee-f48c5048ab0c
will be elected as the primary member since its lowest version of the group
with higher member weight and lowest server uuid.
E5: Group members during multi-primary-mode to single-primary-mode switch:
8.0.19 weight 50,
8.0.20 weight 90,
8.0.20 weight 95
The server with 8.0.19 weight: 50 will be elected as the primary since its
lowest patch version of the group.
E6: Group members during multi-primary-mode to single-primary-mode switch:
8.0.14 weight 90,
8.0.20 weight 50,
8.0.20 weight 90,
8.0.20 weight 95
The server with 8.0.20 weight: 95 will be elected as the primary since its
lowest major version (patch version ignored due to 8.0.14 presence in group)
of the group.
E7: Group members in single-primary-mode, change of primary to 8.0.21 weight 50:
8.0.14 weight 90 uuid: 4a5d0f6e-6ad1-11e7-9aee-f48c5048ab0c (old primary),
8.0.20 weight 70 uuid: 5a5d0f6e-6ad1-11e7-9aee-f48c5048ab0c (secondary),
8.0.21 weight 50 uuid: 6a5d0f6e-6ad1-11e7-9aee-f48c5048ab0c (new requested
primary),
Command executed:
SELECT group_replication_set_as_primary("6a5d0f6e-6ad1-11e7-9aee-f48c5048ab0c")
The server with 8.0.21 will be elected as the primary (patch version ignored
due to 8.0.14 presence in group) of the group.
WRITE VERSION COMPATIBILITY
--------------------------
If new member version is higher than the lowest member version (considering
the patch on 8.0) of the group then new member will be in read-only mode.
This only applies to multi-primary mode.
If group enters multi-primary mode by executing UDF
group_replication_switch_to_multi_primary_mode, lowest member versions of the
group (considering the patch version) will be writable while other members
will be in read-only.
Examples
E1 Group members in multi-primary-mode:
8.0.19,
8.0.20
Server with version 8.0.19 will be writable while 8.0.20 will be read_only.
E2 Group members during single-primary-mode to multi-primary-mode switch:
8.0.14(Secondary),
8.0.15(Secondary),
8.0.20(Secondary),
8.0.21(Primary),
Server with version 8.0.14 and 8.0.15 will be writable (already out releases)
while 8.0.20 and 8.0.21 will be read_only.
Primary server 8.0.21 will enable read_only mode.
E3 Group members in multi-primary-mode:
8.0.20,
8.0.21,
Server 8.0.20 is writable and 8.0.21 is read-only.
Server with version 8.0.12 enters the group.
Server 8.0.20 will continue to be writable. Server 8.0.21 will continue to be
read-only.
E4 Group members in multi-primary-mode:
8.0.20,
8.0.21,
Server 8.0.20 is writable and 8.0.21 is read-only.
Server with version 5.7.21 enters the group using option
group_replication_allow_local_lower_version_join.
Server 8.0.20 will continue to be writable. Server 8.0.21 will continue to be
read-only.
LOWEST VERSION COMPATIBILITY
----------------------------
If new member version is lower than the lowest member version (considering the
patch on 8.0) of the group then new member will not join the group.
Examples
E1: Group members:
8.0.19,
8.0.20,
8.0.20
Servers with version 8.0.18 or lower are not allowed to join.
NOTE:
In 5.7 releases and 8.0 releases till version 8.0.16, new member version is
compared with version of each group member including self. (not considering
patch level).
This is equivalent to comparing new member version with highest member version
of the group(without patch level).
Present behaviour example: 5.7.21, 8.0.14. Now 5.7.21 cannot join group, it
fails comparison with 8.0.14. Post this WL change 5.7.21 will be able to join
group since only comparison with 5.7.21 will be done.
DONOR VERSION COMPATIBILITY
---------------------------
A. While forming the list of possible donors group ONLINE members with equal
or lower version than the new member will be considered.
B. If allow_local_lower_version_join is set, all ONLINE members of group will
be a valid donor since there will be no equal or lower version member.
Examples
E1: Group members:
5.7.22,
8.0.20,
8.0.21
A new member 8.0.20 can use 5.7.22 OR 8.0.20 as a donor.
Security
--------
Not applicable.
Upgrade
-------
There will be impact post the Upgrade process since the server will re-join
with different server version after the process.
Post upgrade if server release is greater than 8.0.16, then depending upon
the versions of the members in the group, the new member may
behave differently as listed below:
1. Member will not join the group if its version is lower than the lowest
member version of the group (considering patch level)
1.a: If post upgrade, versions are not going to be same patch level,
new lowest member version should be first started.
2. Member will be in the read-only in multi-primary mode if its version is
not lowest in the group (considering patch level)
2.a: This will impact in-place upgrade, if upgrade is at patch level, post
this WL, number of write members available in multi-primary mode will
be less during upgrade process, since post upgrade members will set
read-only
2.b: During in-place upgrade, once all members are upgraded, lowest members
of the group (including patch level) will become writable and there
will be no need to manually set super_read_only=OFF.
3. If all the members in the group are at versions greater than 8.0.16 then
member is allowed to become primary, only if member is lowest version
(including patch level) in the group.
3.a: If primary has to stay same post upgrade, primary only upgrade
(including patch level) is not possible post this WL. Secondary
member(s) of the group should be upgraded to version higher or equal to
that of Primary member version.
Examples
E1(Single-primary upgrade)
M1: 8.0.20 (old primary)
M2: 8.0.20 (new primary)
M3: 8.0.20
DBA wants to upgrade all group members to 8.0.21
Set higher member weight on M2 and upgrade M2.
Upgrade M3 and M1 now.
Post M1 and M3 upgrade M2 will become Primary.
E2(Single-primary upgrade)
M1: 8.0.20 (primary)
M2: 8.0.20
M3: 8.0.20
DBA wants to upgrade all group members to 8.0.21
Upgrade M2 and M3 to version 8.0.21 or higher.
Post M2 and M3 upgrade, M1 can be upgraded to 8.0.21. Using UDF
group_replication_set_as_primary M1 can become primary.
E3:(Multi-primary upgrade)
M1: 8.0.20,
M2: 8.0.20
DBA wants to upgrade all group members to 8.0.21
Upgrade M1.
Post upgarde of M1, M1 will be read-only till M2 is upgraded to 8.0.21.
Once M2 is upgraded to 8.0.21, M1 will become writable.
E4:(Multi-primary upgrade)
M1: 8.0.20,
M2: 8.0.20
DBA wants to upgrade M1 to 8.0.21 from 8.0.20 and M2 to 8.0.22 from 8.0.20.
M1 should be upgraded first. M2 will be read-only post upgrade.
Cross-version Replication
-------------------------
A member will not be allowed to join if its version is lower than the lowest
version of the group.
A member will join group in read-only mode if the member version is higher
than the lowest version of the group.
Protocol
--------
Not applicable.
User Interface
--------------
No changes.
Observability
-------------
Not applicable.
Deployment and Installation
---------------------------
No changes.
LLD:
# Changes to be committed:
# modified: mysql-
test/suite/group_replication/r/gr_compatibility_rules.result
# modified: mysql-
test/suite/group_replication/r/gr_compatibility_rules_and_force_lower_version_me
mber.result
# new file: mysql-
test/suite/group_replication/r/gr_mru_lower_member_join.result
# new file: mysql-
test/suite/group_replication/r/gr_mru_member_leave.result
# new file: mysql-
test/suite/group_replication/r/gr_mru_primary_election.result
# modified: mysql-
test/suite/group_replication/r/gr_multiple_version_members.result
# modified: mysql-
test/suite/group_replication/t/gr_compatibility_rules.test
# modified: mysql-
test/suite/group_replication/t/gr_compatibility_rules_and_force_lower_version_me
mber.test
# modified: mysql-
test/suite/group_replication/t/gr_exit_state_action_on_join_lower_version.test
# new file: mysql-
test/suite/group_replication/t/gr_mru_lower_member_join.cnf
# new file: mysql-
test/suite/group_replication/t/gr_mru_lower_member_join.test
# new file: mysql-
test/suite/group_replication/t/gr_mru_member_leave.test
# new file: mysql-
test/suite/group_replication/t/gr_mru_primary_election.cnf
# new file: mysql-
test/suite/group_replication/t/gr_mru_primary_election.test
# modified: mysql-
test/suite/group_replication/t/gr_multiple_version_members.test
# modified: mysql-
test/suite/group_replication/t/gr_primary_mode_group_operations_legacy_election.
test
# modified: plugin/group_replication/include/compatibility_module.h
# modified: plugin/group_replication/src/compatibility_module.cc
# modified: plugin/group_replication/src/gcs_event_handlers.cc
# modified: plugin/group_replication/src/plugin.cc
# modified:
plugin/group_replication/src/plugin_handlers/primary_election_invocation_handler
.cc
# modified: plugin/group_replication/src/recovery_state_transfer.cc
# modified:
unittest/gunit/group_replication/group_replication_compatibility_module-t.cc
1. Function is_agile identifies if particular version has agile related changes.
Some FRs may need it to confirm all members in GROUP follow agile model.
+int is_agile(Member_version version) {
+ uint32 major_version = version.get_major_version ();
+ uint32 minor_version = version.get_minor_version ();
+ uint32 patch_version = version.get_patch_version ();
+ return (major_version >= 8 &&
+ ((major_version == 0x08 && minor_version == 0x00) ?
+ patch_version >= 0x16 : true));
+ // return (version > Member_version(8.0.15) ? 1 : 0);
+}
+
2. Version comparison check,
Function: check_incompatibility
In function check_incompatibility, if version being compared is lowest version
then:
- return COMPATIBLE, if member version is equal to lowest version.
- return READ_COMPATIBLE, if member version is higher then lowest version.
- return INCOMPATIBLE, if member version is lower then lowest version.
We simply do: return (my_version > group_lowest_version) ? READ_COMPATIBLE :
INCOMPATIBLE_LOWER_VERSION;
3. Finding version with which member version compatibility has to be checked,
function check_version_compatibility_with_group
We create a set of unique versions: std::set unique_version_set;
We store group lowest version in: Member_version lowest_version;
Iterate over all group_members(Not self) and find lowest version and update
set with each unique version.
3.a. For each unique version check if someone has manually added in-compatiblity
via : check_incompatibility -> check_local_incompatibility
This is exception where we add special incompatiblity via add_incompatiblity
function for range of MySQL Version.
Else looping through each version is not needed.
3.b. For lowest version, check member version compatibility, read incompatible
or incompatible.
This fulfills FR2, FR3.
4. For primary election we already have sorted algorithm, sorting algorithm
moves lowest version pointer considerinlg only major and minor version.
If the lowest version i.e. first_member->get_member_version() is MRU consider
patch version too during comparison.
File: plugin_handlers/primary_election_invocation_handler.cc
Function: sort_and_get_lowest_version_member_position
+ if (is_agile(first_member->get_member_version()) &&
+ !(first_member->get_member_version() == (*it)->get_member_version())) {
+ lowest_version_end = it;
+ break;
+ }
if (lowest_major_version !=
(*it)->get_member_version().get_major_version()) {
lowest_version_end = it;
5. For donors add only compatible member while looping through each member.
if (get_allow_local_lower_version_join()) {
if (is_online && not_self)
suitable_donors.push_back(member);
}
else if (is_online && not_self && member->get_member_version() <=
local_member_info->get_member_version()) {
suitable_donors.push_back(member);
}
5.b. Exception
When force_lower_version is set there will not no lower version to recover from.
Then FR4 will not work, so all ONLINE members will be considered when
force_lower_version is set.
6. Improvements in error logging:
6.a. Enum typedef enum st_compatibility_types may be checked to identify exact
issue, incompatible or read_compatible.
Copyright (c) 2000, 2024, Oracle Corporation and/or its affiliates. All rights reserved.