WL#10412: Group Replication: notify listener services on relevant group events
Affects: Server-8.0
—
Status: Complete
EXECUTIVE SUMMARY ================= This worklog implements a notification framework in the group replication plugin. On view changes, recovery state updates, network partitioning and primary election, the plugin will call out to listeners registered in the service registry and notify listeners that an event has ocurred. X plugin (or other listeners) can then take action/react. IMPORTANT NOTICES ================= The overall direction in this area is that the MySQL server will be able to notify client side applications, that rely on the X protocol to speak to MySQL, about meaningful events. For instance, the MySQL server will notify the Router that the membership has changed and some server has now gone away. Then the router can take action and update routing tables or instruct slave servers to reconnect to other master servers in the group. (Maybe not the router in this case, but some orchestration mechanism.) This worklog adds a framework, relying on the components infrastructure, that can be used by the X plugin to subscribe to membership and other meaningful events triggered by the Group Replication plugin. This will allow X connections to get notifications about such events. For HA and orchestration purposes, this is significantly important, since client applications need not poll the server to get this information, but instead just need to subscribe to it. This worklog is a stepping stone towards this mode of operations, since it implements a notification framework inside the server for events related to Group Replication. Consequently, relevant GR events can be propagated to the X plugin which in turn will propagate these to the external subscribers through an X connection. We are doing this to be able to deliver the stage 3 of the next generation MySQL server roadmap. USER STORIES ============ - As an X plugin developer, I want to receive notifications when the group membership changes so I can refresh the membership information from the performance schema tables and push that information to the router. - As an X plugin developer, I want to receive notifications when the system runs into a network partition, so I can refresh the status information about my system (and perhaps push it to the router). - As an X plugin developer, I want to receive notifications when a member in the group finishes recovery, so I can refresh the status information about the system (and perhaps push it to the router). - As an X plugin developer I want to receive notifications when the primary changes so I can relay this information to the router. - As a InnoDB cluster developer, I want to be notified of group view changes, so that I can update cluster topology information in places like the metadata schema, DNS, etcd, zookeeper etc - As an X plugin developer, I want to be notified of group view changes so that I can take actions such as disconnecting sessions that have certain expectations about the group (eg that it is connected to the primary) or updating version tokens, so that requests operating under a certain set
Functional Requirements
=======================
FR1. Whenever a member joins a group, a new view SHALL be installed and
the view change listener services SHALL be notified.
FR2. Whenever a member leaves the group, a new view will be installed
and the view change listener services SHALL be notified.
FR3. Whenever the group is bootstrapped and the PRIMARY is elected,
member role listener services SHALL be notified.
FR4. Whenever the PRIMARY member is removed from the group a
secondary is elected and then member role listener services SHALL
be notified.
FR5. Whenever there is a member state update, member state update
listener services SHALL be notified.
FR6. Whenever a majority of members becomes UNREACHABLE, quorum
loss listener services SHALL be notified.
FR7. The following table depicts the set of cases where notifications
SHALL be emitted.
QL - Quorum Lost event, VC - View Changed event,
RC - Role Changed event, SC - State Changed event
|----+---------------------------------------------+------------+--------|
| | Scenario\Where | | |
| | | Server | Others |
| | | triggering | |
| | | event | |
|----+---------------------------------------------+------------+--------|
| 1 | SERVER BOOTSTRAPS GROUP | VC,2xSC | N/A |
| 2 | SERVER JOINS and STARTS RECOVERY | VC,SC | VC,SC |
| 3 | SERVER RECOVERED | SC | SC |
| 4 | SERVER LEAVES GRACEFULLY | VC,SC | VC |
|----+---------------------------------------------+------------+--------|
| 5 | SERVER BOOTSTRAPS+RECOVERS+PRIMARY ELECTION | VC,RC,2xSC | |
| 6 | PRIMARY LEAVES | VC,SC,RC | VC,RC |
|----+---------------------------------------------+------------+--------|
| 7 | A SERVER BECOMES UNREACHABLE | SC, VC | N/A |
| 8 | MAJORITY UNREACHABLE | QL, SC | N/A |
| 9 | MAJORITY UNREACHABLE+FORCE MEMBERS | VC | N/A |
| 10 | MAJORITY UNREACHABLE+STOP | VC, SC | N/A |
|----+---------------------------------------------+------------+--------|
Non-Functional Requirements
===========================
NFR1. The notification implementation will rely on the service registry.
NFR2. These events are likely to be rare, otherwise, there is too much entropy
in the system to be usable. As such, they shall not affect performance
SUMARY OF THE APPROACH
======================
By taking advantage of the service registry, the group replication plugin will
forward the notifications it receives from GCS and those that it generates as
well. Forwarding means that it will iterate over service implementations
registered in the service registry, with a given service name, and call out to
their interface.
This will be done by the GCS receiver thread, which calls into the GCS
handlers registered by group replication on the GCS layer. Therefore, the
notification callback must not block or take significant amount of time to
process.
In pseudo-code, and oversimplified, this is how the act of notifying a
listener looks like, when called from the GCS handler (in this
specific example, we are considering only the on_view_change event,
but notifications can be triggered by other GCS events as well, such
as on_suspicion or on_message):
on_view_cange(view):
[...]
# notify view change listeners
gms_listeners= service_registry.find("group_membership_listener")
for l in gms_listeners:
l.notify_view_change(view.view_id())
DEFINITIONS
===========
role - for now, the role the member is given in single primary mode: primary
or secondary
state - The state of the member: UNREACHABLE, ONLINE, ERROR, OFFLINE, RECOVERING
NOTIFICATIONS AND LISTENERS (SERVICES)
======================================
There is no user visible behavior change in this worklog. There is however a
set of new service interfaces. These should be implemented by other components
in case they want to be notified about events coming from the group.
The GR plugin will interact with the service registry to reach out to the
service implementations registered and notify them about membership changes
and other significant events in the group.
We have split the notifications list into 4 types: (a) VIEW CHANGE; (b) ROLE
CHANGE; (c) QUORUM LOSS; (d) STATE CHANGE. We have chosen to split the
listener services into two as well. One that deals with group membership
events, (a) and (c) above, and one that deals with member changes (b) and (d)
above.
The notification types and the service interfaces are described below.
LOGICAL NOTIFICATIONS
---------------------
The following is a list of events that SHALL trigger a
notification. Note that a single GCS event may trigger 1 or more
notifications. For instance, when a view changes it may also trigger a
primary election. Then, as stated in the requirements section, there
shall be two notifications, one for the role change and one for the
view change.
* VIEW CHANGE
- This notifies that a view change has happened, i.e., a
server has joined or left the group.
Whenever a view changes, the set of active servers in the group can
be known by querying the performance schema table,
replication_group_members.
* ROLE CHANGE
- This notifies that the role of one of the servers in the group
has changed.
A primary changes when:
* The current primary is removed from the group and a new one
is elected.
* When there is no primary selected in the group to begin with.
The group is bootstrapped, the server that starts the group
installs a new view - with only itself in it - and selects
the primary - itself.
Whenever the primary changes, the server UUID of the new primary can
be known by selecting the status variable
group_replication_primary_member from the global_status performance
schema table.
* QUORUM LOSS
- This notifies that a majority of members in the current view
have become unreachable. The current view is the view reported
in performance schema table replication_group_member_stats and
the current members of the view are reported in
replication_group_members table.
Whenever this notification is triggered it means that the state of
50% or more of the servers listed in the performance schema table
replication_group_members is UNREACHABLE.
* STATE CHANGE
- This notifies that a server in the group has finished recovery.
A server changes finishes recovery, when it moves into ONLINE state.
Whenever this notification is triggered, then the state of a server,
as seen in the performance schema table replication_group_members,
has changed.
Each of these notifications map into a callback into the interfaces defined
for the services. Below are the service interfaces.
LISTENER SERVICES
-----------------
The interface devised for the listener is the following one
(note that the recommendation for creating services is to
avoid using C++ classes as arguments. That's the reason why
we do not use std::string below for the view_id arguments):
/**
A service that listens for notifications about view changes or
quorum loss.
@note A GCS event may trigger more than one notification. For
example, when installing a view, it may also trigger a primary
election, thus two notifications shall be emitted.
*/
BEGIN_SERVICE_DEFINITION(group_membership_listener)
/**
This function SHALL be called whenever there is a new view
installed.
The implementation SHALL consume the notification and
return false on success, true on failure.
The implementation MUST NOT block the caller. It MUST
handle the notification quickly or enqueue it and deal
with it asynchronously.
@param view_id The view identifier. This must be copied
if the string must outlive the notification
lifecycle.
@return false success, true on failure.
*/
DECLARE_BOOL_METHOD(notify_view_change, (const char* view_id));
/**
This function SHALL be called whenever the number of members
that show state UNREACHABLE is equal or larger than the number
of members that are not ONLINE or RECOVERING.
The implementation SHALL consume the notification and
return false on success, true on failure.
The implementation MUST NOT block the caller. It MUST
handle the notification quickly or enqueue it and deal
with it asynchronously.
@param view_id The view identifier. This must be copied
if the string must outlive the notification
lifecycle.
@return false success, true on failure.
*/
DECLARE_BOOL_METHOD(notify_quorum_loss, (const char* view_id));
END_SERVICE_DEFINITION(group_membership_listener)
/**
A service that listens for notifications about member state
or member role updates.
@note A GCS event may trigger more than one notification. For
example, when installing a view, it may also trigger a primary
election, thus two notifications shall be emitted.
*/
BEGIN_SERVICE_DEFINITION(group_member_status_listener)
/**
This function SHALL be called whenever the role of a member
changes.
The implementation SHALL consume the notification and
return false on success, true on failure.
The implementation MUST NOT block the caller. It MUST
handle the notification quickly or enqueue it and deal
with it asynchronously.
@param view_id The view identifier. This must be copied
if the string must outlive the notification
lifecycle.
@return false success, true on failure.
*/
DECLARE_BOOL_METHOD(notify_member_role_change, (const char* view_id));
/**
This function SHALL be called whenever the state of a member
changes.
The implementation SHALL consume the notification and
return false on success, true on failure.
The implementation MUST NOT block the caller. It MUST
handle the notification quickly or enqueue it and deal
with it asynchronously.
@param view_id The view identifier. This must be copied
if the string must outlive the notification
lifecycle.
@return false success, true on failure.
*/
DECLARE_BOOL_METHOD(notify_member_state_change, (const char* view_id));
END_SERVICE_DEFINITION(group_member_status_listener)
UPGRADE / DOWNGRADE
===================
This worklog is about internal plumbing to get notifications from GR into
other components in the server. Therefore, since supported plugins are
released together, there is no chance that a plugin engages an incompatible
At the low level, the following changes will need to take place:
C1. Add the new service interfaces to components interfaces placeholder in the
mysql-trunk tree. This way, the interfaces become public.
The interfaces are placed in the following files.
include/mysql/components/services/group_member_status_listener.h
include/mysql/components/services/group_membership_listener.h
C2. Change GR's implementation of the GCS event handlers so that these will
capture which GCS events should actually trigger a notification to the
listeners registered in the service registry.
The notifications one needs to forward are not plain vanilla events coming
from GCS. Sometimes, they need to be triggered after GR has handled them
(e.g., after a primary election, a recovery that finished, etc).
To be able to track if an event from GCS actually resulted on a meaningful
notification to be fired to the listeners (quorum loss, primary election,
etc), we introduce a context event in the GR's implementation of the GCS
handlers. This context, tracks what GCS events actually should result also
in notifications to the listeners in the service registry.
For example, an inner class of the Plugin_gcs_events_handler:
class Event_context
{
private:
bool m_member_role_changed;
bool m_member_state_changed;
bool m_quorum_lost;
bool m_view_changed;
const char* m_view_id;
public:
Event_context() :
m_member_role_changed(false),
m_member_state_changed(false),
m_quorum_lost(false),
m_view_changed(false),
m_view_id(NULL)
{}
void reset()
{
m_member_role_changed= false;
m_member_state_changed= false;
m_view_changed= false;
m_quorum_lost= false;
m_view_id= NULL;
}
void set_member_role_changed() { m_member_role_changed= true; }
void set_member_state_changed() { m_member_state_changed= true; }
void set_quorum_lost() { m_quorum_lost= true; }
void set_view_changed() { m_view_changed= true; }
void set_view_id(const char *v) { m_view_id= v; }
bool get_member_role_changed() { return m_member_role_changed; }
bool get_member_state_changed() { return m_member_state_changed; }
bool get_quorum_lost() { return m_quorum_lost; }
bool get_view_changed() { return m_view_changed; }
const char* get_view_id() { return m_view_id; }
};
At the end of every GCS event handling, if the context flags that a
notification to the listeners in the service registry is required, then
such notification is forwarded.
This context is reset every time the GCS handler function returns.
C3. Implement a notification forwarder function. It is responsible to acquire
the service registry and registry query service handles. Then iterate over
the listener implementations in the registry and call out to them.
Something like:
if (notify(kGroupMembership, r, rq, evts_ctx))
{
log_message(MY_ERROR_LEVEL,
"An undefined error was found while broadcasting an "
"internal group membership notification! "
"This is likely to happen if your components or plugins "
"are not properly loaded or are malfunctioning!");
}
/* notify member status events listeners. */
if (notify(kGroupMemberStatus, r, rq, evts_ctx))
{
log_message(MY_ERROR_LEVEL,
"An undefined error was found while broadcasting an "
"internal group member status notification! "
"This is likely to happen if your components or plugins "
"are not properly loaded or are malfunctioning!");
}
For not acquiring the registry and registry query everytime, we will
introduce a façade to the two registry services used every time
(service registry and registry query service). The façade will also
cache these references. Will acquire them at start up of group
replication and will release them when it group replication is stopped.
Thence there will be a Registry_module that will take ownership of the
setup and teardown of these handles for group replication facilities.
Depending on the event context (see C2) this notification forwarder is
called after the gcs handlers implementation have processed the
notification from the GCS layer.
C4. Change Group Replication CMake to also build any additional C/C++ files.
Copyright (c) 2000, 2025, Oracle Corporation and/or its affiliates. All rights reserved.