WL#10412: Group Replication: notify listener services on relevant group events

EXECUTIVE SUMMARY
=================

This worklog implements a notification framework in the group replication
plugin. On view changes, recovery state updates, network partitioning and
primary election, the plugin will call out to listeners registered in the
service registry and notify listeners that an event has ocurred. X plugin (or
other listeners) can then take action/react.

IMPORTANT NOTICES
=================

The overall direction in this area is that the MySQL server will be able to
notify client side applications, that rely on the X protocol to speak to
MySQL, about meaningful events. For instance, the MySQL server will notify the
Router that the membership has changed and some server has now gone away. Then
the router can take action and update routing tables or instruct slave servers
to reconnect to other master servers in the group. (Maybe not the router in
this case, but some orchestration mechanism.)

This worklog adds a framework, relying on the components infrastructure, that
can be used by the X plugin to subscribe to membership and other meaningful
events triggered by the Group Replication plugin. This will allow X
connections to get notifications about such events.

For HA and orchestration purposes, this is significantly important, since
client applications need not poll the server to get this information, but
instead just need to subscribe to it. This worklog is a stepping stone towards
this mode of operations, since it implements a notification framework inside
the server for events related to Group Replication. Consequently, relevant GR
events can be propagated to the X plugin which in turn will propagate these to
the external subscribers through an X connection.

We are doing this to be able to deliver the stage 3 of the next generation
MySQL server roadmap.

USER STORIES
============

- As an X plugin developer, I want to receive notifications when the   group
  membership changes so I can refresh the membership information from the
  performance schema tables and push that information to the router.

- As an X plugin developer, I want to receive notifications when the system
  runs into a network partition, so I can refresh the status information about
  my system (and perhaps push it to the router).

- As an X plugin developer, I want to receive notifications when a member in
  the group finishes recovery, so I can refresh the status information about
  the system (and perhaps push it to the router).

- As an X plugin developer I want to receive notifications when the primary
  changes so I can relay this information to the router.

- As a InnoDB cluster developer, I want to be notified of group view changes,
  so that I can update cluster topology information in places  like the
  metadata schema, DNS, etcd, zookeeper etc

- As an X plugin developer, I want to be notified of group view changes so
  that I can take actions such as disconnecting sessions that have certain
  expectations about the group (eg that it is connected to the primary) or
  updating version tokens, so that requests operating under a certain set

Functional Requirements
=======================

FR1. Whenever a member joins a group, a new view SHALL be installed and
     the view change listener services SHALL be notified.

FR2. Whenever a member leaves the group, a new view will be installed
     and the view change listener services SHALL be notified.

FR3. Whenever the group is bootstrapped and the PRIMARY is elected,
     member role listener services SHALL be notified.

FR4. Whenever the PRIMARY member is removed from the group a
     secondary is elected and then member role listener services SHALL
     be notified.

FR5. Whenever there is a member state update, member state update
     listener services SHALL be notified.

FR6. Whenever a majority of members becomes UNREACHABLE, quorum
     loss listener services SHALL be notified.

FR7. The following table depicts the set of cases where notifications
     SHALL be emitted.

  QL - Quorum Lost event, VC - View Changed event,
  RC - Role Changed event, SC - State Changed event

 |----+---------------------------------------------+------------+--------|
 |    | Scenario\Where                              |            |        |
 |    |                                             | Server     | Others |
 |    |                                             | triggering |        |
 |    |                                             | event      |        |
 |----+---------------------------------------------+------------+--------|
 |  1 | SERVER BOOTSTRAPS GROUP                     | VC,2xSC    | N/A    |
 |  2 | SERVER JOINS and STARTS RECOVERY            | VC,SC      | VC,SC  |
 |  3 | SERVER RECOVERED                            | SC         | SC     |
 |  4 | SERVER LEAVES GRACEFULLY                    | VC,SC      | VC     |
 |----+---------------------------------------------+------------+--------|
 |  5 | SERVER BOOTSTRAPS+RECOVERS+PRIMARY ELECTION | VC,RC,2xSC |        |
 |  6 | PRIMARY LEAVES                              | VC,SC,RC   | VC,RC  |
 |----+---------------------------------------------+------------+--------|
 |  7 | A SERVER BECOMES UNREACHABLE                | SC, VC     | N/A    |
 |  8 | MAJORITY UNREACHABLE                        | QL, SC     | N/A    |
 |  9 | MAJORITY UNREACHABLE+FORCE MEMBERS          | VC         | N/A    |
 | 10 | MAJORITY UNREACHABLE+STOP                   | VC, SC     | N/A    |
 |----+---------------------------------------------+------------+--------|

Non-Functional Requirements
===========================

NFR1. The notification implementation will rely on the service registry.

NFR2. These events are likely to be rare, otherwise, there is too much entropy
      in the system to be usable. As such, they shall not affect performance

SUMARY OF THE APPROACH
======================

By taking advantage of the service registry, the group replication plugin will
forward the notifications it receives from GCS and those that it generates as
well. Forwarding means that it will iterate over service implementations
registered in the service registry, with a given service name, and call out to
their interface.

This will be done by the GCS receiver thread, which calls into the GCS
handlers registered by group replication on the GCS layer. Therefore, the
notification callback must not block or take significant amount of time to
process.

In pseudo-code, and oversimplified, this is how the act of notifying a
listener looks like, when called from the GCS handler (in this
specific example, we are considering only the on_view_change event,
but notifications can be triggered by other GCS events as well, such
as on_suspicion or on_message):

  on_view_cange(view):
    [...]

    # notify view change listeners
    gms_listeners= service_registry.find("group_membership_listener")
    for l in gms_listeners:
      l.notify_view_change(view.view_id())

DEFINITIONS
===========

role  - for now, the role the member is given in single primary mode: primary
        or secondary
state - The state of the member: UNREACHABLE, ONLINE, ERROR, OFFLINE, RECOVERING

NOTIFICATIONS AND LISTENERS (SERVICES)
======================================

There is no user visible behavior change in this worklog. There is however a
set of new service interfaces. These should be implemented by other components
in case they want to be notified about events coming from the group.

The GR plugin will interact with the service registry to reach out to the
service implementations registered and notify them about membership changes
and other significant events in the group.

We have split the notifications list into 4 types: (a) VIEW CHANGE; (b) ROLE
CHANGE; (c) QUORUM LOSS; (d) STATE CHANGE. We have chosen to split the
listener services into two as well. One that deals with group membership
events, (a) and (c) above, and one that deals with member changes (b) and (d)
above.

The notification types and the service interfaces are described below.

LOGICAL NOTIFICATIONS
---------------------

The following is a list of events that SHALL trigger a
notification. Note that a single GCS event may trigger 1 or more
notifications. For instance, when a view changes it may also trigger a
primary election. Then, as stated in the requirements section, there
shall be two notifications, one for the role change and one for the
view change.

* VIEW CHANGE

  - This notifies that a view change has happened, i.e., a
    server has joined or left the group.

  Whenever a view changes, the set of active servers in the group can
  be known by querying the performance schema table,
  replication_group_members.

* ROLE CHANGE

  - This notifies that the role of one of the servers in the group
    has changed.

    A primary changes when:

    * The current primary is removed from the group and a new one
      is elected.
    * When there is no primary selected in the group to begin with.

      The group is bootstrapped, the server that starts the group
      installs a new view - with only itself in it - and selects
      the primary - itself.

  Whenever the primary changes, the server UUID of the new primary can
  be known by selecting the status variable
  group_replication_primary_member from the global_status performance
  schema table.

* QUORUM LOSS

  - This notifies that a majority of members in the current view
    have become unreachable. The current view is the view reported
    in performance schema table replication_group_member_stats and
    the current members of the view are reported in
    replication_group_members table.

  Whenever this notification is triggered it means that the state of
  50% or more of the servers listed in the performance schema table
  replication_group_members is UNREACHABLE.

* STATE CHANGE

  - This notifies that a server in the group has finished recovery.

    A server changes finishes recovery, when it moves into ONLINE state.

  Whenever this notification is triggered, then the state of a server,
  as seen in the performance schema table replication_group_members,
  has changed.

Each of these notifications map into a callback into the interfaces defined
for the services. Below are the service interfaces.

LISTENER SERVICES
-----------------

The interface devised for the listener is the following one
(note that the recommendation for creating services is to
 avoid using C++ classes as arguments. That's the reason why
 we do not use std::string below for the view_id arguments):

/**
  A service that listens for notifications about view changes or
  quorum loss.

  @note A GCS event may trigger more than one notification. For
  example, when installing a view, it may also trigger a primary
  election, thus two notifications shall be emitted.
*/
BEGIN_SERVICE_DEFINITION(group_membership_listener)
  /**
    This function SHALL be called whenever there is a new view
    installed.

    The implementation SHALL consume the notification and
    return false on success, true on failure.

    The implementation MUST NOT block the caller. It MUST
    handle the notification quickly or enqueue it and deal
    with it asynchronously.

    @param view_id The view identifier. This must be copied
                   if the string must outlive the notification
                   lifecycle.

    @return false success, true on failure.
  */
  DECLARE_BOOL_METHOD(notify_view_change, (const char* view_id));

  /**
    This function SHALL be called whenever the number of members
    that show state UNREACHABLE is equal or larger than the number
    of members that are not ONLINE or RECOVERING.

    The implementation SHALL consume the notification and
    return false on success, true on failure.

    The implementation MUST NOT block the caller. It MUST
    handle the notification quickly or enqueue it and deal
    with it asynchronously.

    @param view_id The view identifier. This must be copied
                   if the string must outlive the notification
                   lifecycle.

    @return false success, true on failure.
  */
  DECLARE_BOOL_METHOD(notify_quorum_loss, (const char* view_id));

END_SERVICE_DEFINITION(group_membership_listener)

/**
  A service that listens for notifications about member state
  or member role updates.

  @note A GCS event may trigger more than one notification. For
  example, when installing a view, it may also trigger a primary
  election, thus two notifications shall be emitted.
*/
BEGIN_SERVICE_DEFINITION(group_member_status_listener)
  /**
    This function SHALL be called whenever the role of a member
    changes.

    The implementation SHALL consume the notification and
    return false on success, true on failure.

    The implementation MUST NOT block the caller. It MUST
    handle the notification quickly or enqueue it and deal
    with it asynchronously.

    @param view_id The view identifier. This must be copied
                   if the string must outlive the notification
                   lifecycle.

    @return false success, true on failure.
  */
  DECLARE_BOOL_METHOD(notify_member_role_change, (const char* view_id));

  /**

    This function SHALL be called whenever the state of a member
    changes.

    The implementation SHALL consume the notification and
    return false on success, true on failure.

    The implementation MUST NOT block the caller. It MUST
    handle the notification quickly or enqueue it and deal
    with it asynchronously.

    @param view_id The view identifier. This must be copied
                   if the string must outlive the notification
                   lifecycle.

    @return false success, true on failure.
  */
  DECLARE_BOOL_METHOD(notify_member_state_change, (const char* view_id));
END_SERVICE_DEFINITION(group_member_status_listener)

UPGRADE / DOWNGRADE
===================

This worklog is about internal plumbing to get notifications from GR into
other components in the server. Therefore, since supported plugins are
released together, there is no chance that a plugin engages an incompatible

At the low level, the following changes will need to take place:

C1. Add the new service interfaces to components interfaces placeholder in the
    mysql-trunk tree. This way, the interfaces become public.

    The interfaces are placed in the following files.

    include/mysql/components/services/group_member_status_listener.h
    include/mysql/components/services/group_membership_listener.h

C2. Change GR's implementation of the GCS event handlers so that these will
    capture which GCS events should actually trigger a notification to the
    listeners registered in the service registry.

    The notifications one needs to forward are not plain vanilla events coming
    from GCS. Sometimes, they need to be triggered after GR has handled them
    (e.g., after a primary election, a recovery that finished, etc).

    To be able to track if an event from GCS actually resulted on a meaningful
    notification to be fired to the listeners (quorum loss, primary election,
    etc), we introduce a context event in the GR's implementation of the GCS
    handlers. This context, tracks what GCS events actually should result also
    in notifications to the listeners in the service registry.

    For example, an inner class of the Plugin_gcs_events_handler:

		  class Event_context
		  {
		  private:
		    bool m_member_role_changed;
		    bool m_member_state_changed;
		    bool m_quorum_lost;
		    bool m_view_changed;
		    const char* m_view_id;
		  public:
		    Event_context() :
		      m_member_role_changed(false),
		      m_member_state_changed(false),
		      m_quorum_lost(false),
		      m_view_changed(false),
		      m_view_id(NULL)
		    {}

		    void reset()
		    {
		      m_member_role_changed= false;
		      m_member_state_changed= false;
		      m_view_changed= false;
		      m_quorum_lost= false;
		      m_view_id= NULL;
		    }
		    void set_member_role_changed() { m_member_role_changed= true; }
		    void set_member_state_changed() { m_member_state_changed= true; }
		    void set_quorum_lost() { m_quorum_lost= true; }
		    void set_view_changed() { m_view_changed= true; }
		    void set_view_id(const char *v) { m_view_id= v; }

		    bool get_member_role_changed() { return m_member_role_changed; }
		    bool get_member_state_changed() { return m_member_state_changed; }
		    bool get_quorum_lost() { return m_quorum_lost; }
		    bool get_view_changed() { return m_view_changed; }
		    const char* get_view_id() { return m_view_id; }
		  };

    At the end of every GCS event handling, if the context flags that a
    notification to the listeners in the service registry is required, then
    such notification is forwarded.

    This context is reset every time the GCS handler function returns.

C3. Implement a notification forwarder function. It is responsible to acquire
    the service registry and registry query service handles. Then iterate over
    the listener implementations in the registry and call out to them.

    Something like:

      if (notify(kGroupMembership, r, rq, evts_ctx))
      {
        log_message(MY_ERROR_LEVEL,
                    "An undefined error was found while broadcasting an "
                    "internal group membership notification! "
                    "This is likely to happen if your components or plugins "
                    "are not properly loaded or are malfunctioning!");
      }

      /* notify member status events listeners. */
      if (notify(kGroupMemberStatus, r, rq, evts_ctx))
      {
        log_message(MY_ERROR_LEVEL,
                    "An undefined error was found while broadcasting an "
                    "internal group member status notification! "
                    "This is likely to happen if your components or plugins "
                    "are not properly loaded or are malfunctioning!");
      }

    For not acquiring the registry and registry query everytime, we will
    introduce a façade to the two registry services used every time
    (service registry and registry query service). The façade will also
    cache these references. Will acquire them at start up of group
    replication and will release them when it group replication is stopped.

    Thence there will be a Registry_module that will take ownership of the
    setup and teardown of these handles for group replication facilities.

    Depending on the event context (see C2) this notification forwarder is
    called after the gcs handlers implementation have processed the
    notification from the GCS layer.

C4. Change Group Replication CMake to also build any additional C/C++ files.