WL#9856: MySQL GCS: Instrument locking structures in GCS/XCom

Executive Summary
=================

This worklog will instrument mutex and condition synchronization objects
on GCS and will expose them automatically in P_S tables metrics, using
the Server's Performance Schema Interface to instrument the aforementioned
synchronization objects as well as declaring and registering the keys of
the instrumented mutex and condition objects.

Since XCom is currently single threaded, this worklog will not affect it.


User Stories
============

-As a MySQL DBA, I want to monitor the mutex and condition synchronization
objects of GCS, so that I am able to determine if there are any concurrency
related issues.

-As a MySQL dev, I want to have mutex and condition synchronization objects
instrumentation in GCS, so I can determine if there are any abnormal wait
events like bottlenecks or deadlocks.

------------------------
Functional requirements:
------------------------
FR1. GCS must be able to be built with or without instrumentation.
FR2. If instrumentation is enabled, each instrumented condition or mutex
must correspond to an entry in the setup_instruments table, stating its name
and if its instrumentation is timed and enabled.
FR3. If instrumentation is enabled while the node is in a group or trying to
join one, the cond_instances table must contain one entry per instrumented
GCS condition, stating its name and memory address.
FR4. If instrumentation is enabled while the node is in a group or trying
to join one, the mutex_instances table must contain 6 entries, current value
of the XCOM_MAX_HANDLERS macro in the gcs_xcom_utils.cc file, with the name
wait/synch/mutex/group_rpl/GCS_Xcom_handler::m_lock, which refer to
the 6 instances of the m_lock mutex, present in the Xcom_handler class.
FR5. If instrumentation is enabled while the node is in a group or trying
to join one, the mutex_instances table must contain one entry for each of
the remaining instrumented GCS mutexes, stating its name, memory address
and the id of the thread that locked it, or NULL if it is unlocked.
FR6. If instrumentation is enabled, the events_waits_current,
events_waits_history and events_waits_history_long tables must contain
entries regarding current and past mutex locks by threads.
FR7. If instrumentation is disabled globally, no instrumentation information
should be displayed.


----------------------------
Non-Functional requirements:
----------------------------
NFR1. The code modifications resulting from this WL must not degrade the
performance of GCS and XCom.
NFR2. Each instrumented code element must be univocally identified so that
we can monitor and understand the behavior of each individual element as a
separate entity.

Since the build process of GCS was already modified in WL#10622 to enable
instrumentation, it is only necessary to modify its source code in order to
leverage it for synchronization objects, such as mutexes and conditions.

The source code of GCS is then modified with the declaration and registration
of the instrumentation keys, and with the addition of the corresponding
instrumentation keys as parameters for the invoked synchronization objects
init functions. These are wrapped in the Xplatform, which is modified
by replacing the existing platform-specific code with invocations to the
corresponding instrumented Server's mysql_mutex_* and mysql_cond_* functions
for mutexes and conditions, respectively.


Interface changes
=================

This WL introduces minor interface changes to GCS, namely by modifying
the signatures of the init methods of My_xp_mutex and My_xp_cond to
include the additional instrumentation key parameter.

-For instance, in the Gcs_suspicions_manager constructor, defined in the
gcs_xcom_control_interface.cc file, there's the following invocation to the
init method of m_suspicions_mutex:

m_suspicions_mutex.init(NULL);

This will be replaced by the following code:

m_suspicions_mutex.init(
    key_GCS_MUTEX_Gcs_suspicions_manager_m_suspicions_mutex, NULL);


-We have the following invocation to initialize a cond in the Gcs_xcom_engine
constructor, implemented in the gcs_xcom_notification.cc file, we have the
following invocation to initialize a cond:

m_wait_for_notification_cond.init();

This invocation will be replaced by:

m_wait_for_notification_cond.init(
    key_GCS_COND_Gcs_xcom_engine_m_wait_for_notification_cond);

To enable the instrumentation of the synchronization objects used in GCS
(mutexes, conditions), we resort to the corresponding instrumented functions
provided by the Server's mysys library. Besides allowing us to remove the
platform-specific code contained in XPlatform, this also leverages the
Server's existing mechanism for enabling or disabling instrumentation,
not only in build time, but also in runtime.


========================================
1. Changes to the build process of GCS
========================================

In this WL, the building process of GCS does not need any modification,
since the required changes were performed in WL#10622.

The existing XCOM_STANDALONE macro will be used on GCS to distinguish if it is
to be used separately from the Server. When this macro is disabled, GCS will be
built using the Server's instrumented mutex and cond functions. The
implementation to use when XCOM_STANDALONE is enabled is out of the
scope of this WL.


======================================
2. Changes to the source code of GCS
======================================


2.1.Instrumentation keys
========================

Mutex and Cond instrumentation keys must be declared with the PSI_mutex_key
and PSI_cond_key types, respectively, in the gcs_psi.cc file. The keys
should also be declared as extern in the corresponding header, gcs_psi.h,
to allow their usage throughout the code.

Hence, the following mutex and cond instrumentation keys will be declared:

gcs_psi.h
---------
extern PSI_mutex_key key_GCS_MUTEX_Gcs_async_buffer_m_wait_for_events_mutex,
                     key_GCS_MUTEX_Gcs_async_buffer_m_free_buffer_mutex,
                     key_GCS_MUTEX_Gcs_suspicions_manager_m_suspicions_mutex,
                     key_GCS_MUTEX_Gcs_xcom_group_management_m_nodes_mutex,
                     key_GCS_MUTEX_Gcs_xcom_interface_m_wait_for_ssl_init_mutex,
                     
key_GCS_MUTEX_Gcs_xcom_engine_m_wait_for_notification_mutex,
                     
key_GCS_MUTEX_Gcs_xcom_view_change_control_m_wait_for_view_mutex,
                     
key_GCS_MUTEX_Gcs_xcom_view_change_control_m_current_view_mutex,
                     
key_GCS_MUTEX_Gcs_xcom_view_change_control_m_joining_leaving_mutex,
                     key_GCS_MUTEX_Gcs_xcom_proxy_impl_m_lock_xcom_cursor,
                     key_GCS_MUTEX_Gcs_xcom_proxy_impl_m_lock_xcom_ready,
                     key_GCS_MUTEX_Gcs_xcom_proxy_impl_m_lock_xcom_comms_status,
                     key_GCS_MUTEX_Gcs_xcom_proxy_impl_m_lock_xcom_exit,
                     key_GCS_MUTEX_Xcom_handler_m_lock;


extern PSI_cond_key key_GCS_COND_Gcs_async_buffer_m_wait_for_events_cond,
                    key_GCS_COND_Gcs_async_buffer_m_free_buffer_cond,
                    key_GCS_COND_Gcs_xcom_interface_m_wait_for_ssl_init_cond,
                    key_GCS_COND_Gcs_xcom_engine_m_wait_for_notification_cond,
                    
key_GCS_COND_Gcs_xcom_view_change_control_m_wait_for_view_cond,
                    key_GCS_COND_Gcs_xcom_proxy_impl_m_cond_xcom_ready,
                    key_GCS_COND_Gcs_xcom_proxy_impl_m_cond_xcom_comms_status,
                    key_GCS_COND_Gcs_xcom_proxy_impl_m_cond_xcom_exit;
---------



gcs_psi.cc
----------
PSI_mutex_key key_GCS_MUTEX_Gcs_async_buffer_m_wait_for_events_mutex,
              key_GCS_MUTEX_Gcs_async_buffer_m_free_buffer_mutex,
              key_GCS_MUTEX_Gcs_suspicions_manager_m_suspicions_mutex,
              key_GCS_MUTEX_Gcs_xcom_group_management_m_nodes_mutex,
              key_GCS_MUTEX_Gcs_xcom_interface_m_wait_for_ssl_init_mutex,
              key_GCS_MUTEX_Gcs_xcom_engine_m_wait_for_notification_mutex,
              key_GCS_MUTEX_Gcs_xcom_view_change_control_m_wait_for_view_mutex,
              key_GCS_MUTEX_Gcs_xcom_view_change_control_m_current_view_mutex,
              
key_GCS_MUTEX_Gcs_xcom_view_change_control_m_joining_leaving_mutex,
              key_GCS_MUTEX_Gcs_xcom_proxy_impl_m_lock_xcom_cursor,
              key_GCS_MUTEX_Gcs_xcom_proxy_impl_m_lock_xcom_ready,
              key_GCS_MUTEX_Gcs_xcom_proxy_impl_m_lock_xcom_comms_status,
              key_GCS_MUTEX_Gcs_xcom_proxy_impl_m_lock_xcom_exit,
              key_GCS_MUTEX_Xcom_handler_m_lock;


PSI_cond_key key_GCS_COND_Gcs_async_buffer_m_wait_for_events_cond,
             key_GCS_COND_Gcs_async_buffer_m_free_buffer_cond,
             key_GCS_COND_Gcs_xcom_interface_m_wait_for_ssl_init_cond,
             key_GCS_COND_Gcs_xcom_engine_m_wait_for_notification_cond,
             key_GCS_COND_Gcs_xcom_view_change_control_m_wait_for_view_cond,
             key_GCS_COND_Gcs_xcom_proxy_impl_m_cond_xcom_ready,
             key_GCS_COND_Gcs_xcom_proxy_impl_m_cond_xcom_comms_status,
             key_GCS_COND_Gcs_xcom_proxy_impl_m_cond_xcom_exit;
----------

Two static arrays, one with PSI_mutex_info elements and another with
PSI_cond_info elements, are declared, where each tuple contains the following
data regarding a mutex or a cond:
-the instrumentation key's memory address;
-the instrument name;
-PSI flags, which are defined in psi_base.h and could be, for instance,
PSI_FLAG_SINGLETON, to indicate that the element is a singleton;
-the volatility index of the instrument;
-the documentation for the instrument.


Concerning the registration of mutex instrumentation keys, the PSI_mutex_info
element has an additional field to be filled, which concerns the volatility
of the mutex.
Since there are no special requirements to the volatility of mutexes, we will
define it as 0, since it is the same value used in instrumented mutexes in
the plugin_psi.cc file of the Group Replication plugin.


gcs_psi.cc
----------
static PSI_mutex_info all_gcs_psi_mutex_keys_info[]=
{
  {&key_GCS_MUTEX_Gcs_async_buffer_m_free_buffer_mutex, 
"GCS_Gcs_async_buffer::m_free_buffer_mutex", PSI_FLAG_SINGLETON, 0, 
PSI_DOCUMENT_ME},
  {&key_GCS_MUTEX_Gcs_suspicions_manager_m_suspicions_mutex, 
"GCS_Gcs_suspicions_manager::m_suspicions_mutex", PSI_FLAG_SINGLETON, 0, 
PSI_DOCUMENT_ME},
  {&key_GCS_MUTEX_Gcs_xcom_group_management_m_nodes_mutex, 
"GCS_Gcs_xcom_group_management::m_nodes_mutex", PSI_FLAG_SINGLETON, 0, 
PSI_DOCUMENT_ME},
  {&key_GCS_MUTEX_Gcs_xcom_interface_m_wait_for_ssl_init_mutex, 
"GCS_Gcs_xcom_interface::m_wait_for_ssl_init_mutex", PSI_FLAG_SINGLETON, 0, 
PSI_DOCUMENT_ME},
  {&key_GCS_MUTEX_Gcs_xcom_engine_m_wait_for_notification_mutex, 
"GCS_Gcs_xcom_engine::m_wait_for_notification_mutex", PSI_FLAG_SINGLETON, 0, 
PSI_DOCUMENT_ME},
  {&key_GCS_MUTEX_Gcs_xcom_view_change_control_m_wait_for_view_mutex, 
"GCS_Gcs_xcom_view_change_control::m_wait_for_view_mutex", PSI_FLAG_SINGLETON, 
0, PSI_DOCUMENT_ME},
  {&key_GCS_MUTEX_Gcs_xcom_view_change_control_m_current_view_mutex, 
"GCS_Gcs_xcom_view_change_control::m_current_view_mutex", PSI_FLAG_SINGLETON, 0, 
PSI_DOCUMENT_ME},
  {&key_GCS_MUTEX_Gcs_xcom_view_change_control_m_joining_leaving_mutex, 
"GCS_Gcs_xcom_view_change_control::m_joining_leaving_mutex", PSI_FLAG_SINGLETON, 
0, PSI_DOCUMENT_ME},
  {&key_GCS_MUTEX_Gcs_xcom_proxy_impl_m_lock_xcom_cursor, 
"GCS_Gcs_xcom_proxy_impl::m_lock_xcom_cursor", PSI_FLAG_SINGLETON, 0, 
PSI_DOCUMENT_ME},
  {&key_GCS_MUTEX_Gcs_xcom_proxy_impl_m_lock_xcom_ready, 
"GCS_Gcs_xcom_proxy_impl::m_lock_xcom_ready", PSI_FLAG_SINGLETON, 0, 
PSI_DOCUMENT_ME},
  {&key_GCS_MUTEX_Gcs_xcom_proxy_impl_m_lock_xcom_comms_status, 
"GCS_Gcs_xcom_proxy_impl::m_lock_xcom_comms_status", PSI_FLAG_SINGLETON, 0, 
PSI_DOCUMENT_ME},
  {&key_GCS_MUTEX_Gcs_xcom_proxy_impl_m_lock_xcom_exit, 
"GCS_Gcs_xcom_proxy_impl::m_lock_xcom_exit", PSI_FLAG_SINGLETON, 0, 
PSI_DOCUMENT_ME},
  {&key_GCS_MUTEX_Xcom_handler_m_lock, "GCS_Xcom_handler::m_lock", 
PSI_FLAG_SINGLETON, 0, PSI_DOCUMENT_ME}
};


static PSI_cond_info all_gcs_psi_cond_keys_info[]=
{
  {&key_GCS_COND_Gcs_async_buffer_m_wait_for_events_cond, 
"GCS_Gcs_async_buffer::m_wait_for_events_cond", PSI_FLAG_SINGLETON, 0, 
PSI_DOCUMENT_ME},
  {&key_GCS_COND_Gcs_async_buffer_m_free_buffer_cond, 
"GCS_Gcs_async_buffer::m_free_buffer_cond", PSI_FLAG_SINGLETON, 0, 
PSI_DOCUMENT_ME},
  {&key_GCS_COND_Gcs_xcom_interface_m_wait_for_ssl_init_cond, 
"GCS_Gcs_xcom_interface::m_wait_for_ssl_init_cond", PSI_FLAG_SINGLETON, 0, 
PSI_DOCUMENT_ME},
  {&key_GCS_COND_Gcs_xcom_engine_m_wait_for_notification_cond, 
"GCS_Gcs_xcom_engine::m_wait_for_notification_cond", PSI_FLAG_SINGLETON, 0, 
PSI_DOCUMENT_ME},
  {&key_GCS_COND_Gcs_xcom_view_change_control_m_wait_for_view_cond, 
"GCS_Gcs_xcom_view_change_control::m_wait_for_view_cond", PSI_FLAG_SINGLETON, 0, 
PSI_DOCUMENT_ME},
  {&key_GCS_COND_Gcs_xcom_proxy_impl_m_cond_xcom_ready, 
"GCS_Gcs_xcom_proxy_impl::m_cond_xcom_ready", PSI_FLAG_SINGLETON, 0, 
PSI_DOCUMENT_ME},
  {&key_GCS_COND_Gcs_xcom_proxy_impl_m_cond_xcom_comms_status, 
"GCS_Gcs_xcom_proxy_impl::m_cond_xcom_comms_status", PSI_FLAG_SINGLETON, 0, 
PSI_DOCUMENT_ME},
  {&key_GCS_COND_Gcs_xcom_proxy_impl_m_cond_xcom_exit, 
"GCS_Gcs_xcom_proxy_impl::m_cond_xcom_exit", PSI_FLAG_SINGLETON, 0, 
PSI_DOCUMENT_ME}
};
----------


2.1.1 Key registration
======================

All the keys used for instrumenting mutexes and conditions must be registered
during the initialization of GCS, by invoking the mysql_mutex_register and
mysql_cond_register functions with a static array of PSI_mutex_info and
PSI_cond_info elements, respectively. Both invocations will be performed
inside the register_gcs_mutex_cond_psi_keys function, implemented in the
gcs_psi.cc file. This function is executed in the initialize method of the
Gcs_xcom_interface class, during the startup of GCS.
All the keys are registered under the "group_rpl" category, as it refers to
Group Replication, which is the Server's plugin that includes GCS.


gcs_psi.h
---------
/**
  Registers the psi keys for the mutexes and conds that will be instrumented.
*/

void register_gcs_mutex_cond_psi_keys();
---------


gcs_psi.cc
----------
void register_gcs_mutex_cond_psi_keys()
{
  const char *category = "group_rpl";
  int count= static_cast(array_elements(all_gcs_psi_cond_keys_info));

  mysql_cond_register(category, all_gcs_psi_cond_keys_info, count);

  count= static_cast(array_elements(all_gcs_psi_mutex_keys_info));

  mysql_mutex_register(category, all_gcs_psi_mutex_keys_info, count);
}
----------



2.2 XPlatform
=============
Some of the classes in the XPlatform of GCS have code that is inspired by
Server's mysys. This code will be replaced by invocations to the corresponding
functions on mysys, reducing our codebase by removing duplicate code from
the my_xp_cond.cc and my_xp_mutex.cc files.
It will be possible to remove the large majority of the code dealing with
multiple platforms, such as the following classes: My_xp_cond_pthread,
My_xp_cond_win, My_xp_mutex_pthread, My_xp_mutex_win.

The My_xp_cond_server and My_xp_mutex_server classes will then extend the
corresponding abstract class and implement the declared methods directly.
The init method of each of these code elements will have to be modified in
order to receive the instrumentation key as parameter.
Then, My_xp_cond_impl and My_xp_mutex_impl will extend the My_xp_cond_server
and My_xp_mutex_server classes.

These changes to the code can be seen in the following excerpts.


my_xp_cond.h:
-------------

#include 
"plugin/group_replication/libmysqlgcs/include/mysql/gcs/xplatform/my_xp_mutex.h"
#include 
"plugin/group_replication/libmysqlgcs/include/mysql/gcs/xplatform/my_xp_util.h"
#include "mysql/psi/mysql_cond.h"

/**
  @class My_xp_cond

  Abstract class used to wrap condition for various implementations.

  A typical use case of cond is:

  @code{.cpp}

  My_xp_cond *cond= new My_xp_cond_impl();
  cond->init(cond_PSI_key);

  cond->signal();

  @endcode
*/
class My_xp_cond
{
public:
  /**
    Initialize cond.

    @param key cond instrumentation key

    @return success status
  */

  virtual int init(PSI_cond_key key)= 0;


  /**
    Destroy cond.

    @return success status
  */

  virtual int destroy()= 0;


  /**
    Wait for cond to be signaled during some time before unlocking mutex.

    @param mutex mutex to unlock
    @param abstime time to wait
    @return success status
  */

  virtual int timed_wait(mysql_mutex_t *mutex,
                         const struct timespec *abstime)= 0;


  /**
    Wait for cond to be signaled to unlock mutex.

    @param mutex mutex to unlock
    @return success status
  */

  virtual int wait(mysql_mutex_t *mutex)= 0;


  /**
    Signal cond.

    @return success status
  */

  virtual int signal()= 0;


  /**
    Broadcast cond.

    @return success status
  */

  virtual int broadcast()= 0;


  /**
    Get reference to native cond.

    @return native cond
  */

  virtual mysql_cond_t *get_native_cond()= 0;

  virtual ~My_xp_cond() {}
};

#ifndef XCOM_STANDALONE
class My_xp_cond_server : public My_xp_cond
{
public:
  explicit My_xp_cond_server();
  virtual ~My_xp_cond_server();

  int init(PSI_cond_key key);
  int destroy();
  int timed_wait(mysql_mutex_t *mutex, const struct timespec *abstime);
  int wait(mysql_mutex_t *mutex);
  int signal();
  int broadcast();
  mysql_cond_t *get_native_cond();

protected:
  mysql_cond_t *m_cond;
};
#endif

#ifndef XCOM_STANDALONE
class My_xp_cond_impl : public My_xp_cond_server
#endif
{
public:
  explicit My_xp_cond_impl() {}
  ~My_xp_cond_impl() {}
};
-------------


my_xp_cond.cc:
--------------
#include 
"plugin/group_replication/libmysqlgcs/include/mysql/gcs/xplatform/my_xp_cond.h"

#ifndef XCOM_STANDALONE
My_xp_cond_server::My_xp_cond_server()
:m_cond(static_cast(malloc(sizeof(*m_cond))))
{}


My_xp_cond_server::~My_xp_cond_server()
{
  free(m_cond);
}


int My_xp_cond_server::init(PSI_cond_key key)
{
  return mysql_cond_init(key, m_cond);
}


int My_xp_cond_server::destroy()
{
  return mysql_cond_destroy(m_cond);
}


int My_xp_cond_server::timed_wait(mysql_mutex_t *mutex,
                                  const struct timespec *abstime)
{
  return mysql_cond_timedwait(m_cond, mutex, abstime);
}


int My_xp_cond_server::wait(mysql_mutex_t *mutex)
{
  return mysql_cond_wait(m_cond, mutex);
}


int My_xp_cond_server::signal()
{
  return mysql_cond_signal(m_cond);
}


int My_xp_cond_server::broadcast()
{
  return mysql_cond_broadcast(m_cond);
}


mysql_cond_t *My_xp_cond_server::get_native_cond()
{
  return m_cond;
}
#endif
--------------


my_xp_mutex.h:
--------------
#include "mysql/psi/mysql_mutex.h"

/**
  @class My_xp_mutex

  Abstract class used to wrap mutex for various implementations.

  A typical use case is:

  @code{.cpp}

  My_xp_mutex *mutex= new My_xp_mutex_impl();
  mutex->init(mutex_PSI_key, NULL);

  mutex->lock();
  ...
  mutex->unlock();

  @endcode
*/
class My_xp_mutex
{
public:
  /**
    Initialize mutex.

    @param key mutex instrumentation key
    @param attr mutex attributes reference
    @return success status
  */

  virtual int init(PSI_mutex_key key, const native_mutexattr_t *attr)= 0;


  /**
    Destroy mutex.

    @return success status
  */

  virtual int destroy()= 0;


  /**
    Lock mutex.

    @return success status
  */

  virtual int lock()= 0;


  /**
    Trylock mutex.

    @return success status
  */

  virtual int trylock()= 0;


  /**
    Unlock mutex.

    @return success status
  */

  virtual int unlock()= 0;


  /**
    To get native mutex reference.

    @return native mutex pointer
  */

  virtual mysql_mutex_t *get_native_mutex()= 0;


  virtual ~My_xp_mutex() {}
};

#ifndef XCOM_STANDALONE
class My_xp_mutex_server : public My_xp_mutex
{
public:
  explicit My_xp_mutex_server();
  virtual ~My_xp_mutex_server();

  int init(PSI_mutex_key key, const native_mutexattr_t *attr);
  int destroy();
  int lock();
  int trylock();
  int unlock();
  mysql_mutex_t *get_native_mutex();

protected:
  mysql_mutex_t *m_mutex;
};
#endif


#ifndef XCOM_STANDALONE
class My_xp_mutex_impl : public My_xp_mutex_server
#endif
{
public:
  explicit My_xp_mutex_impl() {}
  ~My_xp_mutex_impl() {}
};

class My_xp_mutex_util
{
public:
  /**
    Initialize mutex attributes object

    @param attr mutex attributes reference
    @return success status
  */

  static int attr_init(native_mutexattr_t *attr);


  /**
    Destroy mutex attributes object

    @param attr mutex attributes reference
    @return success status
  */

  static int attr_destroy(native_mutexattr_t *attr);
};
--------------


my_xp_mutex.cc:
---------------
#include 
"plugin/group_replication/libmysqlgcs/include/mysql/gcs/xplatform/my_xp_mutex.h"

#ifndef XCOM_STANDALONE
My_xp_mutex_server::My_xp_mutex_server()
  :m_mutex(static_cast(malloc(sizeof(*m_mutex))))
{}


My_xp_mutex_server::~My_xp_mutex_server()
{
  free(m_mutex);
}


mysql_mutex_t *My_xp_mutex_server::get_native_mutex()
{
  return m_mutex;
}


int My_xp_mutex_server::init(PSI_mutex_key key, const native_mutexattr_t *attr)
{
  if (m_mutex == NULL)
    return -1;

  return mysql_mutex_init(key, m_mutex, attr);
}


int My_xp_mutex_server::destroy()
{
  return mysql_mutex_destroy(m_mutex);
}


int My_xp_mutex_server::lock()
{
  return mysql_mutex_lock(m_mutex);
}


int My_xp_mutex_server::trylock()
{
  return mysql_mutex_trylock(m_mutex);
}


int My_xp_mutex_server::unlock()
{
  return mysql_mutex_unlock(m_mutex);
}
#endif


int My_xp_mutex_util::attr_init(native_mutexattr_t *attr)
{
  /*
    On Windows there is no initialization of mutex attributes.
    Therefore, we simply return 0.
  */
#ifdef _WIN32
  return 0;
#else
  return pthread_mutexattr_init(attr);
#endif
}


int My_xp_mutex_util::attr_destroy(native_mutexattr_t *attr)
{
  /*
    On Windows there is no destruction of mutex attributes.
    Therefore, we simply return 0.
  */
#ifdef _WIN32
  return 0;
#else
  return pthread_mutexattr_destroy(attr);
#endif
}
---------------


2.3 Usage example
=================

This WL introduces minor interface changes to GCS, namely by modifying
the signatures of the init methods of My_xp_mutex and My_xp_cond to
include the additional instrumentation key parameter.

-Here is an example of the usage of the modified My_xp_mutex init method, since
it is the only method that requires changes to enable mutex instrumentation.
Until now, to initialize m_suspicions_mutex, inside the Gcs_suspicions_manager
constructor, defined in the gcs_xcom_control_interface.cc file, we would use:

m_suspicions_mutex.init(NULL);

This will be replaced by the following code:

m_suspicions_mutex.init(
    key_GCS_MUTEX_Gcs_suspicions_manager_m_suspicions_mutex, NULL);


-To exemplify the new way of initializing a cond, we can see the current
way of initializing m_wait_for_notification_cond in the Gcs_xcom_engine
constructor, which is implemented in the gcs_xcom_notification.cc file:

m_wait_for_notification_cond.init();

This invocation will be modified to accommodate the cond instrumentation
key conveyed as parameter to the init method:

m_wait_for_notification_cond.init(
    key_GCS_COND_Gcs_xcom_engine_m_wait_for_notification_cond);


2.4 Tests
=========
The existing invocations to the init methods of My_xp_mutex and My_xp_cond,
in the unit and smoke tests, must be modified through the addition of the
instrumentation key parameter. Since instrumentation is not available in
these cases and it is also not required, the parameter will be defined as
PSI_NOT_INSTRUMENTED.

To verify the correct implementation of the formal requirements of this WL,
a new MTR test will be defined, where:
-the registration of the instrumentation keys for mutexes and conditions will
be verified in the tables setup_instruments, namely by the correspondence
of a new entry per key;
-the existence of 19 instances of mutexes in the mutex_instances tables,
which correspond to 6 instances of the m_lock mutex of the Xcom_handler class,
and one instance for each of the other 13 instrumented mutexes.
-the existence of 8 instances of conds in the cond_instances table which
correspond to the 8 distinct instrumented cond elements.



=========================
3. Instrumentation data
=========================

When code instrumentation is enabled, the instrumentation data is inserted
and, or updated in the appropriate performance schema tables in runtime.


3.1 Instances Tables
====================
The performance schema *_instances tables contain entries regarding the
currently alive instances of the instrumented code elements. In the scope of
this WL, we'll look more closely into the cond_instances and mutex_instances
tables.

-The cond_instances table lists all the instrumented conditions during the
server execution. Its columns, NAME and OBJECT_INSTANCE_BEGIN, contain the
instrument name associated with each instrumented condition and its memory
address, respectively.

-The mutex_instances table lists all the instrumented mutexes during the server
execution, and it has two columns that are identical in name and content to
those in the cond_instances table. It also has an additional column named
LOCKED_BY_THREAD_ID, which contains the THREAD_ID of the locking thread,
or NULL in case it is unlocked.


3.2 Wait Events Tables
======================

Since the instrumented mutex elements deal with waits, we'll look more
closely into the performance schema wait event tables:
-events_waits_current, which contains the current wait events;
-events_waits_history, containing the 10 most recent wait events per thread;
-events_waits_history_long, which contains the most recent 10,000 global
wait events, i.e. across all threads.


These three tables share the same structure, having the following columns:
-THREAD_ID: ID of the thread associated with the event.
-EVENT_ID: Number of the thread's current event when it starts. The THREAD_ID
and EVENT_ID values combined uniquely identify the entry.
-END_EVENT_ID: Set to NULL when the event starts and updated to the thread
current event number when the event ends.
-EVENT_NAME: The name of the instrument that produced the event, corresponding
to the NAME value from the setup_instruments table.
-SOURCE: Name of the source file that contains the instrumented code that
produced the event, followed by file's the line number.
-TIMER_START, TIMER_END, TIMER_WAIT: These columns provide timing information
for the event in picoseconds. If the value of the TIMED column of the event
in setup_instruments is NO, these columns will all be NULL. If the event
hasn't finished, TIMER_END is the current timer value and TIMER_WAIT is the
time elapsed so far (TIMER_END − TIMER_START).
-SPINS: The number of spin rounds for a mutex. If the value is NULL, the
code does not use spin rounds or spinning is not instrumented.
-OBJECT_SCHEMA, OBJECT_NAME, OBJECT_TYPE: All are NULL for synchronization
objects.
-OBJECT_INSTANCE_BEGIN: Memory address of the synchronization object.
-INDEX_NAME: The name of the index used, PRIMARY or NULL, in case no index
was used.
-NESTING_EVENT_ID: The EVENT_ID value of the event within which this event
is nested.
-NESTING_EVENT_TYPE: The nesting event type that can be TRANSACTION, STATEMENT,
STAGE, WAIT or NULL.
-OPERATION: The type of operation performed, which should contain lock
for mutexes.
-NUMBER_OF_BYTES: The number of bytes read or written by the operation. Not
relevant for mutexes.
-FLAGS: Reserved for future use.


The size of the history tables can be modified in
performance_schema_events_waits_history_size and
performance_schema_events_waits_history_long_size.
These three wait events tables can be disabled in the setup_consumers table,
by switching the value of the ENABLED column to NO.



3.3 Workflow
============

When a thread tries to lock a mutex, a new entry is added in the
events_waits_current table, showing that it is waiting on a mutex, in the
EVENT_NAME column, and identifying the mutex in the OBJECT_INSTANCE_BEGIN
column.

When the thread successfully locks the mutex, the TIMER_END and TIMER_WAIT
columns of that same entry are updated and the event is added to the
events_waits_history and events_waits_history_long tables. The mutex_instances
table is also modified, since mutex entry now shows it is owned by the
thread.

When the mutex is unlocked, the value of the THREAD_ID column of
this entry is NULL again, showing it has no owner.

When the mutex is destroyed, the corresponding row is removed from
mutex_instances. The same occurs with a cond. When it is destroyed, its
entry is removed from the cond_instances table.


To detect bottlenecks or deadlocks between threads that involve mutexes,
the following tables can be queried:
-events_waits_current, where we can see what mutex a thread is waiting for;
-mutex_instances, where we can find which thread currently owns a mutex.

For instance, to identify events that have not yet completed and have taken
longer than N picoseconds thus far, the following expression can be used
in queries:

SELECT * FROM performance_schema.events_waits_current WHERE END_EVENT_ID IS NULL 
AND TIMER_WAIT > N;


For more details on these tables see:
https://dev.mysql.com/doc/refman/8.0/en/performance-schema-instance-tables.html
https://dev.mysql.com/doc/refman/8.0/en/cond-instances-table.html
https://dev.mysql.com/doc/refman/8.0/en/mutex-instances-table.html
https://dev.mysql.com/doc/refman/8.0/en/performance-schema-wait-tables.html
https://dev.mysql.com/doc/refman/8.0/en/events-waits-current-table.html
https://dev.mysql.com/doc/refman/8.0/en/events-waits-history-table.html
https://dev.mysql.com/doc/refman/8.0/en/events-waits-history-long-table.html