WL#10200: Change GCS/XCOM to have dynamic debugging and tracing

EXECUTIVE SUMMARY
-----------------
The goal of this WL is to allow users to dynamically filter out debugging and 
tracing messages per sub-system (i.e. GCS, XCOM, etc) without a noticeable 
impact on performance in production. Currently, it is extremely hard to trace 
bugs in production because the error log does not provide detailed information 
on the execution and enabling debug and trace requires a different build and 
deployment.

Two different logging infrastructures will be created. One to report error,
warning and information messages and another one for debugging and tracing.
The error, warning and information messages will be sent to the sink that is
currently provided by the server. Debug and trace messages will be sent to a
different sink, which must be prepared to handle a possible high rate of
logging messages.

In addition to that, messages will carry information on the sub-systems in
GCS/XCOM (e.g. GCS Engine Thread) that generated them so that we can 
dynamically
filter out messages from sub-systems that we temporarily do not want to see
writing messages into the sink.

Users with super privileges will be giving the ability to dynamically
enable debugging and tracing in production at their convenience. The sink used
to store the debugging and tracing data will be a file and will have the
appropriate permissions to avoid any security issue. Note, however, that
implementing any mechanism to manage the file such as rotating, purging it,
etc, is out of scope. Users should do so manually at their discretion.


USER STORIES
------------
- As a MySQL DBA I want to gather detailed information on group replication
  modules in production environments in order to help the analysis of issues.


LIMITATIONS/ISSUES
------------------
Debugging and tracing will be sent to an asynchronous circular buffer that
will gather messages and will eventually send them to a sink. We don't have
the intention of creating the necessary mechanism to manage files, rotate
them, for example. 

Having said that, we are planning to send the debugging and tracing messages
to a dump file. An asynchronous circular buffer, which is a solution already 
implemented in GCS, will be responsible for gathering messages and writing to 
it.

FUNCTIONAL REQUIREMENTS
-----------------------
  F-1 It must be possible to dynamically filter out logging and
      debugging in GCS/XCOM.

  F-2 It must be possible to dynamically filter out debug messages per
      sub-system.

      Using the following global variable:
        SET GLOBAL group_replication_communication_debug_options= 
"GCS_DEBUG_ALL"

  F-3 Only users with appropriate permissions (i.e super privileges)
      can filter out debug messages.

  F-4 It must be possible to send the error, warning and information
      messages to a sink, debug and trace messages to a different sink.

  F-5 Error, warning and information messages will be sent to a sink
      specified by group replication (i.e. server)

  F-6 Debug and trace messages will sent to a file when group replication
      is in use.

  F-7 By default the file used as debug sink will be named "GCS_DEBUG_TRACE"
      and will be placed in the data directory.

  F-8 The debug sink file will have the appropriate permissions following
      the server standard, i.e. -64--.

  F-9 The debug sink file cannot be set to overwrite any present or future 
      file in the MySQL directory.

  F-10 By default, if GCS is in use without group replication all messages
       will be sent to the same sink which is the standard output.

  F-11 The debug sink file will only exist if GCS/XCOM is compiled along with
       the server.

  F-12 All debug messages are filtered out by default to avoid any performance 
       impact.

NON-FUNCTIONAL REQUIREMENTS
---------------------------
  NF-1 The logging infrastructure must support structured messages
       where the sub-system that it came from can be easily identified.

  NF-2 There should not be any interleaved and unintelligible output.

  NF-3 The filtering mechanism must not cause a noticeable impact on
       performance.

       If all debug messages are filtered out, the throughput should be the 
       same as it was before this WL. However, if nothing is filtered out, 
       there will be a performance penalty which should not exceed 10%-15%.

  NF-4 There must be an asynchronous sink which is optimized to handle
       a high rate of logging messages.

  NF-5 Interfaces should be type safe so that we can catch possible
       misuses in compile time.

  NF-6 No sensitive information such as passwords, user names, etc
       will be written to the debug sink.

  NF-7 The debug file should not be copied by MEB because it does not contain
       any relevant information to restore the server state.

LOGGING AND DEBUGGING (Developer's Perspective)
-----------------------------------------------
Developers who want to print out information, warning or an error message
must use the following macros:

  . Information message:

    MYSQL_GCS_LOG_INFO("Message" << append_stuff);

  . Warning message:

    MYSQL_GCS_LOG_WARN("Message " << append_stuff);

  . Error message:

    MYSQL_GCS_LOG_ERROR("Message " << append_stuff);

  . Fatal message:

    MYSQL_GCS_LOG_FATAL("Message " << append_stuff);

Developers who want to print out debug and trace messages must use the
following macros:

  MYSQL_GCS_LOG_DEBUG("Message %s", append_stuff);

  MYSQL_GCS_LOG_TRACE("Message %s", append_stuff);

Note that debug and trace messages use a c-style format while the other use a 
stream solution only available in c++. We have decided to use a c-style 
solution because XCOM is written in C and having a common style will make the 
solution faster as there is no need for unecessary copies when converting from 
one type to another. Note also that fatal is mapped to error because the server 
does not have this level but XCOM needs it.

It is worth saying that debug and trace macros are internally mapped to other 
macros as follows:

#define MYSQL_GCS_LOG_DEBUG(x) \
  MYSQL_GCS_LOG_DEBUG_WITH_OPTION(GCS_DEBUG_BASIC | GCS_DEBUG_TRACE, x)

#define MYSQL_GCS_LOG_TRACE(x) \
  MYSQL_GCS_LOG_DEBUG_WITH_OPTION(GCS_DEBUG_TRACE, x)

The GCS_DEBUG_BASIC and GCS_DEBUG_TRACE are the current available sub-systems
that can be easily used to redefine the old macros MYSQL_GCS_LOG_TRACE(x) and
MYSQL_GCS_LOG_TRACE(x) without changing all its occurrences in the code.

In the future, we will start using MYSQL_GCS_LOG_DEBUG_WITH_OPTION(SUB-SYSTEM,
x) directly after creating a fine-grained set of sub-systems. The idea is that
a sub-system represents a logical part of the code such as a module or a class
that can have its associated debug messages dynamically filtered out.

If there was a sub-system GCS_DEBUG_THREAD, one could enable the debug messages
generated by it as follows:

  SET GLOBAL group_replication_communication_debug_options = "GCS_DEBUG_THREAD"


LOGGING AND DEBUGGING (User's Perspective)
------------------------------------------
Only users with associated super privileges can execute the following command:

  SET GLOBAL group_replication_communication_debug_options = "GCS_DEBUG_BASIC, 
GCS_DEBUG_TRACE"

This will make GCS/XCOM start logging debug information produced by the sub-
system GCS_DEBUG_BASIC and/or GCS_DEBUG_TRACE. Note that new filters are 
defined whenever the command is executed. This means that if we execute the 
next command, both GCS_DEBUG_BASIC and GCS_DEBUG_TRACE from the previous 
example will be automatically ignored as new filters are defined:

  SET GLOBAL group_replication_communication_debug_options = "GCS_DEBUG_THREAD, 
GCS_DEBUG_MSGS"

In order to filter out all modules, we must set the option to an empty string:

  SET GLOBAL group_replication_communication_debug_options = ""

  or

  SET GLOBAL group_replication_communication_debug_options = "GCS_DEBUG_NONE" 

Debug options will be almost always immediately validated except when they are 
defined during server initialization. This is a limitation of the plugin life-
cycle which does not call the validation functions in this case. However, it 
will be immediately checked during group replication initialization and any 
invalid option will prevent it from starting. 

The set of available options, which are strings, can be dynamically set by any 
user with super privileges as any group replication option. Currently, the 
following options are available: 

  . "GCS_DEBUG_NONE"   --- Disables all (both GCS and XCOM)
  . "GCS_DEBUG_BASIC"  --- Enables basic debugging information in GCS
  . "GCS_DEBUG_TRACE"  --- Enables trace information in GCS
  . "XCOM_DEBUG_BASIC" --- Enables basic debugging information in XCOM
  . "XCOM_DEBUG_TRACE" --- Enables trace information in XCOM
  . "GCS_DEBUG_ALL"    --- Enables all (both GCS and XCOM)

Note that
                        
  . GCS_DEBUG_NONE has effect only when provided without any other option.
  . GCS_DEBUG_ALL subsumes all other option.

Debug and trace will be stored in GCS_DEBUG_TRACE file which will be located
in the MySQL's data directory and will have the following permissions: same
owner and group that files created by MySQL have and -rw-r-----.


CONFIGURING THE SYSTEM (Developer's Perspective)
------------------------------------------------
-- Defining current logger --

The MYSQL_GCS_LOG_INFO, MYSQL_GCS_LOG_WARN and MYSQL_GCS_LOG_ERROR  macros
previously defined will call the log event method of an instance of the
Logger_interface which we call logger object from now on. Group replication
has to create an instance of the Logger_interface and inject the object into
GCS/XCOM through the following method:

  Gcs_xcom_interface::set_logger(logger);

If this is not done before calling Gcs_xcom_interface::initialize, GCS/XCOM
will create their own instance of the logger, i.e. Gcs_default_logger. Note 
that there will be no change in behavior here.

-- Defining logger's severity --

For the error, warning and information messages the current severity in use 
will depend on the server configuration. However, the current plugin log
service does not have any interface to fetch the current severity in use,
for that reason, GCS/XCOM will simply generate the information and the upper
layers shall be responsible for deciding whether it should be filtered out
or not. Due to the low rate of such messages, we believe this will not
become a bottleneck.

When the server provides better interfaces, we can improve the current
solution.

-- Defining current debugger --

The MYSQL_GCS_LOG_DEBUG_WITH_OPTION macro previously defined will call the log
event method of the Gcs_default_debugger object which we call debugger
object for now on. This object will be automatically created by GCS/XCOM.

-- Defining sinks for debugger and logger --

The logger and debugger object will have an associated sink object which 
implements the Sink_interface and it represents the message destination. It can 
be the terminal, a file, another process, a remote machine, etc. This WL will 
provide two different sinks:

  . Gcs_sink_output - Prints out messages to the standard output.

  . Gcs_file_sink -  Persists messages to a file.

The sink used by either the Gcs_default_logger or Gcs_default_debugger will be 
wrapped in the Gcs_async_buffer object which gathers a set of messages and 
asynchronously flush them to the associated sink. Note, however, that the 
Gcs_file_sink is only available when GCS/XCOM is compiled with the server and 
group replication is in use. If GCS/XCOM is not compiled with the server or is 
running without group replication, the Gcs_output_sink will be used by default 
and the same instance will be shared between both the logger and debugger.

The created file object has its location defined by group replication through 
the following configuration paramaters:

  . "communication_debug_file" and "communication_debug_path".

GCS SHARED DATA STRUCTURES
--------------------------
. The code in "gcs/gcs_log_system.h" will be slightly changed to accomodate a 
  circublar buffer that uses both mutexes and atomic variables. We don't put 
  the classes methods and attributes here to avoid cluttering the specification 
  and only provide information on the changes made:

  . Gcs_log_events_recipient_interface and associated objects- Removed them
    because they were playing the role of a final sink and replaced its
    occurrences by Sink_interface or an associated object when necessary.

  . Renamed Gcs_ext_logger_impl to Gcs_async_buffer.

. We also take the opportunity to rename some classes:

  . Ext_logger_interface       --> Logger_interface
  . Gcs_logger                 --> Gcs_log_manager
  . Gcs_ext_logger_impl        --> Gcs_async_buffer
  . Gcs_simple_ext_logger_impl --> Gcs_default_logger

. Common_interface class is inherited by both the logger and sink interfaces
  and provides common methods to initialize and finalize an object of this
  class.

  class Common_interface
  {
    virtual ~Common_interface() {}

    virtual enum_gcs_error initialize()= 0;

    virtual enum_gcs_error finalize()= 0;
  };

  It has the following methods:

  . enum_gcs_error initialize()

    The purpose of this method is to initialize any resources used in the.
    It is called, for example, by the Gcs_log_manager::initialize methods.

    It returns GCS_OK in case everything goes well. Any other value of
    gcs_error in case of error.

  . enum_gcs_error finalize();

    The purpose of this method is to free any resources used in the system. It 
    is invoked, for example, by the Gcs_log_manager::finalize during the GCS 
    termination procedure.

    It returns GCS_OK in case everything goes well. Any other value of
    gcs_error in case of error.

. Sink that can be used by both the logger and debugger and is ultimately
  responsible for writing the information into a file, standard output or
  something else. Currently, only a file and the standard output are supported.

  class Sink_interface : public Common_interface
  {
  public:
    virtual ~Sink_interface() {}

    virtual void log_event(const std::string &message)= 0;
  };

  It has the following methods:

    . void log_event(const std::string &message);

      The purpose of this method is to deliver to effective log the information
      and should be invoked by the logger or debugger object.

There will be two sinks implemented: standard output and file. Note that the 
file-based sink requires "my_sys" support and for that reason will be only 
available when compiled with the server.

  error Gcs_output_sink::initialize()
  {
    int ret_out= setvbuf(stdout, NULL, _IOLBF, BUFSIZ);

    if(ret_out == 0)
      return GCS_OK;
    else
    {
      #if defined(WIN_32) || defined(WIN_64)
      int errno= WSAGetLastError();
      #endif
      std::cerr << "Unable to invoke setvbuf correctly! " << strerror(errno)
        << std::endl;

      return GCS_NOK;
    }
  }

  enum_gcs_error Gcs_output_sink::finalize()
  {
    return GCS_OK;
  }


  void Gcs_output_sink::log_event(const std::string &message)
  {
    std::cout << message.str();
  }


  #ifdef MYSQL_SERVER
  Gcs_file_sink::Gcs_file_sink(const std::string &file_name, const std::string 
&data_home) :
    m_fd(0), m_file_name(file_name), m_data_home(data_home)
  {
  }

  enum_gcs_error Gcs_file_sink::initialize()
  {
    char file_name_buffer[FN_REFLEN];

    fn_format(file_name_buffer, m_file_name.c_str(), m_data_home.c_str(), "", 
              MYF(0));

    if ((m_fd= my_create(file_name_buffer, 0640, O_CREAT | O_WRONLY | O_APPEND, 
         MYF(0))) < 0)
      return GCS_NOK;

    return GCS_OK;
  }

  enum_gcs_error Gcs_file_sink::finalize()
  {
    my_sync(m_fd, MYF(0));
    my_close(m_fd, MYF(0));
    return GCS_OK;
  }

  void Gcs_file_sink::log_event(const std::string &message)
  {
    size_t written;
    written= my_write(m_fd, (const uchar *) message.c_str(), message.length(), 
                      MYF(0));

    if (written == MY_FILE_ERROR)
    {
  #if defined(WIN_32) || defined(WIN_64)
      int errno= WSAGetLastError();
  #endif
      MYSQL_GCS_LOG_ERROR(
        "Error writting to debug file: " << strerror(errno) << "."
      );
    }
  }
  #endif


GCS DATA STRUCTURES FOR LOGGING
-------------------------------
. Group replication must implement the class Logger_interface and wrap
  the logging features provided by the server. Further down, we can see
  how it can inject an object of this class into GCS/XCOM.

  class Logger_interface : public Common_interface
  {
  public:
    virtual ~Logger_interface() {}
    virtual void log_event(const gcs_log_level_t logging_level, const 
                           std::string &message)= 0;
  };

  It has the following methods:

  . void log_event(const gcs_log_level_t logging_level, const const std::string 
                   &message)

    The purpose of this method is to deliver to the logging system any event
    to be logged.

    It shouldn't be invoked directly in the code, as it is wrapped by some
    macros which deal with the rendering of the logging message into a final
    string that is then handed alongside with the level to this method.

    Usually any object that inherits from this interface will have an 
associated
    sink which will be responsible for writing the message into the log.

. The Gcs_log_manager class is storing the logging system to be used by the
  application as a singleton.

  class Gcs_log_manager
  {
  private:
    static Logger_interface *logger;

  public:
    static enum_gcs_error initialize(Logger_interface *logger);

    static Logger_interface *get_logger();

    static enum_gcs_error finalize();
  };

  It has the following member variables:

  . Logger_interface *logger

    Reference to a log interface object.

  It has the following methods:

  . enum_gcs_error initialize(Logger_interface *logger)

    The purpose of this static method is to set the received logging system on
    the log singleton, and to initialize it, by invoking its implementation of
    the Logger_interface::initialize method.

    This allows any resources needed by the logging system to be initialized,
    and ensures its usage throughout the lifecycle of the current GCS
    application.

    It returns GCS_OK in case everything goes well. Any other value of
    gcs_error in case of error.

  . Logger_interface *get_logger()

    This static method retrieves the currently set logging system, allowing the
    logging macros to invoke its log_event method.

  . enum_gcs_error finalize()

    The purpose of this static method is to free any resources used in the
    logging system.

    It is invoked by the Gcs_logger::finalize method during the GCS interface
    termination procedure, and also by the Gcs_logger::initialize method in
    case a logging system was set previously.

    It returns GCS_OK in case everything goes well. Any other value of
    gcs_error in case of error.

. The Gcs_default_logger class implements the Logger_interface class which
  may be used if group replication does not inject its own implementation
  as described further down.

  class Gcs_default_logger : public Logger_interface
  {
  public:
    Gcs_default_logger(Gcs_async_buffer *sink);

    ~Gcs_default_logger() {}

    enum_gcs_error initialize();

    enum_gcs_error finalize();
  };

  Gcs_default_logger::Gcs_default_logger(Gcs_async_buffer *sink) : m_sink(sink)
  {
  }

  enum_gcs_error Gcs_default_logger::initialize()
  {
    return GCS_OK;
  }

  enum_gcs_error Gcs_default_logger::finalize()
  {
    return GCS_OK;
  }

  void Gcs_default_logger::log_event(const gcs_log_level_t level, const 
std::string &message)
  {
    std::stringstream log;

    log << gcs_log_levels[level] << message << std::endl;

    m_sink->log_event(log.str());
  }


GCS DATA STRUCTURES FOR DEBUGGING
---------------------------------
. Unlike the logging infra-structure that requires group replication to define
  an object that inherits from the Logger_interface and inject it into 
GCS/XCOM,
  the debugging infra-structure will be automatically handled within GCS/XCOM
  requiring only information of a base directory where the debug messages will
  be dumped into a file. Internally, XCOM/GCS will have:

  class Gcs_default_debugger
  {
  public:
    Gcs_default_debugger(Gcs_async_buffer* sink);
    virtual ~Gcs_default_debugger() {}

    enum_gcs_error initialize();
    enum_gcs_error finalize();
    void log_event(const std::string &message);
    ...
  }

  Gcs_default_debugger::Gcs_default_debugger(Sink_interface *sink)
  : m_sink(sink)
  {
  }

  enum_gcs_error Gcs_default_debugger::initialize()
  {
    return GCS_OK;
  }

  enum_gcs_error Gcs_default_debugger::finalize()
  {
    return GCS_OK;
  }

  void Gcs_default_debugger::log_event(const std::string &message)
  {
     MYSQL_GCS_LOG_DEBUG(message.c_str());
  }

  void log_event(int64_t options, const char *message)
  {
     log_event(options, "%s", message);
  }

  void log_event(const char* format, va_list args)
  {
     ...
  }

  template 
  Gcs_default_debugger::void log_event(const int64_t options, Args ... args)
  {
     ...
  }


. The Gcs_debug_manager class is storing the debugging system to be used by the
  application as a singleton and provides the necessary methods to access
  and set the debugging options.

  In GCS, race conditions involving the variable used to stored the debugging
  options will be avoided through the use of atomic primitives. If XCOM is used 
  alone, there will be no protection against concurrent updates but we don't 
  expect that to be a problem since we are planning to use XCOM along with GCS.

  typedef enum
  {
    GCS_DEBUG_NONE       = 0x00000000,
    GCS_DEBUG_BASIC      = 0x00000001,
    GCS_DEBUG_TRACE      = 0x00000002,
    XCOM_DEBUG_BASIC     = 0x00000004,
    XCOM_DEBUG_TRACE     = 0x00000008,
    GCS_DEBUG_ALL        = ~GCS_DEBUG_NONE
  } gcs_debug_option_t;

  static const char* const gcs_debug_options[]=
  {
    "GCS_DEBUG_BASIC",
    "GCS_DEBUG_TRACE",
    "XCOM_DEBUG_TRACE",
    "XCOM_DEBUG_TRACE",
    "GCS_DEBUG_ALL",
    "GCS_DEBUG_NONE",
  };

  class Gcs_debug_manager
  {
  private:
    static Gcs_default_debugger *m_debugger;

    static std::atomic m_debug_options;

    static const char * const m_debug_none;

    static const char * const m_debug_all;

  public:
    static bool test_debug_options(const int64_t debug_options);

    static int64_t get_current_debug_options();

    static int64_t get_current_debug_options(std::string **res_debug_options);

    static int64_t get_valid_debug_options();

    static bool is_valid_debug_options(const int64_t debug_options);

    static bool is_valid_debug_options(const std::string &debug_options);

    static bool get_debug_options(const int64_t debug_options,
                                  std::string **res_debug_options);

    static bool get_debug_options(const std::string &debug_options,
                                  int64_t *res_debug_options);

    static unsigned int get_number_debug_options();

    static bool set_debug_options(const int64_t debug_options);

    static bool force_debug_options(const int64_t debug_options);

    static bool set_debug_options(const std::string &debug_options);

    static bool force_debug_options(const std::string &debug_options);

    static bool unset_debug_options(const int64_t debug_options);

    static bool unset_debug_options(const std::string &debug_options);

    static enum_gcs_error initialize(Gcs_default_debugger *debugger);

    static Gcs_default_debugger *get_debugger();

    static enum_gcs_error finalize();
  };


  It has the following methods:

    . static bool test_debug_options(const int64_t debug_options);

      Verify whether any of the debug options are defined.

    . static int64_t get_current_debug_options();

      Get the current set of debug options.

    . static int64_t get_current_debug_options(std::string 
**res_debug_options);

      Get the current set of debug options as a string separated by comma.

      Although, a boolean value is returned in this case it will always
      return false. Note that the parameter res_debug_options cannot be NULL.

    . static int64_t get_valid_debug_options();

      Get the set of valid debug options excluding GCS_DEBUG_NONE and
      GCS_DEBUG_ALL.

    . static bool is_valid_debug_options(const int64_t debug_options);

      Check whether the set of debug options is valid or not including
      GCS_DEBUG_NONE and GCS_DEBUG_ALL.

    . static bool is_valid_debug_options(const std::string &debug_options);

      Check whether the set of debug options is valid or not including
      GCS_DEBUG_NONE and GCS_DEBUG_ALL.

    . static bool get_debug_options(const int64_t debug_options,
                                  std::string **res_debug_options);

      Get the set of debug options passed as parameter as a string separated by
      comma.

      If there is any invalid debug option in the debug_options parameter, true
      is returned. Note that the parameter res_debug_options cannot be NULL.

    . static bool get_debug_options(const std::string &debug_options,
                                  int64_t *res_debug_options);
      Get the set of debug options passed as parameter as an unsigned integer.

      If there is any invalid debug option in the debug_options parameter, true
      is returned. Note that the parameter res_debug_options cannot be NULL.

    . static unsigned int get_number_debug_options();

      Get the the number of possible debug options.

    . static bool set_debug_options(const int64_t debug_options);

      Extend the current set of debug options with new debug options expressed 
      as an unsigned integer parameter.

      If there is any invalid debug option in the debug_options parameter, true
      is returned.

    . static bool force_debug_options(const int64_t debug_options);

      Change the current set of debug options by the new debug options
      expressed as an unsigned integer parameter.

      If there is any invalid debug option in the debug_options parameter, true
      is returned.

    . static bool set_debug_options(const std::string &debug_options);

      Extend the current set of debug options with new debug options expressed
      as a string.

      If there is any invalid debug option in the debug_options parameter, true
      is returned.

    . static bool force_debug_options(const std::string &debug_options);

      Changed the current set of debug options by the new debug options 
      expressed as a string.

      If there is any invalid debug option in the debug_options parameter, true
      is returned.

    . static bool unset_debug_options(const int64_t debug_options);

      Reduce the current set of debug options by disabling the debug options
      expressed as an unsigned integer parameter.

      If there is any invalid debug option in the debug_options parameter, true
      is returned.

    . static bool unset_debug_options(const std::string &debug_options);

      Reduce the current set of debug options by disabling the debug options
      expressed as a string.

      If there is any invalid debug option in the debug_options parameter, true
      is returned.

    . static enum_gcs_error initialize(Gcs_default_interface *debugger);

      The purpose of this static method is to set the received debugging system 
      and to initialize it. This allows any resources needed by the debugging 
      system to be initialized, and ensures its usage throughout the lifecycle 
      of the current GCS application.

    . static Gcs_default_debugger *get_debugger();

      This static method retrieves the current set debugging system, allowing 
      the debugging macros to invoke its log_event method.

    . static enum_gcs_error finalize();

      The purpose of this static method is to free any resources used in the
      debugging system.

      It is invoked by the Gcs_debug_manager::finalize method during the GCS 
      interface termination procedure, and also by the 
      Gcs_debug_manager::initialize method in case a debugging system was set 
      previously.


GCS MACROS FOR LOGGING
----------------------
#define MYSQL_GCS_LOG(l,x)                      \
  do \
  { \
      std::ostringstream temp; \
      temp << x; \
      Gcs_logger::get_logger()->log_event(l, temp.str()); \
  } \
  while (0); \

#define MYSQL_GCS_LOG_INFO(x) \
    MYSQL_GCS_LOG(GCS_LOG_INFO, x)

#define MYSQL_GCS_LOG_WARN(x) \
    MYSQL_GCS_LOG(GCS_LOG_WARN, x)

#define MYSQL_GCS_LOG_ERROR(x) \
    MYSQL_GCS_LOG(GCS_LOG_ERROR, x)


GCS MACROS FOR DEBUGGING
------------------------
define GCS_PREFIX "[GCS] "
#define GCS_DEBUG_PREFIX "[MYSQL_GCS_DEBUG] "

#define MYSQL_GCS_DEBUG_EXECUTE(x) \
  MYSQL_GCS_DEBUG_EXECUTE_WITH_OPTION(GCS_DEBUG_BASIC | GCS_DEBUG_TRACE, x)

#define MYSQL_GCS_TRACE_EXECUTE(x) \
  MYSQL_GCS_DEBUG_EXECUTE_WITH_OPTION(GCS_DEBUG_TRACE, x)

#define MYSQL_GCS_DEBUG_EXECUTE_WITH_OPTION(option, x) \
  do \
  { \
    if (Gcs_debug_manager::test_debug_options(option)) \
    { \
      x; \
    } \
  } \
  while (0);

#define MYSQL_GCS_LOG_DEBUG(x) \
  MYSQL_GCS_LOG_DEBUG_WITH_OPTION(GCS_DEBUG_BASIC | GCS_DEBUG_TRACE, x)

#define MYSQL_GCS_LOG_TRACE(x) \
  MYSQL_GCS_LOG_DEBUG_WITH_OPTION(GCS_DEBUG_TRACE, x)

#define MYSQL_GCS_LOG_DEBUG_WITH_OPTION(options, ...) \
  do \
  { \
    Gcs_default_debugger *debugger= Gcs_debug_manager::get_debugger(); \
    debugger->log_event(options, __VA_ARGS__); \
  } \
  while (0);



PLUGIN OPTION
-------------
Group replication will have to create the following option so that it can
enable debug at any time.

Note that the option may be deprecated when we integrate the current solution
with the server's WL(s). Besides, it will not be possible to list the debug
options available and the current ones defined. The main motivation is to
minimize the number of changes to the server and avoid deprecating additional
options in the future.

  static MYSQL_SYSVAR_STR(
    communication_debug_options,                /* name */
    communication_debug_options_var,            /* var */
    PLUGIN_VAR_OPCMDARG | PLUGIN_VAR_MEMALLOC,  /* optional var | malloc string 
*/
    "The set of debug options, comma separated. E.g., DEBUG_BASIC, DEBUG_ALL.",
    check_communication_debug_options,          /* check func */
    NULL,                                       /* update func */
    ""                                          /* default */
  );

Only users with super privileges will be able to change the debug option. The
check function will be used to validate the debug options if group replication
is running. Otherwise, any value is accepted and it will be checked when group 
group replication is started.


XCOM DATA STRUCTURES/FUNCTIONS
------------------------------
-- Logging Infrastructure --

  typedef void (*xcom_logger)(const int64_t level, const char *message);
  extern xcom_logger xcom_log;
  void set_xcom_logger(xcom_logger callback);
  void xcom_output_log(const int64_t level, const char *message);

  . Pointer to the logging callback:

   typedef void (*xcom_logger)(const int64_t level, const char *message);

  . Callback used by the logging macros:

    extern xcom_logger xcom_log;

  . Function used by external entities to set a non-default logger
    callback:

    void set_xcom_logger(xcom_logger callback);

  . If GCS is not compiled with XCOM, this function will be injected as a
    pointer into xcom_log and will print out logging messages to the
    console:

    void xcom_output_log(const int64_t level, const char *message);

-- Debugging Infrastructure --

  typedef enum
  {
    GCS_DEBUG_NONE       = 0x00000000,
    GCS_DEBUG_BASIC      = 0x00000001,
    GCS_DEBUG_TRACE      = 0x00000002,
    XCOM_DEBUG_BASIC     = 0x00000004,
    XCOM_DEBUG_TRACE     = 0x00000008,
    GCS_DEBUG_ALL        = ~GCS_DEBUG_NONE
  } gcs_debug_option_t;

  typedef void (*xcom_debugger)(const char *format, ...);
  extern xcom_debugger xcom_debug;
  void set_xcom_debugger(xcom_debugger callback);
  void xcom_default_debug(const char *format, ...);

  . Pointer to the debugging callback:

    typedef void (*xcom_debugger)(const char *format, ...);

  . Callback used by the debugging macros:

    extern xcom_debugger xcom_debug;

  . Function used by external entities to set a non-default debugger
    callback:

    void set_xcom_debugger(xcom_debugger callback);

  . If GCS is not compiled with XCOM, this function will be injected as a
    pointer into xcom_debug and will print out debugging messages to the
    console:

    void xcom_default_debug(const char *format, ...);
    

  extern uint64_t xcom_debug_options;
  typedef int (*xcom_debugger_check)(const int64_t debug_options);
  extern xcom_debugger_check xcom_debug_check;
  void set_xcom_debugger_check(xcom_debugger_check callback);
  int xcom_default_debug_check(const int64_t options);

  . Define the current debugging options enabled. It is used only when
    GCS is not compiled with XCOM:

    extern uint64_t xcom_debug_options;

  . Pointer to the callback function that check whether a debug option is 
    enabled or not:

    typedef int (*xcom_debugger_check)(const int64_t debug_options);

  . Function used by external entities to set a non-default debugging
    option callback:

    void set_xcom_debugger_check(xcom_debugger_check callback);

  . If GCS is not compiled with XCOM, this function will be injected as a
    pointer into xcom_test_debug to check if the xcom_debug_option is set:

    int xcom_default_debug_check(const int64_t options);

By default, the xcom_debug_option will be set to XCOM_DEBUG_NONE. Besides, the
enumeration xcom_debug_option_t is a copy or a sub-set of the options defined
in the GCS modules.

Note that we don't provide user interfaces so XCOM can dynamically change the
current debugging option when compiled without GCS. This is out of the scope
of this project.


XCOM MACROS FOR LOGGING
-----------------------
#define PRINT_LOUT(level) xcom_log(level, xcom_log_buffer)

#define G_LOG_LEVEL(level,...) { GET_GOUT; ADD_F_GOUT(__VA_ARGS__); 
PRINT_LOUT(level); FREE_GOUT; }

#define G_WARNING(...) G_LOG_LEVEL(XCOM_LOG_WARN,__VA_ARGS__)
#define G_ERROR(...) G_LOG_LEVEL(XCOM_LOG_ERROR,__VA_ARGS__)
#define G_MESSAGE(...) G_LOG_LEVEL(XCOM_LOG_INFO, __VA_ARGS__)
#define G_FATAL(...) G_LOG_LEVEL(XCOM_LOG_FATAL, __VA_ARGS__)


XCOM MACROS FOR DEBUGGING
-------------------------
#define PRINT_GOUT xcom_debug("%s", xcom_log_buffer)

#define G_DEBUG_LEVEL(level, ...) { if (xcom_debug_check(level)) { xcom_debug
(__VA_ARGS__) }

#define G_DEBUG(...) G_DEBUG_LEVEL(X_XCOM_DEBUG_BASIC | X_XCOM_DEBUG_TRACE, 
                                   __VA_ARGS__)
#define G_TRACE(...) G_DEBUG_LEVEL(X_XCOM_DEBUG_TRACE, __VA_ARGS__)


CONFIGURING THE XCOM SYSTEM
---------------------------
When bootstrapping GCS, it will call the following functions to configure the
logger, debugger and test option callbacks:

  set_xcom_logger(cb_xcom_logger);
  void cb_xcom_logger(const int64_t level, const char *message)
  {
    Gcs_log_manager::get_logger()->log_event(level, message);
  }

  set_xcom_debugger(cb_xcom_debugger);
  void cb_xcom_debugger(const char *format, ...)
  {
    ...
  }

  set_xcom_test_debug(cb_xcom_test_debug);
  int cb_xcom_test_debug(const int64_t options)
  {
    return Gcs_debug_manager::get_debugger()->test_debugging_option(options);
  }