WL#6630: Semisync separate acks collector

Affects: Server-5.7   —   Status: Complete

Objective: optimize replication delivery and the server-to-client response time.
See WL#6355 for related tasks.

O1. to improve semi-sync communication framework

The current semi-sync design is subject to Dump thread hiccups because the one
thread dispatches 2 different and quite hot task - replicate
replicated transaction events to slave and release the connection to
respond back to the client):

                    ack
              ..............
              V            :
    Conn -> Dump ------- > IO
     ^        :  replicate
     :........:
       release

Dump thread could be a real bottle neck. It can't proceed with 2nd
transaction until the 1st one has not yet been acknowledged from the
slave.  Obviously the 2nd transaction propagation plus ack time,
contributing to the response time to the client, is suboptimal.  It's
reasonable to claim that replication throughput is suboptimal as well
for the same reason of hiccup-prone design.

Alternatively, a special ack Collector (Col) thread could be deployed
on the master that would redistribute acks to release waiting
Connections:

    Conn_2             ack
      ^...... Col <...........
      V                      :
    Conn_1 -> Dump ------- > Slave
      ^        :
      :........:
Functional Requirements
=======================
No

Non-Functional Requirements
===========================
NF-1: Reduce semisync delay by using separate threads to send and receive semisync 
      acknowledgement. So event streams and ACK streams can be sent and received 
      simultaneously.
No Interface.
No any change on user level.
ACK Receive Thread In Semisync Master
=====================================
* Start/Stop ack receive thread
  The thread is controlled automatically by semisync master.

  - It is started automatically when enabling semisync master through
    SET rpl_semi_sync_master_enabled = ON

  - It is stopped automatically when disabling semisync master through
    SET rpl_semi_sync_master_enabled = OFF

* Mornitor ack receive thread

  - Ack receive thread status is put in performance_schema.threads table.
    Users can query its status from the table just like mornitoring other
    threads.

  - Column values
    name:                thread/semisync/Ack_receiver
    type:                BACKGROUND
    processlist_state:   A string describes its status.
                         1. Waiting for semi-sync slave connection.
                         2. Waiting for semi-sync ACK from slave .
                         3. Reading semi-sync ACK from slave.

* Semisync slave list
  Semisync master maintains a semisync slave list. Ack receive thread
  listens on all semisync slaves in the list and receive acks from them
  when ack thread is on.

  - Register semisync slave
    Dump threads call transmit_start hook to register its slave to semisync
    master. Only the slave supporting semisync slave is added in the semisync
    list.

  - Unregister semisync slave
    Dump threads call transmit_stop to remove a semisync slave from the
    semisync slave list.

  Note:
    The semisync slave list is maintained even when ack receive thread is
    down. It helps ack thread to listen the semisync slaves connected before
    semisync master is enabled.

* Share TCP connections between dump thread and ack receive thread.

  - Slave IO supports only TCP connections. TCP is full-duplex connection,
    Send and write can work simultaneously in separate threads.

  - But NET and Vio objects are designed for synchronous, it can not be shared
    between threads. So ack receive thread has its own NET and Vio which shares
    sockets with the dump threads.

  - select() is used to listen all semisync slaves.

* Ack_receiver Class
  It encapsulates the code which controls receive thread on master

  - m_status
    Status of receive thread

    DEFINITION:
    enum status m_status;
    enum status { ST_UP, ST_STOPPING, ST_DOWN };
    ST_UP    means ack receive thread is created and is working.
    ST_DOWN  means ack receive thread is destroyed.
    ST_STOPPING means a user is disabling semisync master, and ack receive
                thread is being destroyed.
    Check stop() for detail.

  - m_slaves
    A slave vector which includes slaves' useful information here.

    DEFINITION:
    Slave_vector m_slaves

  - m_mutex
    m_slaves and m_status are shared between user sessions(dump threads) and
    ack thread. So they should be protected by a mutex.

  - start()
    Start receive thread if it is down.

    DEFINITION:
    bool start();

    LOGIC:
    if m_status is ST_DOWN
    {
      set m_status to ST_UP.
      create receive thread.
    }

  - stop()
    Stop receive thread if it is UP.

    DEFINITION:
    void stop();

    LOGIC:
    if m_status is ST_UP
    {
      acquire m_mutex
      set m_status to ST_STOPPING.
      send a signal to wake up ack thread, it may be waiting for a signal.
      wait until m_status is ST_DOWN.
      release m_mutex
    }

  - add_slave()
    Add a new semisync slave to slave list.

    DEFINITION:
    bool add_slave(THD *thd);

    LOGIC:
    initialze slave information.
    acquire m_mutex
    add the slave's information into m_slaves.
    send a signal to ack receive thread. It may be waiting for a signal.
    release m_mutex

  - remove_slave()
    remove a semisync slave from slave list.

    DEFINITION:
    void remove_slave(THD *thd)

    LOGIC:
    acquire m_mutex
    remove thd of the slave from m_slaves.
    release m_mutex

  - run()
    The handle function of receive thread.

    DEFINITION:
    void run();

    LOGIC:
    initialize pthread related things
    while (1)
    {
      acquire m_mutex
      if m_status is ST_STOPPING then break the loop.

      wait any semisync slave to be added if slave list empty.

      call select to listen on sockets, timeout is 1s.
      restart and continue the loop if error or timeout happens.

      receive and report acks to semisync master.
      release m_mutex
    }

    de-initialize pthread related things

    Note: Giving select a timeout makes other threads can add/remove slaves
          or stop ack receive thread when there is no ack.

* ack_receive_handler
  Receive thread handler

  DEFINITION:
  pthread_handler_t ack_receive_handler(void *arg);

  LOGIC:
  call ack_receiver::run().