WL#9210: InnoDB: Clone remote replica

Enhance server clone plugin created in WL#9209 to connect to a server from
remote and transfer the snapshot over network.

Here we would support taking a Physical Snapshot of the database and transfer it
over network to another machine/node where the replica needs to be provisioned.

This is one of the child work logs for provisioning a clone for replication.

Functional
----------
F-1: Must support Simple command to clone InnoDB in remote server
     CLONE INSTANCE FROM USER@HOST:PORT 
     IDENTIFIED BY ''
     DATA DIRECTORY [=]''
     [REQUIRE [NO] SSL];

USER     : USER name to connect to donor mysqld server
password : Password for the USER
HOST     : The host where donor server is running
PORT     : Classic protocol PORT of donor server
  * X-protocol PORT is not supported
  * Connecting to the donor server through router is not supported.

F-1A: Must support clone over mysql encrypted connection and certificate
verification. Should use server SSL configuration.

F-1B: Must hide  in performance_schema views and in "show processlist" 

F-2: Must have BACKUP_ADMIN privilege to execute CLONE command.

F-3: Must clone tablespaces outside data directory
       - Clone to same directory in recipient
       - Error if no permission or if file exists
F-3A: Must clone Innodb system tablespace, redo and undo placed outside data
directory in donor. Should place in cloned data directory.

F-4 : Should resume clone operation after a network error during clone.
F-4A: Should error out if network is not up in 5 minutes

F-5: Must allow tracking progress via performance_schema views. The
specification is same as local clone as described in wl#9212.

Non-functional
--------------
NF-1: [Performance] Should transfer large data over network efficiently with
minimal intermediate storage.

NF-2: [Availability] Should not block donor for long time (< 1 second)

NF-3: [Performance] Should not have significant impact on donor performance
      (should not be worse than MEB)

NF-4: [Security] Remote connection must go through MySQL server authentication
mechanism and should be able to use secure connection protocols (SSL) as
supported by MySQL server.

Steps for cloning a database from remote node
---------------------------------------------
1. create dummy database [mysqld--initialize]
2. Start Server: Recipient
3. Connect to recipient
4. Clone database from remote node
     SQL > INSTALL PLUGIN CLONE
     SQL > CLONE REMOTE INSTANCE
5. Shutdown server and restart using cloned data directory

CLIENT(CLONE INSTANCE command)
        | 
 [RECIPIENT SERVER] ---Network--- [DONOR SERVER]
        |                                |
   CLONED DATA                          DATA


Clone operation
---------------
"CLONE INSTANCE" SQL command, in local server[RECIPIENT], would call clone
plugin client interface. The client would first connect to the remote server
[DONOR] using the credentials passed by SQL command. It would then initiate the
communication with the new RPC "COM_CLONE". The client would then start sending
clone specific commands to server - CLONE PROTOCOL

The remote server[DONOR], on receiving COM_CLONE, would call clone plugin server
 interface and and wait for clone specific commands - CLONE PROTOCOL.


RECIPIENT                        DONOR
=========                        =====
    |                              |
    |-----------Connect----------->|  /* Connect to Remote Node */
    |<------success/failure--------|
    |
    |---------COM_CLONE----------->|  /* Switch to clone protocol */
    |<-----success/failure---------|
    |                              |
    |---------COM_INIT------------>|  /* Begin Clone operation */
    |<------COM_RES_LOCS-----------|  /* SE Locators */
    |                              |
    |-------COM_EXECUTE----------->|  /* Stream serialized data */
    |                              |
    |<---COM_RES_DATA_DESCRIPTOR---|  /* Data/Task/State descriptors */
    |<-------COM_RES_DATA----------|  /* Raw Data */
    |                              |
    |<---COM_RES_DATA_DESCRIPTOR---|  /* Data/Task/State descriptors */
    |<-------COM_RES_DATA----------|  /* Raw Data */
    |           ...                |
    |---------COM_ACK------------->|  /* Acknowledge State transfer */
    |           ...
    |------COM_RES_COMPLETE--------|  /* Transfer completed successfully*/
    |
    |----------COM_EXIT----------->|  /* Switch back to mysql protocol */ 
    |<------success/failure--------|
    |
    |--------Disconnect----------->|


CLONE PROTOCOL: The set of RPC commands and response codes for clone operations.
---------------
1. COM_INIT: Negotiate clone protocol version and begin clone operation with
storage engines. The remote SE locators are sent back to client. 

For future extension of the clone protocol, the clone plugins would first do the
version negotiation. If the version passed by the recipient is higher than the
donor version then it would send back it own version to the recipient. In
general they would agree on the lowest among the maximum version supported by
both.

We would also have the database version check here to avoid cloning a when
database versions are not of same.

2. COM_ATTACH: A new thread of control to attach to current clone operation.
This is for multi-threading support. The session thread does COM_INIT to
initialize clone. After that multiple slave threads can be attached to the
current clone operation at any stage. 

3. COM_REINIT: Restart clone operation after network failure. On network failure
, master threads in clone client and clone server waits for all other threads to
exit. The client then tries to re-connect back to server. On successful
reconnect, client sent the current SE locators to server with details of
stage/chunk/block information

The locator for a clone operation is a logical pointer to a snapshot (along with
the transfer state) within innodb storage engine. It is useful to restart a half
way done cloning operation. The client sends previous locator from a
unfinished clone operation. The server returns back the locator after
re-initializing the snapshot with InnoDB.

4. COM_EXECUTE: Start clone and stream data back to client.

The serialized descriptor is specific to SE and would be passed directly to the
SE while storing the data in recipient. We would also send an ID as part of the
data descriptor to find out which SE the data belongs to.

The donor streams the data from snapshot over network. The data is sent in
multiple chunks as response data packets.

A. COM_RES_LOCS : The set of locators from client to server

B. COM_RES_DATA_DESC : A data descriptor that explains the data packet to
follow. First part of the descriptor is the the SE index for clone plugin to
invoke appropriate SE. The second part is specific to SE. For innodb it carries
one of the following ...
  1. State information
  2. Task information
  3. File metadata
  4. Location next data block - file index and offset

C. COM_RES_DATA : Raw Data in length data format

D. COM_RES_COMPLETE : Clone finished successfully 

E. COM_RES_ERROR : Clone finished with Error.

The whole data would be transferred in multiple such chunks with COM_RES_DATA.
After entire data is transferred, the donor indicate the end by sending
CLONE_COM_END.

5. COM_ACK: Acknowledge successful switch to different clone states. This is to
ensure that clone operation can be restarted after network failure. Donor can
only move to next stage after receiving successful ACK from recipient. The
dynamic snapshot cannot move back to previous stage as the archiving might
already have been stopped.

This is also used to communicate any error in clone client back to the clone
server. It is to be noted that during COM_EXECUTE the data is streamed one way
from clone server to clone client.

6. COM_EXIT: Exit clone plugin and protocol and go back to usual server protocol.

7. Clone Plugin configuration variables

clone_buffer_size
-----------------
Scope: Global
Dynamic: Yes
Type: Integer
Default Value:   4MiB
Minimum Value:   1MiB
Maximum Value: 256MiB
Description: Intermediate buffer size for transferring data

clone_max_concurrency
---------------------
Scope: Global
Dynamic: Yes
Type: Integer
Default Value:  8
Minimum Value:  1
Maximum Value: 64
Description: Maximum number of concurrent threads used for clone

7A. Clone SSL client configuration for recipient for connecting to remote server
using SSL certificates. Following three configurations in recipient must match
client key/certificate of donor server for successful authentication. This is
very similar to how mysql client would authenticate to a remote server with
certificate.

clone_ssl_key
-------------
Description: SSL path name of the SSL private key file for recipient

clone_ssl_cert
---------------------
Description:SSL path name of the public key certificate file for recipient 

clone_ssl_ca
-------------------
Description: SSL path name for Certificate Authority (CA) file for recipient

Clone would use the SSL configuration of recipient server for all other SSL
configurations.

1. ssl_cipher
2. tls_version
3. ssl_capath
4. ssl_crl
5. ssl_crlpath