WL#9210: InnoDB: Clone remote replica
Affects: Server-8.0 — Status: Complete
Enhance server clone plugin created in WL#9209 to connect to a server from remote and transfer the snapshot over network. Here we would support taking a Physical Snapshot of the database and transfer it over network to another machine/node where the replica needs to be provisioned. This is one of the child work logs for provisioning a clone for replication.
Functional ---------- F-1: Must support Simple command to clone InnoDB in remote server CLONE INSTANCE FROM USER@HOST:PORT IDENTIFIED BY '
' DATA DIRECTORY [=]' ' [REQUIRE [NO] SSL]; USER : USER name to connect to donor mysqld server password : Password for the USER HOST : The host where donor server is running PORT : Classic protocol PORT of donor server * X-protocol PORT is not supported * Connecting to the donor server through router is not supported. F-1A: Must support clone over mysql encrypted connection and certificate verification. Should use server SSL configuration. F-1B: Must hide in performance_schema views and in "show processlist" F-2: Must have BACKUP_ADMIN privilege to execute CLONE command. F-3: Must clone tablespaces outside data directory - Clone to same directory in recipient - Error if no permission or if file exists F-3A: Must clone Innodb system tablespace, redo and undo placed outside data directory in donor. Should place in cloned data directory. F-4 : Should resume clone operation after a network error during clone. F-4A: Should error out if network is not up in 5 minutes F-5: Must allow tracking progress via performance_schema views. The specification is same as local clone as described in wl#9212. Non-functional -------------- NF-1: [Performance] Should transfer large data over network efficiently with minimal intermediate storage. NF-2: [Availability] Should not block donor for long time (< 1 second) NF-3: [Performance] Should not have significant impact on donor performance (should not be worse than MEB) NF-4: [Security] Remote connection must go through MySQL server authentication mechanism and should be able to use secure connection protocols (SSL) as supported by MySQL server.
Steps for cloning a database from remote node --------------------------------------------- 1. create dummy database [mysqld--initialize] 2. Start Server: Recipient 3. Connect to recipient 4. Clone database from remote node SQL > INSTALL PLUGIN CLONE SQL > CLONE REMOTE INSTANCE 5. Shutdown server and restart using cloned data directory CLIENT(CLONE INSTANCE command) | [RECIPIENT SERVER] ---Network--- [DONOR SERVER] | | CLONED DATA DATA Clone operation --------------- "CLONE INSTANCE" SQL command, in local server[RECIPIENT], would call clone plugin client interface. The client would first connect to the remote server [DONOR] using the credentials passed by SQL command. It would then initiate the communication with the new RPC "COM_CLONE". The client would then start sending clone specific commands to server - CLONE PROTOCOL The remote server[DONOR], on receiving COM_CLONE, would call clone plugin server interface and and wait for clone specific commands - CLONE PROTOCOL. RECIPIENT DONOR ========= ===== | | |-----------Connect----------->| /* Connect to Remote Node */ |<------success/failure--------| | |---------COM_CLONE----------->| /* Switch to clone protocol */ |<-----success/failure---------| | | |---------COM_INIT------------>| /* Begin Clone operation */ |<------COM_RES_LOCS-----------| /* SE Locators */ | | |-------COM_EXECUTE----------->| /* Stream serialized data */ | | |<---COM_RES_DATA_DESCRIPTOR---| /* Data/Task/State descriptors */ |<-------COM_RES_DATA----------| /* Raw Data */ | | |<---COM_RES_DATA_DESCRIPTOR---| /* Data/Task/State descriptors */ |<-------COM_RES_DATA----------| /* Raw Data */ | ... | |---------COM_ACK------------->| /* Acknowledge State transfer */ | ... |------COM_RES_COMPLETE--------| /* Transfer completed successfully*/ | |----------COM_EXIT----------->| /* Switch back to mysql protocol */ |<------success/failure--------| | |--------Disconnect----------->| CLONE PROTOCOL: The set of RPC commands and response codes for clone operations. --------------- 1. COM_INIT: Negotiate clone protocol version and begin clone operation with storage engines. The remote SE locators are sent back to client. For future extension of the clone protocol, the clone plugins would first do the version negotiation. If the version passed by the recipient is higher than the donor version then it would send back it own version to the recipient. In general they would agree on the lowest among the maximum version supported by both. We would also have the database version check here to avoid cloning a when database versions are not of same. 2. COM_ATTACH: A new thread of control to attach to current clone operation. This is for multi-threading support. The session thread does COM_INIT to initialize clone. After that multiple slave threads can be attached to the current clone operation at any stage. 3. COM_REINIT: Restart clone operation after network failure. On network failure , master threads in clone client and clone server waits for all other threads to exit. The client then tries to re-connect back to server. On successful reconnect, client sent the current SE locators to server with details of stage/chunk/block information The locator for a clone operation is a logical pointer to a snapshot (along with the transfer state) within innodb storage engine. It is useful to restart a half way done cloning operation. The client sends previous locator from a unfinished clone operation. The server returns back the locator after re-initializing the snapshot with InnoDB. 4. COM_EXECUTE: Start clone and stream data back to client. The serialized descriptor is specific to SE and would be passed directly to the SE while storing the data in recipient. We would also send an ID as part of the data descriptor to find out which SE the data belongs to. The donor streams the data from snapshot over network. The data is sent in multiple chunks as response data packets. A. COM_RES_LOCS : The set of locators from client to server B. COM_RES_DATA_DESC : A data descriptor that explains the data packet to follow. First part of the descriptor is the the SE index for clone plugin to invoke appropriate SE. The second part is specific to SE. For innodb it carries one of the following ... 1. State information 2. Task information 3. File metadata 4. Location next data block - file index and offset C. COM_RES_DATA : Raw Data in length data format D. COM_RES_COMPLETE : Clone finished successfully E. COM_RES_ERROR : Clone finished with Error. The whole data would be transferred in multiple such chunks with COM_RES_DATA. After entire data is transferred, the donor indicate the end by sending CLONE_COM_END. 5. COM_ACK: Acknowledge successful switch to different clone states. This is to ensure that clone operation can be restarted after network failure. Donor can only move to next stage after receiving successful ACK from recipient. The dynamic snapshot cannot move back to previous stage as the archiving might already have been stopped. This is also used to communicate any error in clone client back to the clone server. It is to be noted that during COM_EXECUTE the data is streamed one way from clone server to clone client. 6. COM_EXIT: Exit clone plugin and protocol and go back to usual server protocol. 7. Clone Plugin configuration variables clone_buffer_size ----------------- Scope: Global Dynamic: Yes Type: Integer Default Value: 4MiB Minimum Value: 1MiB Maximum Value: 256MiB Description: Intermediate buffer size for transferring data clone_max_concurrency --------------------- Scope: Global Dynamic: Yes Type: Integer Default Value: 8 Minimum Value: 1 Maximum Value: 64 Description: Maximum number of concurrent threads used for clone 7A. Clone SSL client configuration for recipient for connecting to remote server using SSL certificates. Following three configurations in recipient must match client key/certificate of donor server for successful authentication. This is very similar to how mysql client would authenticate to a remote server with certificate. clone_ssl_key ------------- Description: SSL path name of the SSL private key file for recipient clone_ssl_cert --------------------- Description:SSL path name of the public key certificate file for recipient clone_ssl_ca ------------------- Description: SSL path name for Certificate Authority (CA) file for recipient Clone would use the SSL configuration of recipient server for all other SSL configurations. 1. ssl_cipher 2. tls_version 3. ssl_capath 4. ssl_crl 5. ssl_crlpath
Copyright (c) 2000, 2023, Oracle Corporation and/or its affiliates. All rights reserved.