WL#12080: Add support for binary log encryption key rotation and cleanup

High-Level Description / Executive Summary

EXECUTIVE SUMMARY
=================
This worklog implements the ability to rotate the binlog master key used
to encrypt file passwords of binary and relay log files.
The binlog master key should be rotated periodically and whenever you
suspect that the key may have been compromised.
Rotating the binlog master key not only changes the key used to encrypt
new file password, but also re-encrypts file passwords of previous existent
encrypted binary or relay log file with the new generated key. It also
cleanups the keyring, removing the binlog encryption keys generated by
the server that are not needed for server operation anymore.

User stories
============
- As a MySQL DBA/operator/instance owner, I want to rotate binlog encryption
  master key periodically without the need to restart the server in order to
  comply with security rules.
- As a MySQL DBA/operator/instance owner, I suspect that the some of existing
  binlog encryption master keys were exposed. I want to rotate the binlog
  master key so, starting this point, all new encrypted binary and relay log
  files shall use the new generated key to encrypt their file passwords, and
  file passwords of existent encrypted binary or relay log files are
  re-encrypted with the new generated key.

Functional and Non-Functional Requirements

FUNCTIONAL REQUIREMENTS
=======================
F-1: MySQL server should introduce a new command to rotate the binlog master
     key used to change the key used to encrypt new file password and
     re-encrypt file passwords of previous existent encrypted binary or
     relay log files with the new generated key with syntax:
       ALTER INSTANCE ROTATE BINLOG MASTER KEY;
     Executing the new command shall emit an error to the client side on
     the session when binlog-encryption is off regardless of the value
     of log_bin.
     The new command shall not be replicated while executing it.

Interface Specification
=======================
I-1: No new files
I-2: No changes in existing syntax
I-3: No new tools
I-4: No impact on existing functionality
I-5: Introduce a new command 'ALTER INSTANCE ROTATE BINLOG MASTER KEY'
     to rotate the binlog encryption key. Executing the new command
     requires the BINLOG_ENCRYPTION_ADMIN or SUPER privilege.


HIGH LEVEL SPECIFICATION
========================
1. MySQL server introduces a new command to rotate the binlog master
   key used to change the key used to encrypt new file password and
   re-encrypt file passwords of previous existent encrypted binary or
   relay log files with the new generated key with syntax:
     ALTER INSTANCE ROTATE BINLOG MASTER KEY;

   Executing the new command shall emit an error to the client side on
   the session when binlog-encryption is off regardless of the value
   of log_bin. We still have a requirement to rotate binlog master key
   for relay logs on binlogless slaves when binlog-encryption is on.
   So rotating binlog master key shall result in an error when
   binlog-encryption is off even in binlogless slaves.

   The new command shall not be replicated while executing it.


2. Binlog master key rotation
   Binlog master key rotation shall only be possible when 'binlog-encryption'
   is on. Requests to rotate the binlog master key when 'binlog-encryption'
   is off shall return an error.

   2.1: Set that the binlog master key rotation has started
     Store a key id "MySQLReplicationKey\_{UUID}\_old" with the
     `master key SEQNO` as its key on keyring.

   2.2: Determine the next SEQNO to be used by a new master key
     Do: `master key SEQNO = master key SEQNO + 1` by P-4 in WL#10957. Then,
     check if a binlog encryption key exists on keyring with the new
     `master key SEQNO`. In order to be compliant with P-4 in WL#10957, the
     server shall repeat this process until reaching a `master key SEQNO`
     without a corresponding binlog encryption key on keyring.
     Store a key id "MySQLReplicationKey\_{UUID}\_new" with the master key SEQNO
     as its key on keyring.

   2.3: Generate the new binlog master key
     Request the keyring to generate a new key by key id
     "MySQLReplicationKey\_{UUID}\_{SEQNO}" using `master key SEQNO` as SEQNO.

   2.4: Remove master key "index" from keyring
     Remove the "MySQLReplicationKey\_{UUID}" key from keyring (if it exists).
     If unable to remove the key, the server shall throw an error (I-15 in
     WL#10957).

   2.5: Store master key "index" on keyring
     Store key id "MySQLReplicationKey\_{UUID}" with the `new master key SEQNO`
     as its key on keyring. This will allow that a server shall be able to
     determine the binlog master key. If unable to store the key, the server
     shall throw an error (I-14 in WL#10957).

   2.6: Rotate and re-encrypt replication logs
     2.6.1: Rotate and re-encrypt binary logs when binary logging is enabled
       2.6.1.1: Rotate the binary log to create a new binary log file
                encrypted with new master key.
       2.6.1.2: Re-encrypt previous existent binary logs as below.
         Starting from the next to last entry on the index file, iterating down
to the first one:
           - If the file is encrypted, re-encrypt it. Otherwise, skip it.
           - If failed to open the file, report an error.

     2.6.2: Rotate and re-encrypt relay logs for each existing replication 
channel
       2.6.2.1: Rotate the relay log (except for GR applier) to create a new
                relay log file encrypted with new master key.
                Note: BUG#92526 MAKE GR REPLICATION CHANNELS ROTATE RELAY LOG
ON FLUSH LOGS as a way to remove GR exceptions from this WL.
       2.6.2.2: Re-encrypt previous existent relay logs as below.
         Starting from the next to last entry on the index file, iterating down
to the first one:
           - If the file is encrypted, re-encrypt it. Otherwise, skip it.
           - If failed to open the file, report an error.

   2.7: Purge unused old binlog encryption keys from keyring
     Retrieve the `last purged SEQNO` from keyring's
     "MySQLReplicationKey\_{UUID}\_last\_purged" or 0 when the key
     is not present on keyring.

     Set `purge start = last purged SEQNO`. If `purge start` is 0, make
     `purge start = 1`.
     Set `purge end = `master key SEQNO - 1`.
     Loop from `purge start` to `purge end`, remove all the keys like
     "MySQLReplicationKey\_{UUID}\_{SEQNO}" from the keyring.

     If `purge end > 0` and `purge end != last purged SEQNO`: make
     `last purged SEQNO = purge end`, remove
     "MySQLReplicationKey\_{UUID}\_last\_purged" key from keyring and store
     "MySQLReplicationKey\_{UUID}\_last\_purged" key into keyring with the
     new `last purged SEQNO`.

     Remove "MySQLReplicationKey\_{UUID}\_old" key from keyring.

   2.8: Finalize binlog master key rotation
     Remove "MySQLReplicationKey\_{UUID}\_new" key from keyring.


3. Binlog master key rotation recovery
   When starting the server, we should get the binlog master key and its
   corresponding sequence number from keyring before opening any new
   replication log, because they will be used to encrypt the new generated
   replication logs when binlog encryption is ON. So before initializing
   replication logs, we need to do binlog master key rotation recovery as
   following:
   Try to retrieve "MySQLReplicationKey\_{UUID}" from keyring. Store its
   content into `master key SEQNO`.
   Try to retrieve "MySQLReplicationKey\_{UUID}\_old" from keyring. Store
   its content into `old master key SEQNO`.
   Try to retrieve "MySQLReplicationKey\_{UUID}\_new" from keyring. Store
   its content into `new master key SEQNO`.

```Take action according to the following table:
--------------------------------------------------------------------------------
-----------
| master key SEQNO | old master key SEQNO| new master key SEQNO | new master key
| Action |
--------------------------------------------------------------------------------
-----------
| doesn't exist    | doesn't exist       | doesn't exist        | N/A          
 | Do binlog master key rotation from START. |
| n > 0            | doesn't exist       | doesn't exist        | N/A          
 | Ordinary server startup with known binlog master key. |
| n >= 0            | n                   | doesn't exist        | N/A         
  | Continue binlog master key rotation from DETERMINE_NEXT_SEQNO. |
| n >= 0            | n                   | m > n                | doesn't
exists | Continue binlog master key rotation from GENERATE_NEW_MASTER_KEY. |
| n > 0            | n                   | m > n                | exists       
 | Continue binlog master key rotation from REMOVE_MASTER_KEY_INDEX. |
| doesn't exist    | n >= 0               | m > n                | exists      
  | Continue binlog master key rotation from STORE_MASTER_KEY_INDEX. |
| m > 0            | n >= 0 (exist)       | m > n                | exists      
  | Continue binlog master key rotation from ROTATE_LOGS. |
| m > 0            |doesn't exist        | m                    | exists       
 | Continue binlog master key rotation from REMOVE_KEY_ROTATION_TAG. |```

  All above actions need to skip log rotation (2.6) and unused encryption
  keys cleanup (2.7), because binary/relay logs will be rotated later on
  at the begin of the recovery.
  Any other combination from the table shall abort the recovery with
  an error if binlog encryption is ON.


4. Update the process of enabling binlog encryption as below
   Do above binlog master key rotation recovery if binlog master key
   rotation is not recovered.


5. Update the process of disabling binlog encryption as below
   Make `master key SEQNO = 0`.

   When binary logging is enabled, rotate the binary log to create a new binary
   log file with plain binary log file format.

   For each existing replication channel, rotate the relay log (except for GR
   applier) to create a new relay log file with plain binary log file format.


UPGRADE/DOWNGRADE and CROSS-VERSION REPLICATION
===============================================
NEW->OLD: Suppose user issues ALTER INSTANCE ROTATE REPLICATION MASTER KEY on
          a new master. Even if there is an error when the statement is
          executed in a client to the OLD slave, it does not cause replication
          to break, because the statement is not logged and replicated when
          executing it.

OLD->NEW: Suppose user issues ALTER INSTANCE ROTATE BINLOG MASTER KEY on an
          old master. The statement is not executed on the old master,
          because it is out wih an error. So this does not cause replication
          to break.

On upgrade, there is one new limitation:
  -- No binlog master key rotation shall be possible when the
     'binlog-encryption' is off.

NEW master, NEW slave - master or slave server is patched with the WL#12080
OLD master, OLD slave - master or slave server is not patched with the WL#12080


OBSERVABILITY
=============
Introduce several errors and emit them. See below.
ER_RPL_ENCRYPTION_CANNOT_ROTATE_BINLOG_MASTER_KEY
  eng "Cannot rotate binary log master key when 'binlog-encryption' is off."
Emit the above error to the client side on the session when executing
the command 'ALTER INSTANCE ROTATE BINLOG MASTER KEY' if
'binlog-encryption' is off.

ER_BINLOG_MASTER_KEY_ROTATION_FAIL_TO_OPERATE_KEY
  eng "Failed to operate binary log master key on keyring, please check if
keyring plugin is loaded. The statement had no effect: the old binary log master
key is still in use, the keyring, binary and relay log files are unchanged, and
the server could not start using a new binary log master key for encrypting new
binary and relay log files."
Emit the above error to the client side on the session when executing
the command 'ALTER INSTANCE ROTATE BINLOG MASTER KEY' if there is an
error when operating binary log master key on keyring before rotating and
re-encrypting binary and relay log files.

ER_BINLOG_MASTER_KEY_ROTATION_FAIL_TO_ROTATE_LOGS
  eng "Failed to rotate one or more binary or relay log files. A new binary log
master key was generated and will be used to encrypt new binary and relay log
files. There may still exist binary or relay log files using the previous binary
log master key."
Emit the above error to the client side on the session when executing
the command 'ALTER INSTANCE ROTATE BINLOG MASTER KEY' if there is an
error when retating binary or relay log files.

ER_BINLOG_MASTER_KEY_ROTATION_FAIL_TO_REENCRYPT_LOG
  eng "%s. A new binary log master key was generated and will be used to encrypt
new binary and relay log files. There may still exist binary or relay log files
using the previous binary log master key."
Emit the above error to the client side on the session when executing
the command 'ALTER INSTANCE ROTATE BINLOG MASTER KEY' if there is an
error when re-encrypting binary or relay log files.

ER_BINLOG_MASTER_KEY_ROTATION_FAIL_TO_CLEANUP_UNUSED_KEYS
  eng "Failed to remove unused binary log encryption keys from the keyring,
please check if keyring plugin is loaded. The unused binary log encryption keys
may still exist in the keyring, and they will be removed upon server restart or
next 'ALTER INSTANCE ROTATE BINLOG MASTER KEY' execution."
Emit the above warning to the client side on the session when executing
the command 'ALTER INSTANCE ROTATE BINLOG MASTER KEY' if there is an
error when cleaning up unused keys.

ER_BINLOG_MASTER_KEY_ROTATION_FAIL_TO_CLEANUP_AUX_KEY
  eng "Failed to remove auxiliary binary log encryption key from keyring, please
check if keyring plugin is loaded. The cleanup of the binary log master key
rotation process did not finish as expected and the cleanup will take place upon
server restart or next 'ALTER INSTANCE ROTATE BINLOG MASTER KEY' execution."
Emit the above warning to the client side on the session when executing
the command 'ALTER INSTANCE ROTATE BINLOG MASTER KEY' if there is an
error when cleaning up auxiliary keys.

ER_BINLOG_MASTER_KEY_RECOVERY_OUT_OF_COMBINATION
  eng "Unable to recover binary log master key, the combination of
new_master_key_seqno=%u, master_key_seqno=%u and old_master_key_seqno=%u are 
wrong."
Emit the above error to the client side on the session when executing
the command 'SET @@GLOBAL.binlog_encryption=ON' if there is an error
when recovering binary log master key rotation.

ER_SERVER_BINLOG_MASTER_KEY_RECOVERY_OUT_OF_COMBINATION
  eng "Unable to recover binary log master key, the combination of
new_master_key_seqno=%u, master_key_seqno=%u and old_master_key_seqno=%u are 
wrong."
Write the above error to server error log while starting up the server
with binlog_encryption enabled if there is an error while recovering
binary log master key rotation.

ER_SERVER_BINLOG_MASTER_KEY_ROTATION_FAIL_TO_CLEANUP_AUX_KEY
  eng "Failed to remove auxiliary binary log encryption key from keyring, please
check if keyring plugin is loaded. The cleanup of the binary log master key
rotation process did not finish as expected and the cleanup will take place upon
server restart or next 'ALTER INSTANCE ROTATE BINLOG MASTER KEY' execution."
Write the above warning to server error log while starting up the server
with binlog_encryption enabled if there is an error on cleaning up auxiliary
keys while recovering binary log master key rotation.

Low-Level Design
================
The 1 and 2 in high level specification will be implemented in step 1, and 3, 4
and 5 in high level specification will be implemented in step 2.