WL#9290: InnoDB: Support Transparent Data Encryption for Redo Log

Affects: Server-8.0   —   Status: Complete

This work will provide encryption support for redo log.

1:Encrypt redo log blocks in I/O layer.
We will en/decrypt the redo log in the I/O layer. Which means, the en/decryption
only happens when the redo log blocks read or write from/to disk. And the redo
log data in memory will stay in unencrypted status.

2:Encryption metadata will be stored in the header of first log file.
There are 2 key levels here, master key and tablespace key.
Master key is stored in keyring plugin, and it's used to en/decrypt tablespace
key and iv. Tablespace key is for en/decrypt redo log blocks, and it will be 
stored into the 3rd block of first redo log file(ib_logfile0).
So, we will not encrypt redo log file header, which is in the first 4 blocks. 

The first 4 blocks of first redo log file will contains these information:
first block: log file header
2nd   block: checkpoint1
3rd   block: encryption metadata
4th   block: checkpoint2

The encryption meta data information of redo log has the same format with it in 
normal tablespaces. like:
encryption version + master key id + server uuid + encrypted tablespace key + IV

This meta data will be read from ib_logfile0 in server startup.

3:The redo log block is encrypted one by one.
For each block, we will not encrypt the block header. It's because we use a bit
in the block header to indicate if it's an encrypted block or not.
When a redo log block is writing into disk, this bit will be set after
this block has been encrypted. And it will be checked when it's read from disk. 
The decrypt function will be called if this bit is set.

4:For key rotation, we will just need to save the new encryption metadata 
information to block 3, and flush it to disk.
Key rotation steps:
Step1: Create a new master key and fetch it from keyring;
Step2: Use the new master key to encrypt redo log tablespace key and iv.
Step3: Store the new encryption metadata into the 3rd block for ib_logfile0.

5:We added a new global variable innodb_redo_log_encrypt=ON/OFF for en/disable 
redo log encryption.
And after user enable redo log encryption, the redo log blocks will be encrypted 
when it's writing into disk. the previous redo log which is already in disk
will be leaved as not encrypted status.
On the other hand, after it's disable, the old redo log blocks will be leave as
encrypted status, and the new redo log blocks will not be encrypted.

If innodb_redo_log_encrypt is enabled in bootstrap, we use a default master key
for encrypting redo log tablespace key and iv. It's because there's no necessary
information like server uuid in bootstrap. And in the next master key rotation, 
the default master key will be thrown away.

Limitations
===========
1:If the ib_logfile0 is removed, redo log encryption will be disabled. It's 
because the encryption metadata in ib_logfile0 has also been removed with 
ib_logfile0.
2:Normal restart without keyring plugin/missing key file is not possible once 
redo log encryption is enabled when server was up. This is because InnoDB need 
to scan some redo blocks in normal startup, and these blocks are possible 
encrypted if redo log encryption was enabled before. User still can startup 
server by trying to startup with --innodb_force_recovery= SRV_FORCE_NO_LOG_REDO.
Requirements

F1 - Support AES encryption
F2 - En/disable redo log encryption by set innodb_redo_log_encrypt=ON/OFF.
F3 - Support Key rotation.
F4 - Read encryption information from header of ib_logfile0 properly (recovery 
after crash)

Non-Functional requirements
NF-1 - Performance impact of redo log encryption will be less than 10%
NF-2 - Will not generate any extra redo log
1:En/disable redo log encryption:
Introduce a global system variable innodb_redo_log_encrypt
  ie:set global innodb_redo_log_encrypt=ON, start encrypt redo log.

2:En/decrypt redo log blocks:
Since the redo log blocks has different format with normal data page,
we need to add new en/decryption functions into class Encryption.
And these new en/decryption function will be called in I/O.

3:Store encryption metadata:
A new function will be added for storing the encryption metadata into
the 3rd block of ib_logfile0. This function will be called when redo log 
encryption is enabling and key rotation. And also, we need add a new function
for reading encryption metadata from the file header.

4:Key rotation and recovery
For key rotation, we need a new handle function as well. And this function
will be called by innobase_encryption_key_rotation.
In this function, the new master key will be used to encrypt the redo log
tablespace key and iv, then store them into the log file header.

1:En/disable redo log encryption:
Add new global variable innodb_redo_log_encrypt definition in ha_innobase.cc:

static MYSQL_SYSVAR_BOOL(redo_log_encrypt, srv_redo_log_encrypt,
  PLUGIN_VAR_OPCMDARG,
  "Enable or Disable Encryption of REDO tablespace.",
  NULL, NULL, FALSE);

2:En/decrypt redo log blocks:
Add member functions for en/decrypt redo log blocks into class Encryption:

In os0file.h:

/** Encryption algorithm. */
struct Encryption {
.......

	/** Check if a log block is encrypted or not
	@param[in]	block	block which need to check
	@return true if it is an encrypted block */
	static bool is_encrypted_log(const byte* block)
		MY_ATTRIBUTE((warn_unused_result));

	/** Encrypt the redo log data contents.
	@param[in]	type		IORequest
	@param[in,out]	src		page data which need to encrypt
	@param[in]	src_len		Size of the source in bytes
	@param[in,out]	dst		destination area
	@param[in,out]	dst_len		Size of the destination in bytes
	@return buffer data, dst_len will have the length of the data */
	byte* encrypt_log(
		const IORequest&	type,
		byte*			src,
		ulint			src_len,
		byte*			dst,
		ulint*			dst_len);
.....
	/** Decrypt the redo data contents.
	@param[in]	type		IORequest
	@param[in,out]	src		Data read from disk, decrypt
					data will be copied to it
	@param[in]	src_len		source data length
	@param[in,out]	dst		Scratch area to use for decrypt
	@param[in]	dst_len		Size of the scratch area in bytes
	@return DB_SUCCESS or error code */
	dberr_t decrypt(
		const IORequest&	type,
		byte*			src,
		ulint			src_len,
		byte*			dst,
		ulint			dst_len)
		MY_ATTRIBUTE((warn_unused_result));

.....
};

3:Store encryption metadata:
Function for reading encryption meta data from log file header:

/* Read the first log file header to get the encryption
information if it exist.
@param[in]	log_space_id	log tablespace id
@return true if success */
bool
log_group_file_header_read_encryption(
	ulint	log_space_id);

Function for storing encryption metadata and key rotation:

/** Rotate the encryption info in the log file header.
@param[in]	is_boot		if it is for bootstrap
@return true if success. */
bool
log_group_file_header_rotate_encryption(bool is_boot);