WL#12063: Encrypting the mysql tablespace

The mysql tablespace is a general tablespace for data dictionary tables and system tables. There is a need to encrypt this tablespace, to protect the tables.

We want to provide the following for the user:

1. Provide a command to encrypt the mysql tablespace for a running server. This command should return to the user when all pages in the tablespace are encrypted.

2. If a crash or shutdown happens during the operation, it should resume when the server restarts.

3. Encrypting the mysql tablespace requires CREATE TABLESPACE on *.* privilege.

4. It should be possible to monitor the progress of mysql tablespace encryption.

5. It should be possible to see if mysql tablesapce is encrypted.

• FR1: During server initialization, mysql.ibd will be unencrypted.
  

"Note: There had been a discussion between Rune/Staale about providing a server option to encrypt 'mysql' tablespace during initialization. It was decided that for simplicity, we do not add such option as of now."

• FR2: A user with privilege "CREATE TABLESPACE on *.*" should be able to alter encryption of mysql.ibd.
  

• FR3: 'ALTER TABLESPACE ... ENCRYPTION' for mysql.ibd should be blocking call

i.e. user should get success/failure if encryption is successful/unsuccessful.

• FR4: Tablespace encryption should be rolled forward on restart if there is a

server crash during the operation.

• FR5: it should be possible to monitor encryption operation on mysql.ibd in performance_schema.events_stages_current.
  

• FR6: It should be possible to see 'encryption' property of mysql.ibd in information_schema.innodb_tablespaces.
  

• FR7: No other ALTER operation should be allowed on mysql.ibd.
  

• WL#9286 provided a design/implementation to encrypt a general tablespace in-place. To encrypt mysql tablespace, same infrastructure will be used i.e. there will be no new/separate implementation of 'tablespace encryption' in InnoDB to encrypt mysql.ibd.

• ALL DDLs are blocked on mysql tablespace as of now but now its required to allow selective DDLs (ALTER ENCRYPT here) on mysql tablespace. For that, a DDL Context will be passed to SE using which SE can decide to block (all other) or allow (ALTER ENCRYPT here for users with CREATE TABLESPACE ON '*.*' privilege) a DDL. (More details in LLD).

• For FR 1, 3, 4, 5, 6 code was already developed and delivered in WL#9286. There is no code changes done for them. Mention of these FRs here is just for functionality verification for mysql tablespace and also a hint for QA/Documentation.

• Performance: Does this change affect performance of reading/writing metadata from/into 'mysql' tablespace ?
  
  • There would be some impact while reading a metadata page from disk because, if encrypted, it would be unencrypted first. But it should be negligible.

• Replication: Is there any impact on replication ?
  • No

• Upgrade: Is there any impact on upgrade from 5.7 or upgrade from older 8.0 database ?
  • No. But upgrade testing should be done.

• Documentation: Is there any inputs to documentation team ?
  • FRs should suffice to give information for documentation.

• SE Specific 'tablespace name validation' API currently takes a bool parameter to indicate if DDL is on tablespace.

typedef bool (*is_valid_tablespace_name_t)(bool tablespace_ddl,
                                           const char *tablespace_name);

It is to be extended to pass the context of DDL on tablespace. This is required to make a decision to allow specific DDLs (ALTER ENCRYPT here) on mysql tablespace. An instance of following class would serve the purpose:

class Tablespace_ddl {
...
 private:
  bool m_tablespace_ddl;
  uint32 m_tablespace_ddl_flags;
};

Where :

m_tablespace_ddl : indicates is it is a tablespace DDL. <br>
m_tablespace_ddl_flags : indicates which tablespace MDL it is. Each byte of this flag represent a DDL operation on tablespace. <br>

And then SE API could be modified as follow:

typedef bool (*is_valid_tablespace_name_t)(Tablespace_ddl tablespace_ddl,
                                           const char *tablespace_name);

• As of now, if there is an update on mysql tablespace metadata (like options, se_private_data etc.) this change is not reflected in cache though it could be persisted on disk. Change is to be done to update the cache.

• When mysql tablespace is opened, DD is not up yet. So we used to pass HARDCODED tablespace flags earlier. But now, mysql tablespace flags depends on its encryption property, we need to rely upon what is store on the disk while opening mysql tablespace file and set in-memory tablespace flags accordingly.