WL#9770: Keyring plugin for Oracle Public Cloud key store

Affects: Server-8.0   —   Status: Complete

Implement a MySQL keyring plugin talking to an Oracle Public Cloud's Key management service (Now known as Oracle Cloud Infrastructure Vault, OCI Vault) (1) back-end.

https://docs.cloud.oracle.com/iaas/Content/KeyManagement/Concepts/keyoverview.htm

MySQL provides keyring plugins for both the OKV and the competing AWS KMS. The OCI Vault is a natural fit for these secrets. So this plugin is a recommended addon for 
MySQL OCI instances. It can work on premise too. And be a great gateway to OCI. Standalone hardware key servers can be expensive. If you trust OCI with your keys, then the 
OCI can be a great substitute for the expensive on-premise hardware appliance.
FR1 - KEY STORAGE
      All keys are to be stored in, and retrieved from the Oracle's Cloud 
Infrastructure Vault service (OCI Vault).
 
FR2 - ENTITY MAPPING
      MySQL keys are uniquely mapped to the OCI Vault internal secrets.
 
FR3 - CONSUMER SUPPORT
      Backend is to support all of the MySQL server keyring consumers.
 
FR4 - AUTHENTICATION
      The plugin will use the provided user's credentials.
 
FR5 - PLUGIN CONFIGURATION
      The plugin will support the following configuration parameters coming from 
persistent system variables:
      FR5.1 - User OCID of the OCI user.
      FR5.2 - Tenancy OCID of the OCI tenancy.
      FR5.3 - Compartment OCID of the MySQL sub-compartment.
      FR5.4 - Vault OCID of the OCI Vault the MySQL will use.
      FR5.5 - OCI encryption endpoint URL
      FR5.6 - OCI Key Management endpoint URL
      FR5.7 - OCI Vault endpoint URL
      FR5.8 - OCI Secrets endpoint URL
      FR5.9 - Key OCID of the master encryption key used for secrets encryption
      FR5.10 - API user private key file location
      FR5.11 - API user private key fingerprint
      FR5.12 - Certificate file location for HTTPS connection verification. 
Optional.
 
FR6 - CONFIGURATION UPDATE
      No live configuration update will be provided for this keyring. Server 
restart will be necessary for changing the configuration.

FR7 - MIGRATION SUPPORT
      The plugin will support key migration service both as a source and a 
destination. The common keyring migration requirements apply.

FR8 - SINGLE SERVER PER OCI COMPARTMENT
      Dedicated OCI compartment is recommended for every instance of the MySQL 
server.

FR9 - PERFORMANCE SCHEMA
      performance_schema.keyring_keys

FR10 - UDF
       Fully support the keyring_udf keyring functions provided.

FR11 - SETUP HELPER
       Provide means to ease the creation of OCI compartments, vaults, obtaining 
their OCIDs and all the necessary requirements for this plugin.

FR12 - SUPPORTED KEY TYPES
       Provide support for all of the OCI Vault supported key types.
The plugin is to be implemented as a part of MySQL keyring infrastructure.
 
FR1 - KEY STORAGE
      All the keys are stored in OCI Vault secure secret storage system. None of 
the key information is to be permanently stored at MySQL server local storage. 
Only the list of the key IDs is stored in the 
server's 
memory cache, the key data is retrieved from the server every time it is 
requested. Random key generation is performed on the OCI Vault Key Management 
system, and the keys are subsequently stored on to the OCI 
Vault 
server.
 
FR2 - ENTITY MAPPING
      MySQL keys are uniquely identified by a set of information:
 
      1. User ID
      2. ID
      3. Type (cipher)
      4. Value
 
      For internal key users, the User ID is not available.
 
      Keys are uniquely mapped to OCI Vault secrets identifiers in the form of:
    
      https://vaults.us-ashburn-
1.oci.oraclecloud.com/20180608/secrets/ocid1.vaultsecret.oc1.iad.abaaaaaaa6iduka
awfj624xscisqs6eix24d7k4mvt42mmdegxs36ck46a7a
      
      where "https://vaults.us-ashburn-1.oci.oraclecloud.com" is path to the 
provided OCI Vault server, while the 
"ocid1.vaultsecret.oc1.iad.abaaaaaaa6idukaawfj624xscisqs6eix24d7k4mvt42mmdegxs36
ck46a7a" is the 
internal OCID 
of 
the key stored in the Vault. The 20180608 is the API version as specified in the 
REST API documentation at https://docs.cloud.oracle.com/en-
us/iaas/api/#/en/key/release/
  
      The MySQL plugin will authenticate against the OCI Vault using the SSH key 
and fingerprint scheme described at https://docs.cloud.oracle.com/en-
us/iaas/Content/API/Concepts/apisigningkey.htm .
 
      The MySQL keyring_OCI plugin won't store keys with missing ID or Value.

      The OCI Vault supports storing existing key data (in a secret) in any 
format. The maximum allowable size for a secret bundle is 25 KB. 
https://docs.cloud.oracle.com/en-
us/iaas/Content/KeyManagement/Tasks/managingsecrets.htm
      The OCI Vault supports generating key of type AES, of length 16, 24, or 32 
bytes. https://docs.cloud.oracle.com/en-
us/iaas/api/#/en/key/release/datatypes/KeyShape
      * Note: the documentation states that RSA keys are also supported, but the 
OCI KMS team confirmed that RSA keys will not be supported. Only AES keys can be 
generated. 

FR3 - CONSUMER SUPPORT
      MySQL services/features depending on keyring infrastructure should be able 
to access and use keyring services backed by plugin. These include:
 
      - InnoDB consumer
      - Audit log consumer
      - Keyring UDF consumer
      - Performance Schema keyring_keys table
      - binlog encryption
 
FR4 - OCI VAULT AUTHENTICATION
      MySQL server will be able to authenticate against OCI Vault tenancy by 
using an application user and private key that is registered to the API user in 
the OCI. As a precaution, 
the key's fingerprint is used 
too.
 
      The Keyring_OCI plugin will permanently store the SSH key and fingerprint 
in memory and use them to connect to the OCI Vault backend.
      The provided credentials are expected to remain valid indefinitely.
 
      The Application authentication scheme is documented here:
      https://docs.cloud.oracle.com/en-
us/iaas/Content/API/Concepts/signingrequests.htm
       
FR5 - PLUGIN CONFIGURATION
      Plugin will support configuration settings stored in persistent system 
variables.

      FR5.1 - User OCID of the OCI user. 
      It is necessary to have the Vault feature enabled for this user. 
      How to obtain the OCID: https://docs.cloud.oracle.com/en-
us/iaas/Content/API/Concepts/apisigningkey.htm#five
* Variable Name: keyring_oci_user
* Variable Type: string
* Description: The OCI User ocid used for connection to the cloud.
* Type: Global
* Settable at runtime: No
* Default: NONE


      FR5.2 - Tenancy OCID of the OCI tenancy.
      How to obtain the OCID: https://docs.cloud.oracle.com/en-
us/iaas/Content/General/Concepts/identifiers.htm#two
* Variable Name: keyring_oci_tenancy
* Variable Type: string
* Description: The OCI Tenancy ocid where the MySQL compartment resides.
* Type: Global
* Settable at runtime: No
* Default: NONE

      FR5.3 - Compartment OCID of the MySQL sub-compartment.
      Create a MySQL (sub)compartment in the tenancy. Obtain the compartment ID.
      Note: Make sure there are no existing Vault Keys or Vault Secrets in this 
compartment and this compartment is not used by other systems than MySQL server 
keyring plugin.
      How to manage compartments and obtain the OCID: 
https://docs.cloud.oracle.com/en-
us/iaas/Content/Identity/Tasks/managingcompartments.htm
* Variable Name: keyring_oci_compartment
* Variable Type: string
* Description: The OCI compartment ocid within the tenancy where the MySQL keys 
reside.
* Type: Global
* Settable at runtime: No
* Default: NONE

      FR5.4 - Vault OCID of the OCI Vault the MySQL will use.
      Create a new Virtual Vault in the MySQL compartment. Obtain the vault 
OCID.
      Note: You can reuse an existing vault that is in a parent compartment of 
the MySQL sub-compartment. Each compartment users can see and use only the keys 
in their respective 
compartment.
      Vault documentation how to create a virtual vault and obtain the vault 
OCID: https://docs.cloud.oracle.com/en-
us/iaas/Content/KeyManagement/Tasks/managingvaults.htm
* Variable Name: keyring_oci_virtual_vault
* Variable Type: string
* Description: The OCI Virtual Vault ocid used for all the encryption 
operations.
* Type: Global
* Settable at runtime: No
* Default: NONE

      FR5.5 - OCI encryption endpoint URLs
      In the Vault details page, copy the OCID, Cryptographic and Management 
endpoints.
      The Encryption endpoint is used for generating ciphertext for new keys. It 
has the form of <>-crypto.kms.us-ashburn-1.oraclecloud.com where xyz is the 
middle part of the Virtual Vault OCID.
* Variable Name: keyring_oci_encryption_endpoint
* Variable Type: string
* Description: The OCI encryption server endpoint.
* Type: Global
* Settable at runtime: No
* Default: NONE

      FR5.6 - OCI Key Management endpoint URL
      The Key Management endpoint is used for listing the existing keys. It has 
the form of <>-management.kms.us-ashburn-1.oraclecloud.com where xyz is the 
middle part of the Virtual Vault OCID.
* Variable Name: keyring_oci_management_endpoint
* Variable Type: string
* Description: The OCI key management server endpoint.
* Type: Global
* Settable at runtime: No
* Default: NONE

      FR5.7 - OCI Vault endpoint URL
      The Vaults endpoint is used for obtaining the value of secrets. It has the 
form of vaults.us-ashburn-1.oci.oraclecloud.com
* Variable Name: keyring_oci_vaults_endpoint
* Variable Type: string
* Description: The OCI vaults server endpoint.
* Type: Global
* Settable at runtime: No
* Default: NONE
* Example: vaults.us-ashburn-1.oci.oraclecloud.com

      FR5.8 - OCI Secrets endpoint URL
      The Secrets endpoint is used for listing, creating and retiring of 
secrets. It has the form of secrets.<>
* Variable Name: keyring_oci_secrets_endpoint
* Variable Type: string
* Description: The OCI vaults server endpoint.
* Type: Global
* Settable at runtime: No
* Default: NONE
* Example: "secrets." + 5.7 OCI Vault endpoint URL (e.g. "secrets.vaults.us-
ashburn-1.oci.oraclecloud.com")

      FR5.9 - Key OCID of the master encryption key used for secrets encryption
      The keyring plugin requires existing cryptographic key in the OCI 
compartment to be used for secret's encryption.
      How to create a key: https://docs.cloud.oracle.com/en-
us/iaas/Content/KeyManagement/Tasks/managingkeys.htm
      Provide a MySQL-specific name for the generated key, and do not use it for 
other purposes.
* Variable Name: keyring_oci_master_key
* Variable Type: string
* Description: The OCI encryption key ocid used for all the secret's encryption.
* Type: Global
* Settable at runtime: No
* Default: NONE

      FR5.10 - API user private key file location
      Upload the public key in the user control panel. 
      How to create and upload an API key: https://docs.cloud.oracle.com/en-
us/iaas/Content/API/Concepts/apisigningkey.htm#two
* Variable Name: keyring_oci_key_file
* Variable Type: string
* Description: The file path to the RSA private key used for OCI authentication.
* Type: Global
* Settable at runtime: No
* Default: NONE

      FR5.11 - API user public key fingerprint
      The key fingerprint can be obtained while creating the API keys, by 
running
      openssl rsa -pubout -outform DER -in ~/.oci/oci_api_key.pem | openssl md5 
-c
      or from the user's control panel: https://docs.cloud.oracle.com/en-
us/iaas/Content/API/Concepts/apisigningkey.htm#How3
* Variable Name: keyring_oci_key_fingerprint
* Variable Type: string
* Description: The RSA private key's fingerprint used for OCI authentication.
* Type: Global
* Settable at runtime: No
* Default: NONE

      FR5.12 - Path to Certificate Authority (CA) bundle.
      Optional path to a file holding one or more certificates to verify the 
peer with.
      If it is not provided, the default CA bundle installed in the system will 
be used for certificate verification.
      If it is set to 'disabled' (without quotes), CURLOPT_SSL_VERIFYPEER = 0 
will be used and no certification verification will be done.
      See https://curl.haxx.se/libcurl/c/CURLOPT_CAINFO.html for more details.
* Variable Name: keyring_oci_ca_certificate
* Variable Type: string
* Description: CA certificate bundle file path used for OCI's certificate 
verification.
* Type: Global
* Settable at runtime: No
* Default: "" - use default system certificate bundle for verification

 
FR6 - CONFIGURATION UPDATE 
      The users will not be able to update the plugin configuration of a running 
plugin. The settings can be changed only offline.
 
FR7 - MIGRATION SUPPORT
      Plugin will support key migration service as a source and a target.
      Plugin configuration in migration mode is configured as described in FR6.

FR8 - SINGLE SERVER PER OCI COMPARTMENT
      The plugin requires a dedicated OCI compartment within the user's tenancy 
for every instance of the MySQL server.
      Sharing the tenancy, while technically possible, can lead to concurrency 
issues. That's why only one server should be allowed to modify the keys and 
secrets. It is possible to 
have other MySQL servers' keyring_oci plugins 
to 
connect to the same tenancy and reuse the existing secrets.

FR9 - PERFORMANCE SCHEMA
      The performance_schema.keyring_keys table is used to query the list of the 
keys known to the active keyring plugins. For the backends that support it, 
displays the backend OCID 
of the stored keys. In the case of the keyring_oci this will be the secret ocid.

FR10 - UDF
       Fully support the keyring_udf keyring functions provided in a similar 
fashion to the other keyring plugins

FR11 - SETUP HELPER
       Setting up the keyring OCI plugin requires setting lots of variables - 
see FR.5. This limits the usability and creates a steep learning curve how to 
use this keyring plugin. 
And many of the OCI customers are not experts 
in cloud, they have 
just an username, password and entry point to the cloud.
       In order to help customers that don't have the whole environment set up, 
an interactive script will guide the user through the process of buying OCI 
subscription (if they don't 
have one yet), obtaining the datacenter 
region ("us-ashburn-1" 
for example), the provided user's (FR5.1) and tenancy (FR5.2) OCIDs, and then 
guide the user how to generate (or generate one for the user automatically) an 
RSA private key (FR5.10 
and FR5.11) Then the script will guide the user 
how to upload this 
key to the OCI. Unfortunately, until the private key is uploaded, no script can 
access the OCI so these steps must be done manually by the user.
       Having the datacenter region, the user and tenancy OCID, the script can 
connect to the cloud and generate the next items necessary for the keyring 
plugin:
       * compartment within the tenancy (FR5.3). It is not necessary but highly 
recommended that each activity in the tenancy happens in a separate compartment. 
This creates a level 
of abstraction and isolation.
       * Virtual Vault (FR5.4). Of course the user must have bought a 
subscription that allows Vault usage.
       * Having the vault OCID the script can obtain the suffix that is used for 
the key and encryption endpoints. "bbo6pux3aacuu" in my case. FR5.5 and FR5.6
       * The vault and secrets endpoint are just "vaults."/"secrets." sub-
domains of the datacenter region ("us-ashburn-1.oci.oraclecloud.com" for 
example). FR5.7 and FR5.8
       * Having the virtual vault created and setup, the script can use it to 
create the master encryption key FR5.9

FR12 - SUPPORTED KEY TYPES
       Provide support for all of the OCI Vault supported key types.
       Currently the OCI Vault supports generating key of type RSA and AES, of 
length 16, 24, or 32 bytes. https://docs.cloud.oracle.com/en-
us/iaas/api/#/en/key/release/datatypes/KeyShape


--- Key Management workflow with Oracle Cloud Vault ---

1. Master Encryption Key
All the secrets, stored in the OCI Vault, need to be encrypted with an OCI key.
Such key will be created by the install script and will be provided as an system 
variable to the keyring_oci plugin by FR5.9

2. Storing existing customer encryption key
MySQL keyring plugins support storing existing customer keys in the keyring.
Such encryption keys will be stored as a "secret" in the OCI Vault. Since the 
2020 March update, the OCI supports storing customer data in the Vault 
(previously known as KMS, Key 
Server)
POST https://vaults.us-ashburn-1.oci.oraclecloud.com/20180608/secrets
{
  "vaultId": 
"ocid1.vault.oc1.iad.bbo6pux3aacuu.abuwcljrjuje2hb2dyc2hkfol6rxukzjdy43bayoklz7m
wszjhstpcwzodhq",
  "compartmentId": 
"ocid1.compartment.oc1..aaaaaaaalas4ioucpe3o3a764cqgwmothsbxmuxkhwq2klz6vrilnx67
ftdq",
  "secretName": "<>",
  "keyId": 
"ocid1.key.oc1.iad.bbo6pux3aacuu.abuwcljsguqx3mhymmxs44l65dp5zc5seg6yctjwsiemg4q
zuwidlhn5s2ia",
  "secretContent": {
      "content": "aGVsbG8h",
      "contentType": "BASE64"
    },
  "freeformTags": {
      "user": "<>",
      "type": "<>",
      "source": "<>"
    }
}
Where the secretName is the ID of the existing key, and secretContent.Content is 
the Base64 value of the existing key's data.
Here, and in all subsequent requests, the keyId is the OCID of the Master 
Encryption Key above.

3. Creating a new encryption key
The OCI vault supports creating encryption key data:
POST https://bbo6pux3aacuu-crypto.kms.us-ashburn-
1.oraclecloud.com/20180608/generateDataEncryptionKey
{
  "includePlaintextKey": true,
  "keyId": 
"ocid1.key.oc1.iad.bbo6pux3aacuu.abuwcljsguqx3mhymmxs44l65dp5zc5seg6yctjwsiemg4q
zuwidlhn5s2ia",
  "keyShape": {
    "algorithm": "<>",
    "length": <>
  }
}
Where the server is the crypto endpoint of the OCI Vault
Here is a sample response from the Vault:
{
    "ciphertext": 
"IXcdmdJdcT3qAJVuL7MWawPKolMHa39uE6JEDYCmaofGQky3tBC5gwPzKk88PFdiD0oqGAMhAAAAAA=
=",
    "plaintext": "4ptRNgEEctssPnaf23guow==",
    "plaintextChecksum": "3212534271"
}
Where the plaintext contains the Base64 encoded encryption key's data.
This newly generated key must be stored in the Key Vault as described in the 
previous point.
Since it is a two-step process, the storing step may fail. As a result, the 
generated ciphertext will be lost. But no object is created on the OCI so 
nothing will be left dangling. 
Consecutive creations of a new key 
will generate new ciphertext.

4. Enumerating the encryption keys stored in the OCI Vault
GET https://vaults.us-ashburn-1.oci.oraclecloud.com/20180608/secrets?
compartmentId=ocid1.compartment.oc1..aaaaaaaap2jegooilnqf2wokm2n4bb2qvh4vcmuntwp
me4fsrbn77n5ft7bq
As described in point.2, the key ID is stored in the "secretName" field. The 
keyId is the internal OCID used for extracting the key data or deleting the key.
Here is an example response from the OCI:
  {
    "compartmentId": 
"ocid1.compartment.oc1..aaaaaaaap2jegooilnqf2wokm2n4bb2qvh4vcmuntwpme4fsrbn77n5f
t7bq",
    "definedTags": {
      "Oracle-Tags": {
        "CreatedBy": "iroylev",
        "CreatedOn": "2020-05-22T11:13:26.827Z"
      }
    },
    "description": null,
    "freeformTags": {
      "source": "generated",
      "type": "AES",
      "user": "root@localhost"
    },
    "keyId": 
"ocid1.key.oc1.iad.bbo6pux3aacuu.abuwcljsguqx3mhymmxs44l65dp5zc5seg6yctjwsiemg4q
zuwidlhn5s2ia",
    "id": 
"ocid1.vaultsecret.oc1.iad.amaaaaaas6idukaama27b4cvb3dxytlwi2l5shp5iu4p5o3wxjt63
rzlsudq",
    "lifecycleDetails": null,
    "lifecycleState": "ACTIVE",
    "rotationDetails": null,
    "secretName": "MyKey_256",
    "timeCreated": "2020-05-22T11:13:26.710Z",
    "timeOfCurrentVersionExpiry": null,
    "timeOfDeletion": null,
    "vaultId": 
"ocid1.vault.oc1.iad.bbo6pux3aacuu.abuwcljrjuje2hb2dyc2hkfol6rxukzjdy43bayoklz7m
wszjhstpcwzodhq"
  }
Here only keys that have "lifecycleState": "ACTIVE" are taken into account. 
Also, keys that are missing the freeformed tags "source", "type" and "user" are 
ignored. Only valid sources 
are generated and uploaded.

5. Extracting the encryption key from the OCI Vault
GET https://secrets.vaults.us-ashburn-
1.oci.oraclecloud.com/20190301/secretbundles/ocid1.vaultsecret.oc1.iad.amaaaaaas
6idukaawfj624xscisqs6eix24d7k4mvt42mmdegxs36ck46a7a
Here is an example response from the OCI:

{
  "secretId": 
"ocid1.vaultsecret.oc1.iad.amaaaaaas6idukaawfj624xscisqs6eix24d7k4mvt42mmdegxs36
ck46a7a",
  "timeCreated": "2020-04-15T14:29:59.452Z",
  "versionNumber": 1,
  "versionName": null,
  "secretBundleContent": {
    "contentType": "BASE64",
    "content": "d2hhdGV2ZXI="
  },
  "timeOfDeletion": null,
  "timeOfExpiry": null,
  "stages": [
    "CURRENT",
    "LATEST"
  ],
  "metadata": null
}
Where the secretBundleContent.content is the Base64 encoded encryption key's 
data to be used by the keyring plugin.

6. Deleting an existing key.
POST https://bbo6pux3aacuu-management.kms.us-ashburn-
1.oraclecloud.com/20180608/keys/ocid1.key.oc1.iad.bbo6pux3aacuu.abuwcljshlr7gsbl
jdvbn73dnazqxxpprzqo2ysut4xt5qqfmww7mjhmrexq/actions/scheduleDeletion
with an empty body {}
Here the internal OCID of the secret is used (not the customer-facing key id).
Note that the OCI Vault does not instantly delete any keys or secrets. It just 
schedules them for deletion, marking their "lifecycleState": "PENDING_DELETION". 
After a grace period 
(one month) they will be deleted. By 
then such keys cannot 
be used for encryption and their secret data cannot be obtained. The keyring_oci 
plugin ignores keys marked for pending deletion.


 

The OCI Vault provides HTTP API based on POST requests.
The following functionality will be used in the plugin:

   1. Query builder and parser.
   The REST API for accessing the OCI infrastructure is described in 
https://docs.cloud.oracle.com/iaas/Content/API/Concepts/usingapi.htm
   RapidJSON library will be used to prepare the request body and parse the 
request response.
   The Request class from request.h will handle the request preparation and 
execution. This class is template specialized for the GET and POST 
operations.
   The Request_headers class from request_headers.h will handle the request 
headers preparation and signature. This class is also template 
specialized for the GET and POST 
operations.

   2. HTTPS connector.
   The libcurl library will be used to access the OCI Vault's endpoint.

   3. API signing key.
   The OCI REST endpoints require the requests to be signed using an API signing 
key. The documentation about these keys is 
https://docs.cloud.oracle.com/en-
us/iaas/Content/API/Concepts/apisigningkey.htm

   4. Request Signature.
   The process of REST request signing is documented at 
https://docs.cloud.oracle.com/en-
us/iaas/Content/API/Concepts/signingrequests.htm
   The Signing_key class from signing_keys.h takes care of signing the REST 
authorization header.

   5. Common plugin interface
   The Keyring_OCI will use the keyring plugin interface that implements keyring 
service through the means of storing the keys in the OCI Vault.
   The Keyring_oci class from keyring_oci.h implements the keyring functionality 
for listing, reading, creating, uploading and deleting secrets.
   The oci.cc file implements the default keyring plugin interface to mysqld and 
handles plugin init/deinit.