8.4.4.10 Using the HashiCorp Vault Keyring Plugin

Note

The keyring_hashicorp plugin is an extension included in MySQL Enterprise Edition, a commercial product. To learn more about commercial products, see https://www.mysql.com/products/.

The keyring_hashicorp keyring plugin communicates with HashiCorp Vault for back end storage. The plugin supports HashiCorp Vault AppRole authentication. No key information is permanently stored in MySQL server local storage. (An optional in-memory key cache may be used as intermediate storage.) Random key generation is performed on the MySQL server side, with the keys subsequently stored to Hashicorp Vault.

The keyring_hashicorp plugin supports the functions that comprise the standard MySQL Keyring service interface. Keyring operations performed by those functions are accessible at two levels:

• SQL interface: In SQL statements, call the functions described in Section 8.4.4.15, “General-Purpose Keyring Key-Management Functions”.

• C interface: In C-language code, call the keyring service functions described in Section 7.6.9.2, “The Keyring Service”.

Example (using the SQL interface):

SELECT keyring_key_generate('MyKey', 'AES', 32);
SELECT keyring_key_remove('MyKey');

For information about the characteristics of key values permitted by keyring_hashicorp, see Section 8.4.4.13, “Supported Keyring Key Types and Lengths”.

To install keyring_hashicorp, use the general instructions found in Section 8.4.4.3, “Keyring Plugin Installation”, together with the configuration information specific to keyring_hashicorp found here. Plugin-specific configuration includes preparation of the certificate and key files needed for connecting to HashiCorp Vault, as well as configuring HashiCorp Vault itself. The following sections provide the necessary instructions.

• Certificate and Key Preparation

• HashiCorp Vault Setup

• keyring_hashicorp Configuration

Certificate and Key Preparation

The keyring_hashicorp plugin requires a secure connection to the HashiCorp Vault server, employing the HTTPS protocol. A typical setup includes a set of certificate and key files:

• company.crt: A custom CA certificate belonging to the organization. This file is used both by HashiCorp Vault server and the keyring_hashicorp plugin.

• vault.key: The private key of the HashiCorp Vault server instance. This file is used by HashiCorp Vault server.

• vault.crt: The certificate of the HashiCorp Vault server instance. This file must be signed by the organization CA certificate.

The following instructions describe how to create the certificate and key files using OpenSSL. (If you already have those files, proceed to HashiCorp Vault Setup.) The instructions as shown apply to Linux platforms and may require adjustment for other platforms.

Important

Certificates generated by these instructions are self-signed, which may not be very secure. After you gain experience using such files, consider obtaining certificate/key material from a registered certificate authority.

1. Prepare the company and HashiCorp Vault server keys.

   Use the following commands to generate the key files:

   openssl genrsa -aes256 -out company.key 4096
   openssl genrsa -aes256 -out vault.key 2048

   The commands produce files holding the company private key (company.key) and the Vault server private key (vault.key). The keys are randomly generated RSA keys of 4,096 and 2,048 bits, respectively.

   Each command prompts for a password. For testing purposes, the password is not required. To disable it, omit the -aes256 argument.

   The key files hold sensitive information and should be stored in a secure location. The password (also sensitive) is required later, so write it down and store it in a secure location.

   (Optional) To check key file content and validity, use the following commands:

   openssl rsa -in company.key -check
   openssl rsa -in vault.key -check

2. Create the company CA certificate.

   Use the following command to create a company CA certificate file named company.crt that is valid for 365 days (enter the command on a single line):

   openssl req -x509 -new -nodes -key company.key
     -sha256 -days 365 -out company.crt

   If you used the -aes256 argument to perform key encryption during key generation, you are prompted for the company key password during CA certificate creation. You are also prompted for information about the certificate holder (that is, you or your company), as shown here:

   Country Name (2 letter code) [AU]:
   State or Province Name (full name) [Some-State]:
   Locality Name (eg, city) []:
   Organization Name (eg, company) [Internet Widgits Pty Ltd]:
   Organizational Unit Name (eg, section) []:
   Common Name (e.g. server FQDN or YOUR name) []:
   Email Address []:

   Answer the prompts with appropriate values.

3. Create a certificate signing request.

   To create a HashiCorp Vault server certificate, a Certificate Signing Request (CSR) must be prepared for the newly created server key. Create a configuration file named request.conf containing the following lines. If the HashiCorp Vault server does not run on the local host, substitute appropriate CN and IP values, and make any other changes required.

   [req]
   distinguished_name = vault
   x509_entensions = v3_req
   prompt = no
   
   [vault]
   C = US
   ST = CA
   L = RWC
   O = Company
   CN = 127.0.0.1
   
   [v3_req]
   subjectAltName = @alternatives
   authorityKeyIdentifier = keyid,issuer
   basicConstraints = CA:TRUE
   
   [alternatives]
   IP = 127.0.0.1

   Use this command to create the signing request:

   openssl req -new -key vault.key -config request.conf -out request.csr

   The output file (request.csr) is an intermediate file that serves as input for creation of the server certificate.

4. Create the HashiCorp Vault server certificate.

   Sign the combined information from the HashiCorp Vault server key (vault.key) and the CSR (request.csr) with the company certificate (company.crt) to create the HashiCorp Vault server certificate (vault.crt). Use the following command to do this (enter the command on a single line):

   openssl x509 -req -in request.csr
     -CA company.crt -CAkey company.key -CAcreateserial
     -out vault.crt -days 365 -sha256

   To make the vault.crt server certificate useful, append the contents of the company.crt company certificate to it. This is required so that the company certificate is delivered along with the server certificate in requests.

   cat company.crt >> vault.crt

   If you display the contents of the vault.crt file, it should look like this:

   -----BEGIN CERTIFICATE-----
   ... content of HashiCorp Vault server certificate ...
   -----END CERTIFICATE-----
   -----BEGIN CERTIFICATE-----
   ... content of company certificate ...
   -----END CERTIFICATE-----

HashiCorp Vault Setup

The following instructions describe how to create a HashiCorp Vault setup that facilitates testing the keyring_hashicorp plugin.

Important

A test setup is similar to a production setup, but production use of HashiCorp Vault entails additional security considerations such as use of non-self-signed certificates and storing the company certificate in the system trust store. You must implement whatever additional security steps are needed to satisfy your operational requirements.

These instructions assume availability of the certificate and key files created in Certificate and Key Preparation. See that section if you do not have those files.

1. Fetch the HashiCorp Vault binary.

   Download the HashiCorp Vault binary appropriate for your platform from https://www.vaultproject.io/downloads.html.

   Extract the content of the archive to produce the executable vault command, which is used to perform HashiCorp Vault operations. If necessary, add the directory where you install the command to the system path.

   (Optional) HashiCorp Vault supports autocomplete options that make it easier to use. For more information, see https://learn.hashicorp.com/vault/getting-started/install#command-completion.

2. Create the HashiCorp Vault server configuration file.

   Prepare a configuration file named config.hcl with the following content. For the tls_cert_file, tls_key_file, and path values, substitute path names appropriate for your system.

   listener "tcp" {
     address="127.0.0.1:8200"
     tls_cert_file="/home/username/certificates/vault.crt"
     tls_key_file="/home/username/certificates/vault.key"
   }
   
   storage "file" {
     path = "/home/username/vaultstorage/storage"
   }
   
   ui = true

3. Start the HashiCorp Vault server.

   To start the Vault server, use the following command, where the -config option specifies the path to the configuration file just created:

   vault server -config=config.hcl

   During this step, you may be prompted for a password for the Vault server private key stored in the vault.key file.

   The server should start, displaying some information on the console (IP, port, and so forth).

   So that you can enter the remaining commands, put the vault server command in the background or open another terminal before continuing.

4. Initialize the HashiCorp Vault server.

   Note

   The operations described in this step are required only when starting Vault the first time, to obtain the unseal key and root token. Subsequent Vault instance restarts require only unsealing using the unseal key.

   Issue the following commands (assuming Bourne shell syntax):

   export VAULT_SKIP_VERIFY=1
   vault operator init -n 1 -t 1

   The first command enables the vault command to temporarily ignore the fact that no company certificate has been added to the system trust store. It compensates for the fact that our self-signed CA is not added to that store. (For production use, such a certificate should be added.)

   The second command creates a single unseal key with a requirement for a single unseal key to be present for unsealing. (For production use, an instance would have multiple unseal keys with up to that many keys required to be entered to unseal it. The unseal keys should be delivered to key custodians within the company. Use of a single key might be considered a security issue because that permits the vault to be unsealed by a single key custodian.)

   Vault should reply with information about the unseal key and root token, plus some additional text (the actual unseal key and root token values differ from those shown here):

   ...
   Unseal Key 1: I2xwcFQc892O0Nt2pBiRNlnkHzTUrWS+JybL39BjcOE=
   Initial Root Token: s.vTvXeo3tPEYehfcd9WH7oUKz
   ...

   Store the unseal key and root token in a secure location.

5. Unseal the HashiCorp Vault server.

   Use this command to unseal the Vault server:

   vault operator unseal

   When prompted to enter the unseal key, use the key obtained previously during Vault initialization.

   Vault should produce output indicating that setup is complete and the vault is unsealed.

6. Log in to the HashiCorp Vault server and verify its status.

   Prepare the environment variables required for logging in as root:

   vault login s.vTvXeo3tPEYehfcd9WH7oUKz

   For the token value in that command, substitute the content of the root token obtained previously during Vault initialization.

   Verify the Vault server status:

   vault status

   The output should contain these lines (among others):

   ...
   Initialized     true
   Sealed          false
   ...

7. Set up HashiCorp Vault authentication and storage.

   Note

   The operations described in this step are needed only the first time the Vault instance is run. They need not be repeated afterward.

   Enable the AppRole authentication method and verify that it is in the authentication method list:

   vault auth enable approle
   vault auth list

   Enable the Vault KeyValue storage engine:

   vault secrets enable -version=1 kv

   Create and set up a role for use with the keyring_hashicorp plugin (enter the command on a single line):

   vault write auth/approle/role/mysql token_num_uses=0
     token_ttl=20m token_max_ttl=30m secret_id_num_uses=0

8. Add an AppRole security policy.

   Note

   The operations described in this step are needed only the first time the Vault instance is run. They need not be repeated afterward.

   Prepare a policy that to permit the previously created role to access appropriate secrets. Create a new file named mysql.hcl with the following content:

   path "kv/mysql/*" {
     capabilities = ["create", "read", "update", "delete", "list"]
   }

   Note

   kv/mysql/ in this example may need adjustment per your local installation policies and security requirements. If so, make the same adjustment wherever else kv/mysql/ appears in these instructions.

   Import the policy file to the Vault server to create a policy named mysql-policy, then assign the policy to the new role:

   vault policy write mysql-policy mysql.hcl
   vault write auth/approle/role/mysql policies=mysql-policy

   Obtain the ID of the newly created role and store it in a secure location:

   vault read auth/approle/role/mysql/role-id

   Generate a secret ID for the role and store it in a secure location:

   vault write -f auth/approle/role/mysql/secret-id

   After these AppRole role ID and secret ID credentials are generated, they are expected to remain valid indefinitely. They need not be generated again and the keyring_hashicorp plugin can be configured with them for use on an ongoing basis. For more information about AuthRole authentication, visit https://www.vaultproject.io/docs/auth/approle.html.

keyring_hashicorp Configuration

The plugin library file contains the keyring_hashicorp plugin and a loadable function, keyring_hashicorp_update_config(). When the plugin initializes and terminates, it automatically loads and unloads the function. There is no need to load and unload the function manually.

The keyring_hashicorp plugin supports the configuration parameters shown in the following table. To specify these parameters, assign values to the corresponding system variables.

Configuration Parameter	System Variable	Mandatory
HashiCorp Server URL	keyring_hashicorp_server_url	No
AppRole role ID	keyring_hashicorp_role_id	Yes
AppRole secret ID	keyring_hashicorp_secret_id	Yes
Store path	keyring_hashicorp_store_path	Yes
Authorization Path	keyring_hashicorp_auth_path	No
CA certificate file path	keyring_hashicorp_ca_path	No
Cache control	keyring_hashicorp_caching	No

To be usable during the server startup process, keyring_hashicorp must be loaded using the --early-plugin-load option. As indicated by the preceding table, several plugin-related system variables are mandatory and must also be set. For example, use these lines in the server my.cnf file, adjusting the .so suffix and file locations for your platform as necessary:

[mysqld]
early-plugin-load=keyring_hashicorp.so
keyring_hashicorp_role_id='ee3b495c-d0c9-11e9-8881-8444c71c32aa'
keyring_hashicorp_secret_id='0512af29-d0ca-11e9-95ee-0010e00dd718'
keyring_hashicorp_store_path='/v1/kv/mysql'
keyring_hashicorp_auth_path='/v1/auth/approle/login'

Note

Per the HashiCorp documentation, all API routes are prefixed with a protocol version (which you can see in the preceding example as /v1/ in the keyring_hashicorp_store_path and keyring_hashicorp_auth_path values). If HashiCorp develops new protocol versions, it may be necessary to change /v1/ to something else in your configuration.

MySQL Server authenticates against HashiCorp Vault using AppRole authentication. Successful authentication requires that two secrets be provided to Vault, a role ID and a secret ID, which are similar in concept to user name and password. The role ID and secret ID values to use are those obtained during the HashiCorp Vault setup procedure performed previously. To specify the two IDs, assign their respective values to the keyring_hashicorp_role_id and keyring_hashicorp_secret_id system variables. The setup procedure also results in a store path of /v1/kv/mysql, which is the value to assign to keyring_hashicorp_commit_store_path.

At plugin initialization time, keyring_hashicorp attempts to connect to the HashiCorp Vault server using the configuration values. If the connection is successful, the plugin stores the values in corresponding system variables that have _commit_ in their name. For example, upon successful connection, the plugin stores the values of keyring_hashicorp_role_id and keyring_hashicorp_store_path in keyring_hashicorp_commit_role_id and keyring_hashicorp_commit_store_path.

Reconfiguration at runtime can be performed with the assistance of the keyring_hashicorp_update_config() function:

1. Use SET statements to assign the desired new values to the configuration system variables shown in the preceding table. These assignments in themselves have no effect on ongoing plugin operation.

2. Invoke keyring_hashicorp_update_config() to cause the plugin to reconfigure and reconnect to the HashiCorp Vault server using the new variable values.

3. If the connection is successful, the plugin stores the updated configuration values in corresponding system variables that have _commit_ in their name.

For example, if you have reconfigured HashiCorp Vault to listen on port 8201 rather than the default 8200, reconfigure keyring_hashicorp like this:

mysql> SET GLOBAL keyring_hashicorp_server_url = 'https://127.0.0.1:8201';
Query OK, 0 rows affected (0.00 sec)

mysql> SELECT keyring_hashicorp_update_config();
+--------------------------------------+
| keyring_hashicorp_update_config()    |
+--------------------------------------+
| Configuration update was successful. |
+--------------------------------------+
1 row in set (0.03 sec)

If the plugin is not able to connect to HashiCorp Vault during initialization or reconfiguration and there was no existing connection, the _commit_ system variables are set to 'Not committed' for string-valued variables, and OFF for Boolean-valued variables. If the plugin is not able to connect but there was an existing connection, that connection remains active and the _commit_ variables reflect the values used for it.

Note

If you do not set the mandatory system variables at server startup, or if some other plugin initialization error occurs, initialization fails. In this case, you can use the runtime reconfiguration procedure to initialize the plugin without restarting the server.

For additional information about the keyring_hashicorp plugin-specific system variables and function, see Section 8.4.4.19, “Keyring System Variables”, and Section 8.4.4.16, “Plugin-Specific Keyring Key-Management Functions”.