4.7.2 S3-compatible Storage

MySQL Shell S3 support has been tested against the following S3-compatible storage services:

MySQL Shell supports configuring AWS credentials in environment variables and in configuration files.

Configuration Parameter Precedence

Configuration parameters are used in order of precedence:

  1. Option. For example, s3Profile takes precedence over the environment variable AWS_PROFILE, and the default profile in the configuration file.

  2. Environment variable. For example, the environment variable AWS_SHARED_CREDENTIALS_FILE takes precedence over the default location of the credentials file, ~/.aws/credentials.

  3. DEFAULT environment variable, if it exists. For example, the environment variable AWS_REGION and the environment variable AWS_DEFAULT_REGION.

  4. Parameter defined in a configuration file.

  5. AWS default values.

For example, for the AWS region, in order of precedence:

  1. s3Region option.

  2. AWS_REGION environment variable.

  3. AWS_DEFAULT_REGION environment variable.

  4. Region defined in the configuration file.

  5. The default value of us-east-1.

AWS credentials are read in the following order of precedence:

  1. Environment variables, if the s3Profile option is not defined.

  2. Assuming a role.

  3. Credentials file, for the specified profile.

  4. credential_process, as defined in the config file. This configuration type contains an external command which retrieves or generates AWS authentication credentials and writes them to stdout.

  5. Config file, for the specified profile.

  6. ECS credentials.

  7. IMDS credentials.

Environment Variables

For information on AWS environment variables, see the following:

You can define the following AWS S3-specific environment variables:

  • AWS_PROFILE: Specifies the name of the profile to use. This can be the name of a profile in a credentials or config file, or the value default to use the default profile. This environment variable overrides the [default] profile named in the configuration file. You can override this environment variable with the --s3Profile option.

  • AWS_SHARED_CREDENTIALS_FILE: The location of the file used to store access keys. Such as ~/.aws/credentials.

  • AWS_CONFIG_FILE: The location of the file used to store configuration profiles. Such as ~/.aws/config.

  • AWS_REGION: Specifies the AWS Region to send the request to. This value overrides the AWS_DEFAULT_REGION environment variable and the profile defined in the configuration file.

  • AWS_DEFAULT_REGION: Specifies the AWS Region to send the request to. This value is overridden by the --s3Region option and the AWS_REGION environment variable, if specified.

  • AWS_ACCESS_KEY_ID: Specifies an AWS access key associated with an IAM user or role.

  • AWS_SECRET_ACCESS_KEY: Specifies the secret key associated with the access key. This variable overrides the aws_secret_access_key defined in the profile.

  • AWS_SESSION_TOKEN: Specifies the session token value required if you are using temporary security credentials. This variable overrides the aws_session_token defined in the profile.

  • AWS_CONTAINER_CREDENTIALS_RELATIVE_URI: (Amazon ECS) value specified is appended to http://169.254.170.2. The resulting URI is used to fetch the credentials. If not set, or is empty, and AWS_CONTAINER_CREDENTIALS_FULL_URI is defined, AWS_CONTAINER_CREDENTIALS_FULL_URI is used instead.

  • AWS_CONTAINER_CREDENTIALS_FULL_URI: (Amazon ECS) the defined host must contain one of the following:

    • 169.254.170.2

    • 169.254.170.23

    • localhost

    • fd00:ec2::23

    • A loopback address

    If this variable is set to a value other than those listed, an error is returned.

  • AWS_CONTAINER_AUTHORIZATION_TOKEN_FILE: (Amazon ECS) defines t an absolute file path to a file that contains the authorization token in plain text.

  • AWS_CONTAINER_AUTHORIZATION_TOKEN: (Amazon ECS) defines the authorization header of the HTTP request to fetch the credentials.

  • AWS_EC2_METADATA_DISABLED: (Amazon IMDS) if enabled, true, IMDS credentials are not used.

  • AWS_EC2_METADATA_V1_DISABLED: (Amazon IMDS) if enabled, true, IMDSv1 is used if IMDSv2 is unavailable.

  • AWS_EC2_METADATA_SERVICE_ENDPOINT: (Amazon IMDS) specifies the address of the IMDS endpoint.

    The hostname component of the endpoint is validated according to the following criteria:

    • Maximum of 253 characters

    • Maximum of 63 characters per label.

    • Permitted characters: a-z, A-Z, 0-9, and -.

    • A label can not begin or end with a hyphen (-).

    • Labels are concatenated with periods.

  • AWS_EC2_METADATA_SERVICE_ENDPOINT_MODE: (Amazon IMDS) one of the following values:

    • IPv4: (default) the IMDS endpoint address is set to http://169.254.169.254/.

    • IPv6: the IMDS endpoint address is set to http://[fd00:ec2::254]/

  • AWS_METADATA_SERVICE_TIMEOUT: (Amazon IMDS) specifies the number of seconds to wait for a successful connection to the IMDS endpoint before timing out. Default value is 1.

  • AWS_METADATA_SERVICE_NUM_ATTEMPTS: (Amazon IMDS) specifies the number of attempts to retrieve data from IMDS before failing and returning an error. Default value is 1.

Configuration Files

MySQL Shell requires the following parameters in one or more configuration files:

  • aws_access_key_id: specifies the access key associated with the user.

  • aws_secret_access_key: specifies the secret key associated with the access key.

  • aws_session_token: specifies the session token required if you are using temporary security credentials. If not present, it is not used to authenticate the user.

  • region: specifies the cloud service region. If not present, the default value of us-east-1 is used.

  • credential_process: This configuration type contains an external command which retrieves or generates AWS authentication credentials and writes them to stdout.

    Note

    credential_process can only be defined in the config file.

    For more information, see the following:

  • role_arn: ARN of an IAM role, with the syntax: arn:aws:iam::account-id:role/role-name.

    If specified, at least one of the following must also be defined:

    • credential_source

    • source_profile

    See Assume role credential provider.

  • role_session_name: Defines a name to attach to the role session.

  • duration_seconds: Defines the maximum duration of the role session, in seconds.

  • external_id: Specifies a unique identifier used by third parties to assume a role.

  • credential_source: specifies the source of the credentials used to assume a role. Use one of the following:

    • Environment: Assumes the role using the credentials defined in the environment variables AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, and (optional) AWS_SESSION_TOKEN.

    • EcsContainer: Assumes the role using the credentials defined in Amazon ECS.

    • Ec2InstanceMetadata: Assumes the role using the credentials defined in Amazon IMDS.

  • source_profile: specifies the name of a profile which contains the credentials required to assume a role.

  • ec2_metadata_v1_disabled: (Amazon IMDS) if enabled, true, IMDSv1 is used if IMDSv2 is unavailable.

  • ec2_metadata_service_endpoint: (Amazon IMDS) specifies the address of the IMDS endpoint.

    The endpoint is validated according to the following criteria:

    • Maximum of 253 characters

    • Maximum of 63 characters per label.

    • Permitted characters: a-z, A-Z, 0-9, and -.

    • A label can not begin or end with a hyphen (-).

    • Labels are concatenated with periods.

  • ec2_metadata_service_endpoint_mode: (Amazon IMDS) one of the following values:

    • IPv4: (default) the IMDS endpoint address is set to http://169.254.169.254/.

    • IPv6: the IMDS endpoint address is set to http://[fd00:ec2::254]/

  • metadata_service_timeout: (Amazon IMDS) specifies the number of seconds to wait for a successful connection to the IMDS endpoint before timing out. Default value is 1.

  • metadata_service_num_attempts: (Amazon IMDS) specifies the number of attempts to retrieve data from IMDS before failing and returning an error. Default value is 1.

If you have installed and configured the AWS CLI, MySQL Shell reads the connection parameters from the default location, .aws/, automatically. To use an alternate configuration, do so from the command line, using the relevant override options.

If you are using Oracle Cloud Infrastructure's S3 compatibility API, see Creating a Customer Secret Key for information on creating the credentials required.

Note

By default, the AWS CLI creates two configuration files, config, which stores parameters such as region and output format, and credentials, which stores access keys and session tokens. It is also possible to place all configuration parameters in a single file, config. For more information, see AWS Configuration and authentication settings . However, if you have defined access key and secret access keys in both files, those defined in the credentials file take precedence.

The following example shows a default pair of AWS CLI configuration files: 

        /home/.aws/credentials
        
        [default]
        aws_access_key_id = AKIA6AV......
        aws_secret_access_key = XHRY579I.....
        
        /home/.aws/config
        
        [default]
        region = us-west-1
        output = json

Connection Retry Strategy

All failed connections to AWS S3 are retried three times, with a 1 second delay between retries.

If a failure occurs 10 minutes after the connection was created, the delay is changed to an exponential back-off strategy:

  • First delay: 3-6 seconds

  • Second delay: 18-36 seconds

  • Third delay: 40-80 seconds


PREV   HOME   UP   NEXT
Download this Manual
PDF (US Ltr) - 2.5Mb
PDF (A4) - 2.5Mb