Remote Storage Options

Oracle Cloud Infrastructure (OCI) Object Storage Options

• osBucketName: string (default: not set) - Name of the OCI Object Storage bucket to use. The bucket must already exist.
• osNamespace: string (default: not set) - Specifies the namespace where the bucket is located, if not given it will be obtained using the tenancy ID on the OCI configuration.
• ociConfigFile: string (default: not set) - Use the specified OCI configuration file instead of the one at the default location.
• ociProfile: string (default: not set) - Use the specified OCI profile instead of the default one.
• ociAuth: string (default: not set) - Use the specified authentication method when connecting to the OCI. Allowed values: api_key (used when not explicitly set), instance_principal, resource_principal, security_token.

Description

If the osBucketName option is used, the specified OCI bucket is used as the file storage. Connection is established using the local OCI configuration file. The directory structure is simulated within the object name.

The osNamespace, ociConfigFile, ociProfile and ociAuth options cannot be used if the osBucketName option is not set or set to an empty string.

The osNamespace option overrides the OCI namespace obtained based on the tenancy ID from the local OCI profile.

The ociAuth option allows to specify the authentication method used when connecting to the OCI:

• api_key - API Key-Based Authentication
• instance_principal - Instance Principal Authentication
• resource_principal - Resource Principal Authentication
• security_token - Session Token-Based Authentication

For more information please see: https://docs.oracle.com/en-us/iaas/Content/API/Concepts/sdk_authentication_methods.htm

OCI Object Storage Pre-Authenticated Requests (PARs)

When using a PAR to perform an operation, the OCI configuration is not needed to execute it (the osBucketName option should not be set).

When using PAR to a specific file, the generated PAR URL should be used as argument for an operation. The following is a file PAR to export a table to 'tab.tsv' file at the 'dump' directory of the 'test' bucket:

https://*.objectstorage.*.oci.customer-oci.com/p/*/n/*/b/test/o/dump/tab.tsv

When using a bucket PAR, the generated PAR URL should be used as argument for an operation. The following is a bucket PAR to create a dump at the root directory of the 'test' bucket:

https://*.objectstorage.*.oci.customer-oci.com/p/*/n/*/b/test/o/

When using a prefix PAR, argument should contain the PAR URL itself and the prefix used to generate it. The following is a prefix PAR to create a dump at the 'dump' directory of the 'test' bucket. The PAR was created using 'dump' as prefix:

https://*.objectstorage.*.oci.customer-oci.com/p/*/n/*/b/test/o/dump/

Note that the prefix PAR URL must end with a slash, because otherwise it will be treated as a PAR to specific file.

Reading from a PAR to specific file

The following access type is required to read from such PAR:

• Permit object reads

Reading from a bucket or a prefix PAR

The following access types are required to read from such PAR:

• Permit object reads
• Enable object listing

Writing to a PAR to specific file

The following access type is required to write to such PAR:

• Permit object writes

Writing to a bucket or a prefix PAR

The following access types are required to write to such PAR:

• Permit object reads and writes
• Enable object listing

Note that read access is required in this case, because otherwise it is not possible to list the objects.

AWS S3 Object Storage Options

• s3BucketName: string (default: not set) - Name of the AWS S3 bucket to use. The bucket must already exist.
• s3CredentialsFile: string (default: not set) - Use the specified AWS credentials file.
• s3ConfigFile: string (default: not set) - Use the specified AWS config file.
• s3Profile: string (default: not set) - Use the specified AWS profile.
• s3Region: string (default: not set) - Use the specified AWS region.
• s3EndpointOverride: string (default: not set) - Use the specified AWS S3 API endpoint instead of the default one.

Description

If the s3BucketName option is used, the specified AWS S3 bucket is used as the file storage. Connection is established using default local AWS configuration paths and profiles, unless overridden. The directory structure is simulated within the object name.

The s3CredentialsFile, s3ConfigFile, s3Profile, s3Region and s3EndpointOverride options cannot be used if the s3BucketName option is not set or set to an empty string.

All failed connections to AWS S3 are retried three times, with a 1 second delay between retries. If a failure occurs 10 minutes after the connection was created, the delay is changed to an exponential back-off strategy:

• first delay: 3-6 seconds
• second delay: 18-36 seconds
• third delay: 40-80 seconds

Handling of the AWS settings

The AWS options are evaluated in the order of precedence, the first available value is used.

1. Name of the AWS profile:
   • the s3Profile option
   • the AWS_PROFILE environment variable
   • the AWS_DEFAULT_PROFILE environment variable
   • the default value of default
2. Location of the credentials file:
   • the s3CredentialsFile option
   • the AWS_SHARED_CREDENTIALS_FILE environment variable
   • the default value of ~/.aws/credentials
3. Location of the config file:
   • the s3ConfigFile option
   • the AWS_CONFIG_FILE environment variable
   • the default value of ~/.aws/config
4. Name of the AWS region:
   • the s3Region option
   • the AWS_REGION environment variable
   • the AWS_DEFAULT_REGION environment variable
   • the region setting from the config file for the specified profile
   • the default value of us-east-1
5. URI of AWS S3 API endpoint
   • the s3EndpointOverride option
   • the default value of https://<s3BucketName>.s3.<region>.amazonaws.com

The AWS credentials are fetched from the following providers, in the order of precedence:

1. Environment variables:
   • AWS_ACCESS_KEY_ID
   • AWS_SECRET_ACCESS_KEY
   • AWS_SESSION_TOKEN
2. Assuming a role
3. Settings from the credentials file for the specified profile:
   • aws_access_key_id
   • aws_secret_access_key
   • aws_session_token
4. Process specified by the credential_process setting from the config file for the specified profile
5. Settings from the config file for the specified profile:
   • aws_access_key_id
   • aws_secret_access_key
   • aws_session_token
6. Amazon Elastic Container Service (Amazon ECS) credentials
7. Amazon Instance Metadata Service (Amazon IMDS) credentials

The items specified above correspond to the following credentials:

• the AWS access key
• the secret key associated with the AWS access key
• the AWS session token for the temporary security credentials

Role is assumed using the following settings from the AWS config file:

• credential_source
• duration_seconds
• external_id
• role_arn
• role_session_name
• source_profile

The multi-factor authentication is not supported. For more information on assuming a role, please consult the AWS documentation.

The process/command line specified by the credential_process setting must write a JSON object to the standard output in the following form:

{
  "Version": 1,
  "AccessKeyId": "AWS access key",
  "SecretAccessKey": "secret key associated with the AWS access key",
  "SessionToken": "temporary AWS session token, optional",
  "Expiration": "RFC3339 timestamp, optional"
}

The Amazon ECS credentials are fetched from a URI specified by an environment variable AWS_CONTAINER_CREDENTIALS_RELATIVE_URI (its value is appended to 'http://169.254.170.2'). If this environment variable is not set, the value of AWS_CONTAINER_CREDENTIALS_FULL_URI environment variable is used instead. If neither of these environment variables are set, ECS credentials are not used.

The request may optionally be sent with an 'Authorization' header. If the AWS_CONTAINER_AUTHORIZATION_TOKEN_FILE environment variable is set, its value should specify an absolute file path to a file that contains the authorization token. Alternatively, the AWS_CONTAINER_AUTHORIZATION_TOKEN environment variable should be used to explicilty specify that authorization token. If neither of these environment variables are set, the 'Authorization' header is not sent with the request.

The reply is expected to be a JSON object in the following form:

{
  "AccessKeyId": "AWS access key",
  "SecretAccessKey": "secret key associated with the AWS access key",
  "Token": "temporary AWS session token",
  "Expiration": "RFC3339 timestamp"
}

The Amazon IMDS credential provider is configured using the following environment variables:

• AWS_EC2_METADATA_DISABLED
• AWS_EC2_METADATA_SERVICE_ENDPOINT
• AWS_EC2_METADATA_SERVICE_ENDPOINT_MODE
• AWS_EC2_METADATA_V1_DISABLED
• AWS_METADATA_SERVICE_TIMEOUT
• AWS_METADATA_SERVICE_NUM_ATTEMPTS

and the following settings from the AWS config file:

• ec2_metadata_service_endpoint
• ec2_metadata_service_endpoint_mode
• ec2_metadata_v1_disabled
• metadata_service_timeout
• metadata_service_num_attempts

For more information on IMDS, please consult the AWS documentation.

The Expiration value, if given, specifies when the credentials are going to expire, they will be automatically refreshed before this happens.

The following credential handling rules apply:

• If the s3Profile option is set to a non-empty string, the environment variables are not used as a potential credential provider.
• If either an access key or a secret key is available in a potential credential provider, it is selected as the credential provider.
• If either the access key or the secret key is missing in the selected credential provider, an exception is thrown.
• If the session token is missing in the selected credential provider, or if it is set to an empty string, it is not used to authenticate the user.

Azure Blob Storage Options

• azureContainerName: string (default: not set) - Name of the Azure container to use. The container must already exist.
• azureConfigFile: string (default: not set) - Use the specified Azure configuration file instead of the one at the default location.
• azureStorageAccount: string (default: not set) - The account to be used for the operation.
• azureStorageSasToken: string (default: not set) - Azure Shared Access Signature (SAS) token, to be used for the authentication of the operation, instead of a key.

Description

If the azureContainerName option is used, the specified Azure container is used as the file storage. Connection is established using the configuration at the local Azure configuration file. The directory structure is simulated within the blob name.

The azureConfigFile, azureStorageAccount and azureStorageSasToken options cannot be used if the azureContainerName option is not set or set to an empty string.

Handling of the Azure settings

The following settings are read from the storage section in the config file:

• connection_string
• account
• key
• sas_token

Additionally, the connection options may be defined using the standard Azure environment variables:

• AZURE_STORAGE_CONNECTION_STRING
• AZURE_STORAGE_ACCOUNT
• AZURE_STORAGE_KEY
• AZURE_STORAGE_SAS_TOKEN

The Azure configuration values are evaluated in the following precedence:

• options parameter
• environment variables
• configuration file

If a connection string is defined either case in the environment variable or the configuration option, the individual configuration values for account and key will be ignored.

If a SAS Token is defined, it will be used for the authorization (ignoring any defined account key).

The default Azure Blob Endpoint to be used in the operations is defined by:

https://<account>.blob.core.windows.net

unless a different endpoint is defined in the connection string.