MySQL Shell 8.0  /  MySQL Shell Utilities  /  Dump Loading Utility

11.6 Dump Loading Utility

MySQL Shell's dump loading utility util.loadDump(), introduced in MySQL Shell 8.0.21, supports the import into a MySQL Database Service DB System (a MySQL DB System, for short) or a MySQL Server instance of schemas or tables dumped using MySQL Shell's Section 11.5, “Instance Dump Utility, Schema Dump Utility, and Table Dump Utility”. The dump loading utility provides data streaming from remote storage, parallel loading of tables or table chunks, progress state tracking, resume and reset capability, and the option of concurrent loading while the dump is still taking place. To get the best functionality, always use the most recent version available of MySQL Shell's dump and dump loading utilities.

For import into a MySQL DB System, MySQL Shell must be installed on an Oracle Cloud Infrastructure Compute instance that has access to the MySQL DB System. If the dump files are in an Oracle Cloud Infrastructure Object Storage bucket, you can access the Object Storage bucket from the Compute instance. If the dump files are on your local system, you need to transfer them to the Oracle Cloud Infrastructure Compute instance using the copy utility of your choice, depending on the operating system you chose for your Compute instance. Ensure the dump was created with the ocimds option set to true in MySQL Shell's instance dump utility or schema dump utility, for compatibility with MySQL Database Service. MySQL Shell's table dump utility does not use this option.

Note

  1. MySQL 5.7 or later is required for the destination MySQL instance.

  2. The dump loading utility uses the LOAD DATA LOCAL INFILE statement, so the global setting of the local_infile system variable on the target MySQL instance must be ON for the duration of the import. By default, this system variable is set to ON in a standard MySQL DB System configuration.

  3. The LOAD DATA LOCAL INFILE statement uses nonrestrictive data interpretation, which turns errors into warnings and continues with the load operation. This process can include assigning default values and implicit default values to fields, and converting invalid values to the closest valid value for the column data type. For details of the statement's behavior, see LOAD DATA.

  4. On the target MySQL instance, the dump loading utility checks whether the sql_require_primary_key system variable is set to ON, and if it is, returns an error if there is a table in the dump files with no primary key. By default, this system variable is set to OFF in a standard MySQL DB System configuration.

  5. The dump loading utility does not automatically apply the gtid_executed GTID set from the source MySQL instance on the target MySQL instance. The GTID set is included in the dump metadata from MySQL Shell's instance dump utility, schema dump utility, or table dump utility, as the gtidExecuted field in the @.json dump file. To apply these GTIDs on the target MySQL instance for use with replication, use the updateGtidSet option or import them manually, depending on the release of the target MySQL instance and the MySQL Shell release. From MySQL Shell 8.0.23, this is supported on MySQL DB System instances. See the description of the updateGtidSet option for details.

For output produced by the instance dump utility or schema dump utility, MySQL Shell's dump loading utility uses the DDL files and tab-separated .tsv data files to set up the server instance or schema in the target MySQL instance, then loads the data. Dumps containing only the DDL files or only the data files can be used to perform these tasks separately. The dump loading utility also lets you separately apply the DDL files and data files from a regular dump that contains both sorts of files.

For output produced by MySQL Shell's table dump utility, from MySQL Shell 8.0.23, the dump contains the information required to set up the schema that originally contained the table. By default, from that release, the schema is recreated in the target MySQL instance if it does not already exist. Alternatively, you can specify the schema option in the dump loading utility to load the table into an alternative schema in the target MySQL instance, which must exist there. In MySQL Shell 8.0.22, the table dump utility's files do not contain the schema information, so the target schema must exist in the target MySQL instance. In that release, by default, the current schema of the global MySQL Shell session is used as the target schema, or the schema option can be used to name the schema.

You can customize the import with further options in the dump loading utility:

  • You can select individual tables or schemas to import or to exclude from the import.

  • Users and their roles and grants are excluded by default, but you can choose to import them.

  • You can specify a different character set for the data in the target MySQL instance to that used in the dump files.

  • You can update the ANALYZE TABLE histograms, even after the data has already been loaded.

  • You can choose to skip binary logging on the target MySQL instance during the course of the import using a SET sql_log_bin=0 statement.

You can carry out a dry run with your chosen set of dump loading options to show what actions would be performed when you run the utility for real with those options.

The waitDumpTimeout option lets you apply a dump that is still in the process of being created. Tables are loaded as they become available, and the utility waits for the specified number of seconds after new data stops arriving in the dump location. When the timeout elapses, the utility assumes the dump is complete and stops importing.

Progress state for an import is stored in a persistent progress state file, which records steps successfully completed and steps that were interrupted or failed. By default, the progress state file is named load-progress.server_uuid.json and created in the dump directory, but you can choose a different name and location. The dump loading utility references the progress state file when you resume or retry the import for a dump, and skips completed steps. De-duplication is automatically managed for tables that were partially loaded. If you interrupt a dump in progress by using Ctrl + C, on the first use of that key combination, no new tasks are started by the utility but existing tasks continue. Pressing Ctrl + C again stops existing tasks, resulting in error messages. In either case, the utility can still resume the import from where it stopped.

You can choose to reset the progress state and start the import for a dump again from the beginning, but in this case the utility does not skip objects that were already created and does not manage de-duplication. If you do this, to ensure a correct import, you must manually remove from the target MySQL instance all previously loaded objects from that dump, including schemas, tables, users, views, triggers, routines, and events. Otherwise, the import stops with an error if an object in the dump files already exists in the target MySQL instance. With appropriate caution, you may use the ignoreExistingObjects option to make the utility report duplicate objects but skip them and continue with the import. Note that the utility does not check whether the contents of the object in the target MySQL instance and in the dump files are different, so it is possible for the resulting import to contain incorrect or invalid data.

Important

Do not change the data in the dump files between a dump stopping and a dump resuming. Resuming a dump after changing the data has undefined behavior and can lead to data inconsistency and data loss. If you need to change the data after partially loading a dump, manually drop all objects that were created during the partial import (as listed in the progress state file), then run the dump loading utility with the resetProgress option to start again from the beginning.

If you need to modify any data in the dump’s data files before importing it to the target MySQL instance, you can do this by combining MySQL Shell’s parallel table import utility util.importTable() with the dump loading utility. To do this, first use the dump loading utility to load only the DDL for the selected table, to create the table on the target server. Then use the parallel table import utility to capture and transform data from the output files for the table, and import it to the target table. Repeat that process as necessary for any other tables where you want to modify the data. Finally, use the dump loading utility to load the DDL and data for any remaining tables that you do not want to modify, excluding the tables that you did modify. For a description of the procedure, see Modifying Dumped Data.

MySQL Shell supports loading dump files from an Object Storage bucket using a pre-authenticated request (PAR). PARs provide a way to let users access a bucket or an object without having their own credentials.

Important

Before using this access method, assess the business requirement for and the security ramifications of pre-authenticated access to a bucket or objects in a bucket. A PAR gives anyone who has the PAR access to the targets identified in the request. Carefully manage the distribution of PARs.

  • From MySQL Shell 8.0.27, MySQL Shell supports using a read access PAR (an Object Read PAR) for all objects in a bucket or objects in a bucket with a specific prefix. For information about creating bucket PARs and prefix PARs, see Using Pre-Authenticated Requests. When using a bucket PAR or prefix PAR, the dump loading utility requires a local progress state file. The content of the file is in JSON format, so a text file with a .json extension is appropriate (for example, progress.json). The following example shows the syntax for loading dump files using a PAR created for all objects in a bucket: 

    shell-js> util.loadDump("BucketPARURL", progressFile: "progress.json"})

    The same syntax is used to load objects in a bucket with a specific prefix, but in this case, the PAR URL includes the prefix: 

    shell-js> util.loadDump("PrefixPARURL", progressFile: "progress.json"})

  • From MySQL Shell 8.0.22, MySQL Shell supports using a read access PAR (an Object Read PAR) created for a MySQL Shell dump manifest file (@.manifest.json) to load data from an Object Storage bucket. For information about creating a PAR for a specific object such as a manifest file, see Using Pre-Authenticated Requests. When the ociParManifest option is enabled, the MySQL Shell dump loading utility creates a manifest file when exporting data to an Object Storage bucket. The manifest file contains a PAR for each item in the dump. Prior to MySQL 8.0.27, if the ocimds option is enabled and a bucket name is provided by the osBucketName option, ociParManifest is enabled automatically. From MySQL Shell 8.0.27, with the introduction of support for PARs for all objects in a bucket or objects in a bucket with a specific prefix, the ociParManifest option is set to false by default and is only enabled if set to true explicitly.

    When using a PAR created for a manifest file, a progress state file is required. The content of the file is in JSON format, so a text file with a .json extension is appropriate (for example, progress.json). The progress state file can be created in the same prefixed location as the dump files in the Object Storage bucket, or it can be created locally. If the progress state file is created in the Object Storage bucket, you must create a read-write access PAR (an Object Read Write PAR) for the progress state file. For information about creating a PAR for a specific object, see Using Pre-Authenticated Requests. You can use any user account with the required permissions to create a PAR for the progress state file. A local progress state file does not require a PAR. Consider using a local progress state file if you do not have the permissions required to create a PAR. Note that a local progress file does not permit resuming progress from a different location in the event of a failure.

    Note

    Creating a dump with the ociParManifest option enabled generates a manifest file containing a PAR for each item in the dump. Generating PARs for each item in a dump is time consuming for large datasets, and an additional PAR must be created for the manifest file and possibly for a progress state file. Also, when PARs expire, the dump must be recreated to regenerate PARs for the items in the dump. For these reasons, using a bucket or prefix PAR (supported from MySQL Shell 8.0.27) is the recommended method for loading MySQL Shell dump files from an Object Storage bucket. When using a bucket or prefix PAR, there is only a single PAR to create and manage, and PARs are not required for each item in the dump.

    The following example shows the syntax for loading dump files using PARs created for the manifest file and a progress state file. If using a local progress state file, the progressFile option specifies the path to the local progress state file instead of a PAR URL. 

    shell-js> util.loadDump("PARURLofManifest", {osBucketName: "mds-bucket", 
          osNamespace: "NamespaceID", progressFile: "RWPARUrlOfJsonProgressFile"})

    While the dump is still in progress, the dump loading utility monitors and waits for new additions to the manifest file, rather than to the Object Storage bucket.

The tables in a dump are loaded in parallel by the number of threads you specify using the threads option, which defaults to 4. If table data was chunked when the dump was created, multiple threads can be used for a table, otherwise each thread loads one table at a time. The dump loading utility schedules data imports across threads to maximize parallelism. From MySQL Shell 8.0.27, a pool of background threads is used to fetch the contents of files. If the dump files were compressed by MySQL Shell's dump utilities, the dump loading utility handles decompression for them.

By default, fulltext indexes for a table are created only after the table is completely loaded, which speeds up the import. You can choose to defer all index creation (except the primary index) until each table is completely loaded. You can also opt to create all indexes during the table import. You can also choose to disable index creation during the import, and create the indexes afterwards, for example if you want to make changes to the table structure after loading.

For an additional improvement to data loading performance, you can disable the InnoDB redo log on the target MySQL instance during the import. Note that this should only be done on a new MySQL Server instance (not a production system), and this feature is not available on MySQL DB System. For more information, see Disabling Redo Logging.

The dump loading utility uses the MySQL Shell global session to obtain the connection details of the target MySQL instance to which the dump is to be imported. You must open the global session (which can have an X Protocol connection or a classic MySQL protocol connection) before running the utility. The utility opens its own sessions for each thread, copying options such as connection compression and SSL options from the global session, and does not make any further use of the global session.

In the MySQL Shell API, the dump loading utility is a function of the util global object, and has the following signature: 

util.loadDump(url[, options])

If you are importing a dump that is located in the Oracle Cloud Infrastructure Compute instance's filesystem where you are running the utility, url is a string specifying the path to a local directory containing the dump files. You can prefix a local directory path with the file:// schema. In this example in MySQL Shell's JavaScript mode, a dry run is carried out to check that there will be no issues when the dump files are loaded from a local directory into the connected MySQL instance: 

shell-js> util.loadDump("/mnt/data/worlddump", {dryRun: true})

If you are importing a dump from an Oracle Cloud Infrastructure Object Storage bucket, url is the path prefix that the dump files have in the bucket, which was assigned using the outputUrl parameter when the dump was created. Use the osBucketName option to provide the name of the Object Storage bucket, and the osNamespace option to identify the namespace for the bucket. In this example in MySQL Shell's JavaScript mode, the dump prefixed worlddump is loaded from an Object Storage bucket into the connected MySQL DB System using 8 threads: 

shell-js> util.loadDump("worlddump", {
          threads: 8, osBucketName: "hanna-bucket", osNamespace: "idx28w1ckztq"})

The namespace for an Object Storage bucket is displayed in the Bucket Information tab of the bucket details page in the Oracle Cloud Infrastructure console, or can be obtained using the Oracle Cloud Infrastructure command line interface. A connection is established to the Object Storage bucket using the default profile in the default Oracle Cloud Infrastructure CLI configuration file, or alternative details that you specify using the ociConfigFile and ociProfile options. For instructions to set up a CLI configuration file, see SDK and CLI Configuration File

util.loadDump() from releases of MySQL Shell previous to 8.0.27 cannot load dumps made by versions of the MySQL Shell dump utilities from later releases.

options is a dictionary of options that can be omitted if it is empty. The following options are available:

dryRun: [ true | false ]

Display information about what actions would be performed given the specified options and dump files, including any errors that would be returned based on the dump contents, but do not proceed with the import. The default is false.

osBucketName: "string"

The name of the Oracle Cloud Infrastructure Object Storage bucket where the dump files are located. By default, the [DEFAULT] profile in the Oracle Cloud Infrastructure CLI configuration file located at ~/.oci/config is used to establish a connection to the bucket. You can substitute an alternative profile to be used for the connection with the ociConfigFile and ociProfile options. For instructions to set up a CLI configuration file, see SDK and CLI Configuration File.

osNamespace: "string"

The Oracle Cloud Infrastructure namespace where the Object Storage bucket named by osBucketName is located. The namespace for an Object Storage bucket is displayed in the Bucket Information tab of the bucket details page in the Oracle Cloud Infrastructure console, or can be obtained using the Oracle Cloud Infrastructure command line interface.

ociConfigFile: "string"

An Oracle Cloud Infrastructure CLI configuration file that contains the profile to use for the connection, instead of the one in the default location ~/.oci/config.

ociProfile: "string"

The profile name of the Oracle Cloud Infrastructure profile to use for the connection, instead of the [DEFAULT] profile in the Oracle Cloud Infrastructure CLI configuration file used for the connection.

threads: number

The number of parallel threads to use to upload chunks of data to the target MySQL instance. Each thread has its own connection to the MySQL instance. The default is 4. if the dump was created with chunking enabled (which is the default), the utility can use multiple threads to load data for a table; otherwise a thread is only used for one table.

backgroundThreads: number

The number of threads in the pool of background threads used to fetch the contents of files. This option, and the thread pool, are available from MySQL Shell 8.0.27. The default is the value of the threads option for a dump loaded from the local server, or four times the value of the threads option for a dump loaded from a non-local server.

progressFile: "string"

Specifies the path to a local progress state file for tracking load progress. Other values are permitted depending on the type of load operation:

When loading a dump from local storage:

  • The progressFile option may be omitted. In this case, a progress state file named load-progress-server-uuid.json is automatically created in the dump directory.

  • The progressFile option can be set to an empty string to disable progress state tracking, which means that the dump loading utility cannot resume a partially completed import.

When loading a dump from OCI Object Storage using a pre-authenticated request (PAR), the progressFile option is mandatory.

  • If the load operation is performed using a bucket or prefix PAR, set the progressFile option to the path of a local progress state file.

  • If the load operation is performed using a manifest file PAR, set the progressFile option to the path of a local progress state file or specify a write PAR for a progress state file residing in the same location as the manifest file.

If a local progress state file or a valid write PAR is specified but the progress state file does not exist, the file will be created.

showProgress: [ true | false ]

Display (true) or hide (false) progress information for the import. The default is true if stdout is a terminal (tty), such as when MySQL Shell is in interactive mode, and false otherwise. The progress information includes the number of active threads and their actions, the amount of data loaded so far, the percentage complete and the rate of throughput. When the progress information is not displayed, progress state is still recorded in the dump loading utility's progress state file.

resetProgress: [ true | false ]

Setting this option to true resets the progress state and starts the import again from the beginning. The default is false. Note that with this option, the dump loading utility does not skip objects that were already created and does not manage de-duplication. If you want to use this option, to ensure a correct import, you must first manually remove from the target MySQL instance all previously loaded objects, including schemas, tables, users, views, triggers, routines, and events from that dump. Otherwise, the import stops with an error if an object in the dump files already exists in the target MySQL instance. With appropriate caution, you may use the ignoreExistingObjects option to make the utility report duplicate objects but skip them and continue with the import.

waitDumpTimeout: number

Setting this option to a value greater than 0 activates concurrent loading of the dump while it is still being produced. The value is a timeout (in seconds) for which the utility waits for further data after all uploaded data chunks in the dump location have been processed. This allows the utility to import the dump while it is still in the process of being created. Data is processed as it becomes available, and the import stops when the timeout is exceeded with no further data appearing in the dump location. The default setting, 0, means that the utility marks the dump as complete when all uploaded data chunks have been processed and does not wait for more data. With the default setting, concurrent loading is disabled.

ignoreExistingObjects: [ true | false ]

Import the dump even if it contains objects that already exist in the target schema in the MySQL instance. The default is false, meaning that an error is issued and the import stops when a duplicate object is found, unless the import is being resumed from a previous attempt using a progress state file, in which case the check is skipped. When this option is set to true, duplicate objects are reported but no error is generated and the import proceeds. This option should be used with caution, because the utility does not check whether the contents of the object in the target MySQL instance and in the dump files are different, so it is possible for the resulting import to contain incorrect or invalid data. An alternative strategy is to use the excludeTables option to exclude tables that you have already loaded where you have verified the object in the dump files is identical with the imported object in the target MySQL instance. The safest choice is to remove duplicate objects from the target MySQL instance before restarting the dump.

ignoreVersion: [ true | false ]

Import the dump even if the major version number of the MySQL instance from which the data was dumped is different to the major version number of the MySQL instance to which the data will be uploaded. The default is false, meaning that an error is issued and the import does not proceed if the major version number is different. When this option is set to true, a warning is issued and the import proceeds. Note that the import will only be successful if the schemas in the dump files have no compatibility issues with the new major version.

From MySQL Shell 8.0.23, this option also permits the import of a dump created without the use of the ocimds option into a MySQL Database Service instance.

Before attempting an import using the ignoreVersion option, use MySQL Shell's upgrade checker utility checkForServerUpgrade() to check the schemas on the source MySQL instance. Fix any compatibility issues identified by the utility before dumping the schemas and importing them to the target MySQL instance.

showMetadata: [ true | false ]

Prints the gtid_executed GTID set and the binary log file name and position from the source instance, taken from the dump metadata included with dumps produced by MySQL Shell's instance dump utility, schema dump utility, or table dump utility. The metadata is printed in YAML format. This option is available from MySQL Shell 8.0.24.

The gtid_executed GTID set is always included in the dump as the gtidExecuted field in the @.json dump file. The dump loading utility does not automatically apply the gtid_executed GTID set from the source MySQL instance on the target MySQL instance. To apply these GTIDs on the target MySQL instance for use with replication, use the updateGtidSet option or import them manually, depending on the release of the target MySQL instance. From MySQL Shell 8.0.23, this is supported on MySQL DB System instances. See the description of the updateGtidSet option for details.

The binary log file name and position are included provided that the user account used to run the dump utility had the REPLICATION CLIENT privilege. The binary log file name and position can be used to set up replication from a source server that does not have GTIDs enabled and does not use GTID-based replication, to a replica that has GTIDs enabled, using the ASSIGN_GTIDS_TO_ANONYMOUS_TRANSACTIONS option of the CHANGE REPLICATION SOURCE TO statement (which is available from MySQL Server 8.0.23).

updateGtidSet: [ off | append | replace ]

Apply the gtid_executed GTID set from the source MySQL instance, as recorded in the dump metadata, to the gtid_purged GTID set on the target MySQL instance. The gtid_purged GTID set holds the GTIDs of all transactions that have been applied on the server, but do not exist on any binary log file on the server. This option is available from MySQL Shell 8.0.22, but in that release it is not supported on MySQL DB System due to a permissions restriction. From MySQL 8.0.23, the option can also be used for a MySQL DB System instance. The default is off, meaning that the GTID set is not applied.

Do not use this option for a dump produced by MySQL Shell's table dump utility, only for dumps produced by MySQL Shell's instance dump utility or schema dump utility. Also, do not use this option when Group Replication is running on the target MySQL instance.

For MySQL instances that are not MySQL DB System instances, when you set append or replace to update the GTID set, also set the skipBinlog option to true. This ensures the GTIDs on the source server match the GTIDs on the target server. For MySQL DB System instances, this option is not used.

For a target MySQL instance from MySQL 8.0, you can set the option to append, which appends the gtid_executed GTID set from the source MySQL instance to the gtid_purged GTID set on the target MySQL instance. The gtid_executed GTID set to be applied, which is shown in the gtidExecuted field in the @.json dump file, must not intersect with the gtid_executed set already on the target MySQL instance. For example, you can use this option when importing a schema from a different source MySQL instance to a target MySQL instance that already has schemas from other source servers.

You can also use replace for a target MySQL instance from MySQL 8.0, to replace the gtid_purged GTID set on the target MySQL instance with the gtid_executed GTID set from the source MySQL instance. To do this, the gtid_executed GTID set from the source MySQL instance must be a superset of the gtid_purged GTID set on the target MySQL instance, and must not intersect with the set of transactions in the target's gtid_executed GTID set that are not in its gtid_purged GTID set.

For a target MySQL instance at MySQL 5.7, set the option to replace, which replaces the gtid_purged GTID set on the target MySQL instance with the gtid_executed GTID set from the source MySQL instance. In MySQL 5.7, to do this the gtid_executed and gtid_purged GTID sets on the target MySQL instance must be empty, so the instance must be unused with no previously imported GTID sets.

In MySQL Shell 8.0.21, where this option is not available, you can apply the GTID set manually on a MySQL Server instance (except where Group Replication is in use). For MySQL DB System, this method is not supported. To apply the GTID set, after the import, use MySQL Shell's \sql command (or enter SQL mode) to issue the following statement on the connected MySQL instance, copying the gtid_executed GTID set from the gtidExecuted field in the @.json dump file in the dump metadata: 

shell-js> \sql SET @@GLOBAL.gtid_purged= "+gtidExecuted_set";

This statement, which works from MySQL 8.0, adds the source MySQL Server instance's gtid_executed GTID set to the target MySQL instance's gtid_purged GTID set. For MySQL 5.7, the plus sign (+) must be omitted, and the gtid_executed and gtid_purged GTID sets on the target MySQL instance must be empty. For more details, see the description of the gtid_purged system variable in the release of the target MySQL instance.

skipBinlog: [ true | false ]

Skips binary logging on the target MySQL instance for the sessions used by the utility during the course of the import, by issuing a SET sql_log_bin=0 statement. The default is false, so binary logging is active by default. For MySQL DB System, this option is not used, and the import stops with an error if you attempt to set it to true. For other MySQL instances, always set skipBinlog to true if you are applying the gtid_executed GTID set from the source MySQL instance on the target MySQL instance, either using the updateGtidSet option or manually. When GTIDs are in use on the target MySQL instance (gtid_mode=ON), setting this option to true prevents new GTIDs from being generated and assigned as the import is being carried out, so that the original GTID set from the source server can be used. The user account must have the required permissions to set the sql_log_bin system variable.

loadIndexes: [ true | false ]

Create (true) or do not create (false) secondary indexes for tables. The default is true. When this option is set to false, secondary indexes are not created during the import, and you must create them afterwards. This can be useful if you are loading the DDL files and data files separately, and if you want to make changes to the table structure after loading the DDL files. Afterwards, you can create the secondary indexes by running the dump loading utility again with loadIndexes set to true and deferTableIndexes set to all.

deferTableIndexes: [ off | fulltext | all ]

Defer the creation of secondary indexes until after the table data is loaded. This can reduce loading times. off means all indexes are created during the table load. The default setting fulltext defers full-text indexes only. all defers all secondary indexes and only creates primary indexes during the table load, and also (from MySQL Shell 8.0.22) indexes defined on columns containing auto-increment values. In MySQL Shell 8.0.21, do not set all if you have any unique key columns containing auto-increment values.

analyzeTables: [ off | on | histogram ]

Execute ANALYZE TABLE for tables when they have been loaded. on analyzes all tables, and histogram analyzes only tables that have histogram information stored in the dump. The default is off. You can run the dump loading utility with this option to analyze the tables even if the data has already been loaded.

characterSet: "string"

The character set to be used for the import to the target MySQL instance, for example in the CHARACTER SET option of the LOAD DATA statement. The default is the character set given in the dump metadata that was used when the dump was created by MySQL Shell's instance dump utility, schema dump utility, or table dump utility, which default to using utf8mb4. The character set must be permitted by the character_set_client system variable and supported by the MySQL instance.

schema: "string"

The existing target schema into which a dump produced by MySQL Shell's table dump utility must be loaded.

From MySQL Shell 8.0.23, this option is not required, because the dump files from the table dump utility contain the information required to set up the schema that originally contained the table. By default, from that release, the schema is recreated in the target MySQL instance if it does not already exist. Alternatively, you can specify the schema option to load the table into an alternative schema in the target MySQL instance, which must exist there.

In MySQL Shell 8.0.22, the dump files from the table dump utility do not contain the schema information, so the target schema must exist in the target MySQL instance. In that release, by default, the current schema of the global shell session is used as the target schema, or the schema option can be used to name the target schema.

excludeSchemas: array of strings

Exclude the named schemas from the import. Note that the information_schema, mysql, ndbinfo, performance_schema, and sys schemas are always excluded from a dump that is created by MySQL Shell's instance dump utility. If a named schema does not exist in the dump files, the utility ignores the item.

includeSchemas: array of strings

Load only the named schemas from the dump files. You can specify both options, in which case a schema name matched by both an includeSchemas string and an excludeSchemas string is excluded.

excludeTables: array of strings

Exclude the named tables from the import. Table names must be qualified with a valid schema name, and quoted with the backtick character if needed. Note that the data for the mysql.apply_status, mysql.general_log, mysql.schema, and mysql.slow_log tables is always excluded from a dump created by MySQL Shell's schema dump utility, although their DDL statements are included. Tables named by the excludeTables option are not uploaded to the target MySQL instance. If a named table does not exist in the schema or the schema does not exist in the dump files, the dump loading utility ignores the item.

includeTables: array of strings

Load only the named tables from the dump files. Table names must be qualified with a valid schema name, and quoted with the backtick character if needed. You can specify both options, in which case a table name matched by both an includeTables string and an excludeTables string is excluded.

loadDdl: [ true | false ]

Setting this option to false excludes the DDL files in the dump from the load. The default is true, meaning that the DDL files are loaded.

loadData: [ true | false ]

Setting this option to false excludes the data files in the dump from the load. The default is true, meaning that the data files are loaded.

loadUsers: [ true | false ]

Import (true) or do not import (false) users and their roles and grants into the target MySQL instance. The default is false, so users are not imported by default. Statements for the current user are skipped. From MySQL Shell 8.0.22, if a user already exists in the target MySQL instance, an error is returned and the user's grants from the dump files are not applied. From MySQL Shell 8.0.22, you can use the excludeUsers or includeUsers option in the dump loading utility to specify user accounts to be excluded or included in the import.

Note

In MySQL Shell 8.0.21, attempting to import users to a MySQL DB System causes the import to fail if the root user account or another restricted user account name is present in the dump files, so the import of users to a MySQL DB System is not supported in that release.

MySQL Shell's schema dump utility and table dump utility do not include users, roles, and grants in a dump, but the instance dump utility can, and does by default. From MySQL Shell 8.0.22, the excludeUsers and includeUsers options can also be used in the instance dump utility to exclude or include named user accounts from the dump files.

If you specify true but the supplied dump files do not contain user accounts, before MySQL Shell 8.0.23, the utility returns an error and stops the import. From MySQL Shell 8.0.23, the utility instead returns a warning and continues.

excludeUsers: array of strings

Exclude the named user accounts from the import. This option is available from MySQL Shell 8.0.22, and you can use it to exclude user accounts that are not accepted for import to a MySQL DB System, or that already exist or are not wanted on the target MySQL instance. Specify each user account string in the format "'user_name'@'host_name'" for an account that is defined with a user name and host name, or "'user_name'" for an account that is defined with a user name only. If you do not supply a host name, all accounts with that user name are excluded. If a named user account does not exist in the dump files, the utility ignores the item.

includeUsers: array of strings

Include only the named user accounts in the import. Specify each user account string as for the excludeUsers option. This option is available from MySQL Shell 8.0.22, and you can use it as an alternative to excludeUsers if only a few user accounts are required in the target MySQL instance. You can also specify both options, in which case a user account matched by both an includeUsers string and an excludeUsers string is excluded.

createInvisiblePKs: [ true | false ]

Add primary keys in invisible columns for each table in the dump that does not contain a primary key. The true setting is applied automatically if the dump was created with the create_invisible_pks option by MySQL Shell’s instance dump utility util.dumpInstance(), schema dump utility util.dumpSchemas(), or table dump utility util.dumpTables(). The primary keys are only added if the DDL for the dump is loaded (loadDdl: true). The invisible columns (which are named "my_row_id") have no impact on applications that use the uploaded tables.

createInvisiblePKs is present from MySQL Shell 8.0.24, and when the true setting is in effect, the target MySQL instance must be MySQL Server 8.0.24 or later, or the load fails. Invisible columns are available from MySQL Server 8.0.23, but a limitation on them in that release prevents the use of this function. The dump loading utility in versions of MySQL Shell before MySQL Shell 8.0.24 silently ignores the dump metadata flag and does not add the primary keys, so ensure that you use the latest version of the utility.

Adding primary keys in this way does not yet enable inbound replication of the modified tables to a High Availability instance, as that feature currently requires the primary keys to exist in both the source server and the replica server. If possible, instead of using this option, consider creating primary keys in the tables on the source server, before dumping them again. From MySQL 8.0.23, you can do this with no impact to applications by using invisible columns to hold the primary keys. This is a best practice for performance and usability, and helps the dumped database to work seamlessly with MySQL Database Service.

maxBytesPerTransaction: number

The maximum number of bytes that can be loaded from a data file in a single LOAD DATA statement. If a data file exceeds the maxBytesPerTransaction value, multiple LOAD DATA statements load data from the file in chunks less than or equal to the maxBytesPerTransaction value. This option is available from MySQL Shell 8.0.27.

The unit suffixes k for kilobytes, M for megabytes, and G for gigabytes can be used. The minimum value is 4069 bytes. If a lesser value is specified, the 4096 byte minimum is used implicitly. If the maxBytesPerTransaction option is unset, the bytesPerChunk value used to dump the data is used as the default setting for files larger than 1.5 * the bytesPerChunk value. If the maxBytesPerTransaction option is unset and the data file is less than 1.5 * the bytesPerChunk value, the data is requested in a single LOAD DATA statement.

If a data file contains a row that is larger than the maxBytesPerTransaction setting, the row's data is requested in a single LOAD DATA statement. A warning is emitted for the first row encountered that exceeds the maxBytesPerTransaction setting.

If a load operation with a configured maxBytesPerTransaction setting is interrupted and resumes execution, chunks that were already loaded are skipped. The resumed load operation uses the current maxBytesPerTransaction setting. The setting used before the operation was interrupted is not saved to the progress state file.

An intended use for this option is to load data in smaller chunks when a data file is too large for the target server's limits, such as the limits defined by the server's group_replication_transaction_size_limit or max_binlog_cache_size settings. For example, If you receive the error "MySQL Error 1197 (HY000): Multi-statement transaction required more than 'max_binlog_cache_size' bytes of storage" when loading data, set maxBytesPerTransaction to a value less than or equal to the server instance’s max_binlog_cache_size setting.

Modifying Dumped Data

MySQL Shell’s parallel table import utility util.importTable() can be used in combination with the dump loading utility util.loadDump() to modify data in the chunked output files before uploading it to the target MySQL instance. You can modify the data for one table at a time by this method. Follow this procedure, which works from MySQL Shell 8.0.23:

  1. Use the dump loading utility with the loadDdl: true and loadData: false options, to load the DDL file only, and create the selected table on the target MySQL instance with no data. 

    shell-js> util.loadDump("/mnt/data/proddump", { 
        > includeTables: ["product.pricing"], 
        > loadDdl: true, 
        > loadData: false});

  2. Use the parallel table import utility to capture and transform the data for the table, and import it to the empty table on the target MySQL instance. In this example, the data for the pricing table is in multiple compressed files, which are specified using wildcard pattern matching. The values from the id and prodname columns in the dump files are assigned unchanged to the same columns in the target table. The values from the price column in the dump files are captured and assigned to the variable @1. The decodeColumns option is then used to reduce the prices by a standard amount, and the reduced prices are placed in the price column of the target table. 

    shell-js> util.importTable ("/mnt/data/proddump/product@pricing@*.zst", {   
        > schema: "product",  
        > table: "pricing",  
        > columns: ["id",  "prodname",  1],
        > decodeColumns: { "price": "0.8 * @1"}});

  3. Repeat Steps 1 and 2 as needed for any other tables in the dump files where you need to modify the data.

  4. When you have finished uploading all the tables and data that needed to be modified, use the dump loading utility to load both the DDL and the data for any remaining tables that you do not need to modify. Be sure to exclude the tables that you did modify in the previous steps. 

    shell-js> util.loadDump("/mnt/data/proddump", {excludeTables: ["product.pricing"]});

PREV   HOME   UP   NEXT
Related Documentation
MySQL Shell 8.0 Release Notes
Download this Manual
PDF (US Ltr) - 1.7Mb
PDF (A4) - 1.7Mb