This section describes how to load data into HeatWave. The following methods are supported:
Loading data manually. This method loads one table at a time and involves executing multiple statements for each table. See Section 2.2.2, “Loading Data Manually”.
Loading data using Auto Parallel Load. This MySQL Autopilot enabled method loads one or more schemas at a time and facilitates loading by automating manual steps and optimizing the number of parallel load threads for a faster load. See Section 2.2.3, “Loading Data Using Auto Parallel Load”.
Loading data using the MySQL HeatWave Console (for users of MySQL HeatWave on AWS). This GUI-based and MySQL Autopilot enabled method loads selected schemas and tables using an optimized number of parallel load threads for a faster load. See Manage HeatWave Data with Workspaces, in the MySQL HeatWave on AWS Service Guide.
HeatWave loads data with batched, multi-threaded reads from
InnoDB. HeatWave then converts the data into
columnar format and sends it over the network to distribute it
among HeatWave nodes in horizontal slices. HeatWave partitions data by
the table primary key, unless the table definition includes data
placement keys. See Section 2.7.2, “Defining Data Placement Keys”.
Concurrent DML operations and queries on the MySQL node are supported while a data load operation is in progress; however, concurrent operations on the MySQL node can affect load performance and vice versa.
After tables are loaded, changes to table data on the MySQL DB System node are automatically propagated to HeatWave. For more information, see Section 2.2.7, “Change Propagation”.
For each table that is loaded in HeatWave, 4MB of memory (the default heap segment size) is allocated from the root heap. This memory requirement should be considered when loading a large number of tables. For example, with a root heap of approximately 400GB available to HeatWave, loading 100K tables would consume all available root heap memory (100K x 4GB = 400GB). As of MySQL 8.0.30, the default heap segment size is reduced from 4MB to a default of 64KB per table, reducing the amount of memory that must be allocated from the root heap for each loaded table.
As of MySQL 8.0.28-u1, HeatWave compresses data as it is loaded, which permits HeatWave nodes to store more data at a minor cost to performance. If you do not want to compress data as it is loaded in HeatWave, you must disable compression before loading data. See Section 2.2.6, “Data Compression”.
Before MySQL 8.0.31, DDL operations are not permitted on
tables that are loaded in HeatWave. In those releases, to alter
the definition of a table, you must unload the table and
before performing the DDL operation. See
Section 2.4, “Modifying Tables”.
For related best practices, see Section 2.9, “Best Practices”.