This section describes how to load data into HeatWave. The following methods are supported:
Loading data manually. This method loads one table at a time and involves executing multiple statements for each table. See Section 2.2.2, “Loading Data Manually”.
Loading data using Auto Parallel Load. This Autopilot-enabled method loads one or more schemas at a time and facilitates loading by automating manual steps and optimizing the number of parallel load threads for a faster load. See Section 2.2.3, “Loading Data Using Auto Parallel Load”.
When data is loaded into HeatWave, it is read from
InnoDB using batched, multi-threaded reads.
It is then converted into columnar format and sent over the
network and distributed among the HeatWave nodes in horizontal
slices. Data is partitioned by the table's primary key unless
data placement keys are defined. See
Section 2.7.2, “Defining Data Placement Keys”.
Concurrent DML operations and queries on the MySQL node are supported while a data load operation is in progress; however, concurrent operations on the MySQL node can affect load performance and vice versa.
After tables are loaded, changes to table data on the MySQL DB System node are automatically propagated to HeatWave. For more information, see Section 2.2.7, “Change Propagation”.
For each table that is loaded in HeatWave, 4MB of memory (the default heap segment size) is allocated from the root heap. This memory requirement should be considered when loading a large number of tables. For example, with a root heap of approximately 400GB available to HeatWave, loading 100K tables would consume all available root heap memory (100K x 4GB = 400GB). From MySQL 8.0.30, the default heap segment size is reduced from 4MB to a default of 64KB per table, reducing the amount of memory that must be allocated from the root heap for each loaded table.
As of MySQL 8.0.28-u1, HeatWave compresses data as it is loaded, which permits HeatWave nodes to store more data at a minor cost to performance. If you do not want to compress data as it is loaded in HeatWave, you must disable compression before loading data. For instructions, see Section 2.2.6, “Data Compression”.
DDL operations are not permitted on tables that are loaded in
HeatWave. To alter the definition of a table, you must unload
the table and remove the
attribute before performing the DDL operation. See
Section 2.4, “Modifying Tables”.
For related best practices, see Section 2.8, “Best Practices”.