After an initial data load, the data can become outdated, and the external tables require a refresh. Subsequent calls to Auto Parallel Load refresh the data. As of MySQL 9.0.0, you can use Lakehouse Incremental Load, to refresh the loaded data.
Lakehouse Incremental Load uses the existing ENGINE_ATTRIBUTE
option. See: Section 5.2.2, “Lakehouse External Table Syntax”.
Using Lakehouse Incremental Load, it is possible to add or remove individual files if they still match the defined file pattern or file prefix. However, to learn how to change file parameters, file names, file patterns or file prefixes, use Section 5.4.2, “Adding or Removing Files Using Selective Load” which is supported in MySQL 9.1.2 and later versions.
After the initial data load, a subsequent call to Auto Parallel Load with
the refresh_external_tables
option set to
true
refreshes the data, see:
Section 2.2.3.2, “Auto Parallel Load Syntax”.
If a subsequent call to Auto Parallel Load includes tables that have not yet been loaded to Lakehouse, then Auto Parallel Load will load them for the first time.
A call to Auto Parallel Load might contain both loaded and unloaded tables. Those that are unloaded will be loaded, and those that are already loaded will be refreshed.