4.4 HeatWave Cluster Data Recovery

HeatWave MySQL triggers the data recovery process when:

• One or more HeatWave Cluster nodes fail and are restarted.

  Note:

  HeatWave MySQL monitors HeatWave nodes status regularly and if there is no response from a node after 60 seconds, HeatWave MySQL automatically attempts to bring the node online.
• The HeatWave Cluster is restarted.
• The DB System is restarted.

HeatWave data can be recovered from the HeatWave Storage Layer, or reloaded from the DB System and Amazon S3.

• Recover from the HeatWave Storage Layer

  HeatWave first tries to recover data from the HeatWave Storage Layer, which was created when you enabled the HeatWave Cluster for the first time. To facilitate recovery, data is persisted to Storage Layer when data is loaded into the HeatWave Cluster and when data changes are propagated from the DB system to the HeatWave cluster. Loading data from the HeatWave storage layer is faster because the data does not need to be converted to the HeatWave storage format, as is required when loading data from the DB system or Amazon S3.

  Note:

  Automatically loaded tables are not recovered from the Storage Layer if the DB system is restarted; they are only recovered if the recovery was triggered by other reasons, and the DB system remains running during the recovery.
• Reload from the DB system or Amazon S3
  Data cannot be recovered from the HeatWave Storage Layer in these situations and all tables must be reloaded from the DB System or Amazon S3 for Lakehouse tables:

  • Recovery from the HeatWave storage layer has failed due to Amazon S3 outage or network outage.
  • The binary logs required to perform change propagation of the data restored from the HeatWave storage layer have been purged.
  • The DB System has been upgraded and the data stored in the HeatWave storage layer is not compatible with the new MySQL version of the DB System.
  • The recovery is triggered by the restart of a DB System of MySQL version 9.2.1 or earlier.
  Even if some tables can be reloaded from the Storage Layer, some might not. After a DB system restart, the following types of tables need to be reloaded from the DB System or Amazon S3:

  • Tables with dictionary encoded columns.
  • Stale tables.
  • Tables that were not captured in periodic metadata checkpoints.
  • Tables where partition load or unload operations were in progress when the DB system stopped abruptly.

  Note:

  Automatically loaded tables are not reloaded during the recovery from the DB System or Amazon S3.

During recovery, the HeatWave Cluster automatically reloads the data. However, if the MySQL Server is in the SUPER_READ_ONLY mode, you cannot load data into the HeatWave Cluster, and the recovery fails. Disable the SUPER_READ_ONLY mode to load data. See Resolving SUPER_READ_ONLY and OFFLINE_MODE Issue Caused by Low Free Storage Space.

See Data Load Progress and Status Monitoring on how to check the progress and status of data loading.

Note:

When you unload a table, the data is removed from the HeatWave Cluster, and it is also removed from AWS S3 too in a background operation. An unloaded table is not recoverable into the HeatWave Cluster by the data recovery process, and must be loaded again manually.

Parent topic: HeatWave Clusters