7.10.2 Troubleshooting InnoDB Cluster Upgrades

This section covers trouble shooting the upgrade process.

Handling Host Name Changes

MySQL Shell uses the host value of the provided connection parameters as the target hostname used for AdminAPI operations, namely to register the instance in the metadata (for the dba.createCluster() and Cluster.addInstance() operations). However, the actual host used for the connection parameters might not match the hostname that is used or reported by Group Replication, which uses the value of the report_host system variable when it is defined (in other words it is not NULL), otherwise the value of hostname is used. Therefore, AdminAPI now follows the same logic to register the target instance in the metadata and as the default value for the group_replication_local_address variable on instances, instead of using the host value from the instance connection parameters. When the report_host variable is set to empty, Group Replication uses an empty value for the host but AdminAPI (for example in commands such as dba.checkInstanceConfiguration(), dba.configureInstance(), dba.createCluster(), and so on) reports the hostname as the value used which is inconsistent with the value reported by Group Replication. If an empty value is set for the report_host system variable, an error is generated. (Bug #28285389)

For a cluster created using a MySQL Shell version earlier than 8.0.16, an attempt to reboot the cluster from a complete outage performed using version 8.0.16 or higher results in this error. This is caused by a mismatch of the Metadata values with the report_host or hostname values reported by the instances. The workaround is to:

1. Identify which of the instances is the “seed”, in other words the one with the most recent GTID set. The dba.rebootClusterFromCompleteOutage() operation detects whether the instance is a seed and the operation generates an error if the current session is not connected to the most up-to-date instance.

2. Set the report_host system variable to the value that is stored in the Metadata schema for the target instance. This value is the hostname:port pair used in the instance definition upon cluster creation. The value can be consulted by querying the mysql_innodb_cluster_metadata.instances table.

   For example, suppose a cluster was created using the following sequence of commands:

   mysql-js> \c clusterAdmin@localhost:3306
   mysql-js> dba.createCluster("myCluster")

   Therefore the hostname value stored in the metadata is “localhost” and for that reason, report_host must be set to “localhost” on the seed.

3. Reboot the cluster using only the seed instance. At the interactive prompts do not add the remaining instances to the cluster.

4. Use Cluster.rescan() to add the other instances back to the cluster.

5. Remove the seed instance from the cluster

6. Stop mysqld on the seed instance and either remove the forced report_host setting (step 2), or replace it with the value previously stored in the Metadata value.

7. Restart the seed instance and add it back to the cluster using Cluster.addInstance()

This allows a smooth and complete upgrade of the cluster to the latest MySQL Shell version. Another possibility, that depends on the use-case, is to simply set the value of report_host on all cluster members to match what has been registered in the Metadata schema upon cluster creation.