Changes in MySQL Shell 8.0.30 (2022-07-26, General Availability)

• AdminAPI Bugs Fixed

• Functionality Added or Changed

• Bugs Fixed

AdminAPI Bugs Fixed

• When a replica cluster was removed from an InnoDB ClusterSet, reconciliation of view change log events was not carried out if the replica cluster had only a single member (the primary member). If that instance was then used to create a new replica cluster, it contained additional transactions that meant replication could not take place with the other clusters in the InnoDB ClusterSet. View change log events are now always reconciled when a replica cluster is dissolved, even when there is only a single member. (Bug #34069295)

• On MySQL sandbox instances created using AdminAPI, an auxiliary MySQL Shell instance is started for AdminAPI operations. Previously, the auxiliary MySQL Shell instance was placed in JSON shell mode using the MYSQLSH_JSON_SHELL environment variable. However, if the user’s MySQL Shell instance was also in JSON shell mode, error messages returned to it by the auxiliary MySQL Shell instance could not be parsed because they were already wrapped in JSON documents, so they were not displayed to the user. Auxiliary MySQL Shell instances on sandbox instances now do not use the MYSQLSH_JSON_SHELL environment variable. (Bug #34066667)

• When the clusterSet.setPrimaryCluster() command or clusterSet.rejoinCluster() command was used, a divergent GTID set on the target cluster was not reconciled because view change events were filtered out too early in the process. If the binary logs had been purged, the situation could not be repaired and the cluster could not continue to operate in the InnoDB ClusterSet. The GTID set reconciliation process has now been fixed. (Bug #34013718)

• ClusterSet.status() returned OK_NO_TOLERANCE instead of OK_NO_TOLERANCE_PARTIAL for ClusterSets with one or more unreachable members and fewer than the acceptable threshold of online members. (Bug #33989031)

• It was not possible to stop an InnoDB cluster metadata upgrade if outdated router instances were detected. When prompted to proceed with the upgrade, or not, the upgrade was performed regardless of the user's choice. (Bug #33941759)

• AdminAPI’s Cluster.rescan() command did not identify the new version of Group Replication’s communication protocol that was introduced in MySQL 8.0.27. The command now offers an upgrade to the new communication protocol when it is available. (Bug #33922830)

• InnoDB Cluster commands validate that the target instance belongs to the cluster before performing the operation. When the server’s address was compared against the server instances registered in the cluster’s metadata, host names were compared using case sensitivity, which is contrary to the rules in RFC 1034 for domain name comparison. This could lead to a target instance not being identified as part of the cluster. Host names are now checked using a case insensitive comparison. (Bug #33893435)

• Cluster.dissolve() did not clean up the metadata and configuration associated with the cluster if the cluster had been removed from a ClusterSet using the force option.

  As of this release, Cluster.dissolve() ensures all recovery and ClusterSet replication accounts are removed, and the metadata schema is removed, also. (Bug #33239404)

• It was possible to add an instance to a cluster although it was already a member of a ReplicaSet. No error was generated as the instance was removed from the ReplicaSet and added to the cluster. As of this release, an error is generated and the action fails. (Bug #31964273)

• The replicationLag value reported by Cluster.status() was not calculated correctly.

  As of this release, it returns one of the following:

  • The time difference between the last transaction commit timestamp and the last transaction applied timestamp, in HH:MM:SS format.

    If multiple workers are used, the value is retrieved from the worker executing the oldest transaction.

  • null: The replication connection or SQL thread is not running.

  • applier_queue_applied: The applier queue has applied everything. That is, if the last queued transaction and the last applied transaction are the same, or the applying transaction is 0.

  For more information, see Checking a cluster's Status with Cluster.status(). (Bug #31693765, Bug #106826, Bug #34002328)

• The InnoDB Cluster Cluster.status() command did not check whether a server instance had been put into offline mode manually by setting the offline_mode system variable. The command now checks for this situation and reports the mode as n/a if the system variable has been set.

  It also reports a warning, for example:

          "instanceErrors": [
                      "WARNING: Instance has offline_mode enabled."
           ],

  (Bug #107057, Bug #34109382)

• MySQL Shell closed unexpectedly if an attempt was made to rejoin an instance to a ReplicaSet after that instance was removed from the ReplicaSet and its server_uuid altered.

  As of this release, the rejoinInstance command ensures the metadata schema is updated with the new server_uuid. (Bug #106847, Bug #34038210)

Functionality Added or Changed

• MySQL Server 8.0.30 introduced GIPK mode, Generated Invisible Primary Keys. When running in this mode, for any InnoDB table that is created without an explicit primary key, the MySQL server automatically adds a generated invisible primary key (GIPK) to the table. This mode is enabled by setting sql_generate_invisible_primary_key to ON.

  MySQL Shell's load utility option createInvisiblePKs uses the server's GIPK mode to generate invisible primary keys for tables which do not have primary keys.

  Under certain circumstances, if a user has insufficient privileges to use GIPK mode, MySQL Shell can fall back to the previous method of generating invisible primary keys.

  See Dump Loading Utility for more information. (Bug #34409792)

• util.checkForServerUpgrade is now run automatically by any of the dump utilities, if the dump option "ocimds": true is defined and the target instance is MySQL 5.7.

  For more information on util.checkForServerUpgrade, see Upgrade Checker Utility. For more information on the dump utilities, see Instance Dump Utility, Schema Dump Utility, and Table Dump Utility. (Bug #34052980)

• MySQL Shell 8.0.30 supports the Group Replication Communication Stack, introduced in MySQL 8.0.27. This allows a simpler and safer setup as it obsoletes the usage of IP allowlists, removes the explicit need for another network address used for internal GCS communication, and introduces user-based authentication.

  This patch adds support in the AdminAPI for the new communication stack by:

  • Adding support to choose which GR communication stack must be used when creating a Cluster or a Replica Cluster, by exposing a new option named communicationStack supporting XCOM or MYSQL.

  • Ensures MYSQL is the default communication stack in use by Clusters running MySQL 8.0.27, or higher.

  • Automatically and transparently generates and sets all configurations required for both communication stacks at cluster creation and for topology changes (addInstance/rejoinInstance/rescan), without user intervention.

  • Allows migrating an existing Cluster to a different communication stack when rebooting it from complete outage by exposing a new option named switchCommunicationStack.

  • Allows setting a value for group_replication_local_address in Cluster.rejoinInstance() by extending the command with a new option localAddress.

  • Adds a new check in Cluster.rescan() to verify whether there is any Cluster member using a recovery account that is not the one created for itself and ensure the right account is used. This ensures instances that fall in that category (added with clone recovery and waitRecovery:0) are able to auto-rejoin the Cluster.

  • Extends Cluster.rejoinInstance() and dba.rebootClusterFromCompleteOutage() to include the ipAllowList and localAddress options.

  • Extends Cluster.rejoinInstance() and dba.rebootClusterFromCompleteOutage() to also perform the instance checks for Cluster compliance as Cluster.addInstance() does.

  • Extends Cluster.status({extended:1}), ClusterSet.status({extended:2}), and Cluster.options() to return the communicationStack values.

  Note

  As of this release, group_replication_tls_source must not be set to mysql_admin.

  (WL #14765)

• The new option, logSql/--log-sql enables you to log all SQL statements executed by MySQL Shell commands or utilities to the MySQL Shell log file.

  Note

  This option deprecates dba.logSql.

  For more information, see MySQL Shell Logging and Debug. (WL #14749)

• MySQL Shell supports the S3-compatible cloud storage services, AWS S3 and Oracle Cloud Infrastructure's S3 compatibility API.

  See MySQL Shell Utilities. (WL #14387)

Bugs Fixed

• Under certain circumstances, when dumping an instance for migration from certain compatible databases, the following error was returned during chunking:

          ERROR: [Worker004]: Error while chunking `xxxx`.`xxxx`: get_uint(8): field type is String

  (Bug #34206985)

• The number of rows loaded in each chunk was not written to the progress file. (Bug #34201239)

• Under certain circumstances, a loadDump operation on a dump which contained a syntax error in a CREATE USER statement could result in a user's password being displayed in the SQL syntax error message.

  As of this release, if a failed query contains either IDENTIFIED or PASSWORD keywords, the quoted strings are anonymized. (Bug #34141432)

• When the dryRun option was set for MySQL Shell's dump loading utility util.loadDump(), if the updateGtidSet option was set to replace or append, the value of the gtid_purged system variable on the target server instance was changed, although a dry run is not meant to affect the target server. The issue has now been fixed. (Bug #34127069)

• On Linux systems, MySQL Shell's password store used the MySQL configuration utility mysql_config_editor as a Secret Store Helper, provided that the MySQL client package was installed on the system, and mysql_config_editor was available in the system’s PATH. Otherwise, the password store functionality was unavailable and MySQL Shell's credential helper functions were unavailable. MySQL Shell now bundles mysql_config_editor in Linux builds, so that the functionality can be used if the MySQL client package is not installed on the system. (Bug #34097411)

• util.debug.collectDiagnostics did not produce an error if a filename was not included in the command. A nameless zip file was created. As of this release, if a filename is not included in the command, an error is displayed and the zip file is not created. (Bug #34048754)

• When you use the shell.create_context function in MySQL Shell's Python mode to create a scope and context wrapper for a program, you can now use the finalize() function to require the release of the file handle created for the logger object in the shell context. (Bug #33988826)

• When MySQL Shell's instance dump utility util.dumpInstance(), schema dump utility util.dumpSchemas(), and table dump utility util.dumpTables() were used to dump tables with compound primary keys, the chunking process could give rise to a filesort operation to optimize the ORDER BY clause. In the case of a large table with multiple chunks being dumped simultaneously, the operation could cause the dump to fail for lack of disk space. This occurred when a BETWEEN clause used the same lower and upper bound, so the chunking process now uses an alternative condition in that situation. (Bug #33976259)

• When MySQL Shell's Python API was used to read date and time data, the conversion did not account for the fact that MySQL can accept invalid date and time values in some SQL modes, leading to Python errors. The data is now only converted into Python values when possible, and in the case of a conversion error, the string representation of the original value is returned. (Bug #33972939)

• MySQL Shell's parallel table import utility util.importTable() and dump loading utility util.loadDump() have a new option sessionInitSQL, which lets you specify a list of SQL statements to run at the start of each client session used for loading data into the target MySQL instance. You can use this option to change session variables. If an error occurs while running the SQL statements, the import stops and returns an error message. (Bug #33969606)

• In file systems where the d_type member does not appear in the dirent structure, GCC versions prior to 8.4.0 cannot detect the file type. MySQL Shell ignores files with unknown file types, so if it was compiled with an affected GCC version, it was unable to load a dump stored on such a file system. Now, MySQL Shell checks its own GCC version and if it is affected, uses the stat() function instead to determine file types. (Bug #33951851)

• Following the extensions to MySQL Shell's shell.prompt() function in MySQL Shell 8.0.29, the resulting JSON documents no longer included a new line at the end so that the end of the document could be identified and subsequent user input could be read. The issue has now been fixed. (Bug #33949419)

• MySQL Shell closed unexpectedly if shell.reconnect() was called without a global session. (Bug #33945641)

• When using the X DevAPI CollectionModify object, methods which accept a document path as a parameter did not properly handle arguments containing spaces. For example, the following attempted to remove an attribute named some:

          col.modify("_id='1'").unset("some path")

  Paths with spaces must be specified using a path expression or quoted using backtick characters. For example:

          col.modify("_id='1'").unset("$.'some path'")

  or

          col.modify("_id='1'").unset("`some path`")

  As of this release, paths containing spaces which are not properly defined with a path expression, such as some path or 'some path', are treated as errors. (Bug #33795166, Bug #33795149)

• The generic Linux packages for MySQL Shell 8.0.28 and 8.0.29 returned an error due to an indirect dependency on the krb5 library for SSH support. The library is now bundled with MySQL Shell and the bundled version takes precedence over any system version. (Bug #33787652)

• /n was not parsed properly, in Python or JavaScript modes, if MySQL Shell was started with --json. An error was returned. (Bug #33678846)

• MySQL Shell's instance dump utility util.dumpInstance(), schema dump utility util.dumpSchemas(), and table dump utility util.dumpTables() upload large files to Oracle Cloud Infrastructure Object Storage buckets by chunking them and using multipart uploads. If the dump operation was canceled by the user or stopped due to an exception in a thread, multipart uploads that had already started were left in an active state in the bucket. The utilities now attempt in these situations to stop any multipart uploads pending completion. (Bug #32583524)

• MySQL Shell's upgrade checker utility util.checkForServerUpgrade() now checks for indexes on generated columns using CONVERT(),which changed its behavior in MySQL 8.0.28, and warns you to check applications that rely on the previous behavior. (Bug #106419, Bug #33840716)

• The upgrade checker utility returned an empty table name and an error for orphaned tables. As of this release, the utility parses the innodb_sys_tables table for the presence of orphaned tables. (Bug #106372, Bug #33822225)

• The upgrade checker utility raised errors, during a 5.7 to 8.0 upgrade check, for MySQL 5.7 schema or table names created using reserved names such as LPT3, AUX, or PRN. These are registered in INFORMATION_SCHEMA.INNODB_SYS_TABLES by appending @@@ to their names. However, their original names were used in TABLES. The upgrade checker raised schema consistency errors for such tables. (Bug #105958, Bug #33707127)