Changes in MySQL Shell 8.0.24 (2021-04-20, General Availability)

• AdminAPI Added or Changed Functionality

• AdminAPI Bugs Fixed

• Functionality Added or Changed

• Bugs Fixed

AdminAPI Added or Changed Functionality

• The API command-line integration, used for scripting MySQL Shell, has been improved to use the function and option metadata to properly parse arguments and associate them to the corresponding parameter in the API call. Previously, the command-line integration was processing all the arguments as strings, causing errors using operations such as:

  > mysqlsh user@hostname:3306 -- cluster setOption "autoRejoinTries"  5

  The command-line integration now interprets the data being passed in based on what the API functions expect to receive. This enhancement also introduces support for lists in command-line integration calls.

  Additionally, you can now access the MySQL Shell online help from the command-line integration. For example, to get help on the shell.options functions, issue:

  $ mysqlsh -- shell options --help

  For more information, see API Command Line Integration.

  In addition, when registering a new MySQL Shell extension function, the new boolean cli option is supported by the shell.addExtensionObjectMember() operation. When an operation is registered with the cli option set to true, the object and the functions are made available for the command-line integration. This enables you to extend the scripting possibilities of MySQL Shell. (Bug #31186637, WL #14297)

AdminAPI Bugs Fixed

• Most of the AdminAPI operations contain metadata preconditions to determine if it is valid or not to execute them. When this check was being done, it was adding two entries to the log, one to indicate the check was about to be done, and another to indicate the metadata state. This meant that operations such as monitoring a cluster, which implies executing regular status requests, resulted in a large number of log entries. Now, these two log messages have been merged into a single entry that gets logged at the info level when the metadata state is not correct, and as debug info when the state is correct. In other words, if the metadata state is correct, the message is only logged when log_level=debug. (Bug #32582745)

• The memberRole is now included in the default output of Cluster.status(). Previously, this information was only included when the extended option had a value of 1 or higher. This makes it easier to know what an instance's role is in the cluster, regardless of whether its operating mode is R/W or R/O. (Bug #32381513)

• dba.checkInstanceConfiguration() was performing an incorrect validation regarding the required privileges. This resulted in an endless loop where the operation detected missing privileges, you would give the grants as specified by the interactive help, but the operation would fail, indicating that the grants were missing. Now, the verification checks the correct list. Additionally, the internal list of required grants included SUPER, which is deprecated in 8.0. The fix replaces the SUPER grant with the fine-grained grants. (Bug #32287986)

• The memberSslMode option did not support the VERIFY_CA and VERIFY_IDENTITY modes for the following operations:

  • dba.createCluster()

  • Cluster.addInstance()

  • Cluster.rejoinInstance()

  Now, the memberSslMode option supports these modes, and when they are used there is a validation to ensure that the CA certificates are supplied. If you choose to use the VERIFY_CA or VERIFY_IDENTITY mode, on each cluster instance you must manually supply the CA certificates using the ssl_ca and/or ssl_capath option. For more information, see Securing InnoDB Cluster.

  Thanks to Daniël van Eeden for the contribution. (Bug #32247631, Bug #32241000, Bug #31227139)

• In version 8.0.23, a check was added to verify if the server_id of all cluster instances is registered in the metadata as an instance attribute, and if not then the metadata is updated accordingly. This check is executed on add, rejoin and rescan operations. However, when upgrading a cluster from a version earlier than 8.0.23 to version 8.0.23 and higher, the server_id was not registered in the metadata unless you performed a manual rejoin. This was being silently ignored because it was not included in any diagnostic messages. Now, a new verification checks if the server_id of cluster instances is missing from the metadata and includes a note message in the instanceErrors attribute in the output of Cluster.status() indicating to use Cluster.rescan() to fix it. (Bug #32226871)

• From MySQL Shell version 8.0.17, AdminAPI stores the replication or recovery accounts used for each added instance in the metadata schema, in the instances table. However, the specific transaction could fail or that entry might have been manually removed from the metadata schema, resulting in failures when trying to add other instances to the cluster, and there was no way to resolve this using AdminAPI. In such a situation, if you tried to add an instance the operation failed with an error. Now, AdminAPI tries to detect problems in the metadata related to these accounts for an InnoDB Cluster or InnoDB ReplicaSet. The status() operation prints a message if the required account is missing in the metadata schema, and also if it is not the one actually used. In such situations, the MySQL Shell help instructs you to either re-add the instance or run rescan() based on the detected problem. The addInstance() operation also prints a hint to call rescan() if any missing recovery users are found in the metadata. (Bug #32157182)

• Cluster.addInstance() was permitting usage of the expelTimeout and consistency options when it should not. These options are cluster level settings that can only be set using dba.createCluster() and Cluster.setOption(). (Bug #29779995)

• The dba.checkInstanceConfiguration() operation detects if the instance has any tables that do not have a primary key. Group Replication requires every table that is to be replicated by the group to have a defined primary key. However, this does not mean that having a table without a primary key causes Group Replication to block or fail. Rather, the outcome is that changes to that table are not replicated but the group continues operating. Previously, if the dba.checkInstanceConfiguration() operation detected a table without a primary key, the operation returned with a status of ok and only mentioned tables with a missing primary key and unsupported engines. Now, if the operation detects such a table, it returns with a status of error. As part of this work, the dba.createCluster() operation has been changed to fail if it finds such tables. (Bug #29771457)

• As part of the fix for Bug#28701263, AdminAPI started setting and persisting a default value of READ_ONLY for the group_replication_exit_state_action system variable. The default used by Group Replication was ABORT_SERVER. However, in MySQL Server 8.0.16 the default value of group_replication_exit_state_action became READ_ONLY so AdminAPI should not change and persist it. Now, on instances running 8.0.16 and later, the value of group_replication_exit_state_action is not modified. (Bug #29037274)

• The exception information listed in the online help had become outdated and unwieldy for the interactive MySQL Shell, so it has been removed. (Bug #28542904)

  References: See also: Bug #29853828, Bug #32426083, Bug #32157120, Bug #28825389.

• Using the allowRootFrom option with the dba.deploySandboxInstance() operation was creating a different remote root account depending on whether MySQL Shell was running in interactive mode or not. Now, the default value of allowRootFrom is consistent between both modes, and the account is created as root@% in both interactive and non-interactive mode. (Bug #27369121)

• When you issue dba.createCluster() and dba.createReplica(), tables are created to store the metadata. If the default storage engine was not InnoDB, these operations could fail. Now, metadata creation operations always use the InnoDB storage engine. (Bug #101446, Bug #32110085)

Functionality Added or Changed

• From MySQL 8.0.24, SQL statements that you issue in MySQL Shell’s SQL mode can be sent to the operating system’s system logging facility (syslog on Unix, or the Windows Event Log). You can select this option by specifying the --syslog command-line option when starting MySQL Shell, or by setting the history.sql.syslog MySQL Shell configuration option. SQL statements that would be excluded from the MySQL Shell code history are also excluded from the system logging facility. (Bug #31995742, Bug #31514599, WL #14358)

• MySQL Shell’s instance dump utility util.dumpInstance(), schema dump utility util.dumpSchemas(), and table dump utility util.dumpTables() can now check for tables that do not contain primary keys. The check is carried out when the ocimds option is enabled for checking compatibility with MySQL Database Service, and an error is reported for every table included in the dump that does not have a primary key. The compatibility option, which implements appropriate measures for compatibility, has two new modification choices to notify MySQL Shell’s dump loading utility to create primary keys in invisible columns for tables that do not have them, or to ignore the missing primary keys. Primary keys are required for MySQL Database Service High Availability, which uses Group Replication. (WL #14506)

Bugs Fixed

• Previously, MySQL Shell retried requests to Oracle Cloud Infrastructure Object Storage a maximum of 5 times, with a 30 second wait in between retries, and a maximum overall wait of 5 minutes. The retry strategy has now been changed to increase the wait window and reduce the possibility of a dump or load operation failing. MySQL Shell now retries a maximum of 10 times, with a 1 minute wait in between retries, and a maximum overall wait of 10 minutes. (Bug #32592962)

• MySQL Shell's instance dump utility util.dumpInstance() stopped with an error if the last schema to be dumped was a schema that contained no tables. The issue has now been fixed. (Bug #32540460)

• MySQL Shell's instance dump utility util.dumpInstance() has been optimized so that it can still be used successfully if there are limitations on the server’s resources such as disk space or the thread stack. To handle such situations, the queries from the utility can be repeated to retrieve smaller chunks of data if required, and file sorting is avoided. (Bug #32528186)

• MySQL Shell's instance dump utility util.dumpInstance() incorrectly removed grants of all privileges to users. The utility now expands GRANT ALL statements in the dump to list all privileges granted on all schemas and tables (*.*), and to list allowed privileges for system schemas. The dump loading utility util.loadDump() now extracts the lists of allowed and revoked global privileges during loading, and strips these from GRANT statements relating to system schemas and to all schemas and tables. (Bug #32526567)

• MySQL Shell's dump loading utility util.loadDump() now grants privileges after all the data is loaded. Previously, an error could occur if the utility tried to grant a privilege on a routine that did not yet exist. (Bug #32526496)

• MySQL Shell's instance dump utility util.dumpInstance(), schema dump utility util.dumpSchemas(), and table dump utility util.dumpTables() could not complete the dump if the gtid_executed system variable or the Information Schema’s COLUMN_STATISTICS table was unavailable. The utilities now display a warning message and log a detailed error message in this situation. These items are not required for a successful dump. (Bug #32515696)

• MySQL Shell’s handling and formatting has been improved for the help text that you provide for dictionary parameters and their options when you register a Python plugin. (Bug #32509309)

• MySQL Shell's instance dump utility util.dumpInstance(), schema dump utility util.dumpSchemas(), and table dump utility util.dumpTables() no longer write a FLUSH TABLES statement to the binary log, as this can interfere with replication. (Bug #32490714)

• From MySQL 8.0.23, MySQL Server supports replication from a source server that does not have GTIDs enabled and does not use GTID-based replication, to a replica that has GTIDs enabled, using the ASSIGN_GTIDS_TO_ANONYMOUS_TRANSACTIONS option of the CHANGE REPLICATION SOURCE TO statement. MySQL Shell's instance dump utility util.dumpInstance(), schema dump utility util.dumpSchemas(), and table dump utility util.dumpTables() now support this functionality by storing the binary log file name and position in the dump metadata, in addition to the gtid_executed GTID set. The additional privilege REPLICATION CLIENT is required in order for the utilities to be able to collect this information, although if the user ID does not have that privilege, the dump continues but without the binary log information.

  The binary log information can be used after loading the dumped data into the replica server to set up replication with a non-GTID source server. MySQL Shell's dump loading utility util.loadDump() prints the binary log and GTID set information from the dump metadata (in YAML format) when you specify the new option showMetadata: true. (Bug #32430402)

• MySQL Shell did not correctly handle an empty array that was added to a collection. The result set normally returned from the server is now skipped in this situation. (Bug #32377134)

• MySQL Shell's instance dump utility util.dumpInstance(), schema dump utility util.dumpSchemas(), and table dump utility util.dumpTables() were unable to chunk table data from a MySQL 5.6 server instance, due to differences in the output of the EXPLAIN SELECT statement for that version. The utilities now account for the difference, and also cache the server version information for faster access. (Bug #32376447)

• MySQL Shell's parallel table import utility util.importTable() set a zero exit code if a non-critical error occurred that did not interrupt the import, such as a directory or file not being found. The utility now sets a non-zero error code instead when the first non-critical error is observed. (Bug #32286186)

• MySQL Shell's upgrade checker utility util.checkForServerUpgrade() now checks for spatial data columns that were originally created in MySQL 5.6. The underlying data type for such columns in MySQL 5.6 does not match their underlying data type in MySQL 8.0, so upgrade of the table is prohibited, and it must be recreated. (Bug #32257211, Bug #101944)

• When MySQL Shell casts a string to a boolean value, the operation is now case insensitive. Previously, the results could differ between platforms. (Bug #32217910)

• When MySQL Shell’s \warnings command was used to show warnings after each statement, warnings were not displayed for a classic MySQL protocol connection. (Bug #32151137)

• MySQL Shell's parallel table import utility util.importTable() now checks whether an uploaded object is a directory, and excludes these from wildcard matching that was specified for files. (Bug #31991122)

• MySQL Shell's dump loading utility util.loadDump() can split oversized chunks of data into smaller chunks for upload. Previously, if loading was stopped then resumed partway through this stage, the rows in the smaller chunks that were already loaded were not taken into account and skipped, which could lead to deadlocks. The utility’s progress file now records the smaller chunks individually so that they can be skipped if the load is stopped and resumed. (Bug #31961688)

• An event that contained a sequence of two semi-colons caused MySQL Shell's instance dump utility util.dumpInstance(), schema dump utility util.dumpSchemas(), and table dump utility util.dumpTables() to enter an infinite loop looking for delimiters. (Bug #31820571)

• The decodeColumns option for MySQL Shell's parallel table import utility util.importTable() could be specified without an accompanying columns option, resulting in the import stopping with an error. (Bug #31407058)

• If a script that was run interactively in MySQL Shell’s Python mode did not have a newline character at the end, and the script ended with a multiline command, MySQL Shell waited for input instead of processing the command. The user had to press Enter to finish running the script, and the last line of the script was incorrectly saved in MySQL Shell’s code history. MySQL Shell now adds an empty line after processing a script input stream, to ensure that this situation does not occur. (Bug #30765725)

• MySQL Shell used a different character set for collations depending on whether X Protocol or classic MySQL protocol was used to connect to the MySQL server instance, leading to inconsistency and in some situations, errors. For MySQL 5.7 instances, MySQL Shell now uses a SET NAMES statement to set all the relevant session system variables to the utf8mb4 character set. For MySQL 8.0 instances, MySQL Shell now sets the collation_connection system variable to the utf8mb4_0900_ai_ci character set. (Bug #30516645)