WL#8845: InnoDB: Redo log format version identifier
InnoDB has several times changed its redo log format by introducing new redo log record types. Format changes would lead to misleading reports of redo log corruption when processing an individual redo log record. In the redo log header (start of the ib_logfile0 file), we will introduce a format version identifier and textual representation of the software version that created the redo log files. Furthermore, we change the checksum of redo log checkpoint pages, so that older versions of MySQL will refuse to start up on redo log files that were created with MySQL 5.7.9 or 5.8 (which include this fix). We will also remove a number of unused fields from the redo log header and checkpoint pages (pages 0, 1, and 3). Some tests will be expanded, because with this fix, the 5.7.9 must refuse to start up with old (not format-tagged) redo log files from MySQL 5.6 and earlier, unless those redo logs are clean. The MySQL 5.8 server will refuse to start up if the redo log was not created by MySQL 5.7.9 or later. If the redo log from MySQL 5.7.9 is not clean, MySQL 5.8 will refuse startup. Downgrade from MySQL 5.8 to MySQL 5.7.9 is refused, unless the user manually removes the ib_logfile* files (hopefully, after a clean shutdown of MySQL 5.8). We will also replace the configuration parameter innodb_log_checksum_algorithm with the Boolean parameter innodb_log_checksums. We make CRC-32C the only checksum on the InnoDB redo log pages when innodb_log_checksums=ON (the default). Checksums on the header page and the checkpoint pages are never disabled.
FR0. Upgrade of one major version (e.g., 5.6 to 5.7, but not 5.6 to 5.8) with respect to this feature will be possible after a clean shutdown (innodb_fast_shutdown=1 or 0, but not 2). On upgrade, the clean redo log will be automatically replaced with a new-format one. FR1. Old versions of MySQL will refuse to start up if the redo log is in a newer format, even if it is clean (after proper shutdown). The error message about this will point to the documentation, e.g., http://dev.mysql.com/doc/refman/5.7/en/upgrading-downgrading.html where we could add more advice, such as suggesting a slow shutdown (innodb_fast_shutdown=0) of the newer server version, followed by removal of ib_logfile* FR2. Newer versions of MySQL will refuse to start up if the redo log is dirty and in an older format, with a clear error message. FR3. Post-WL8845 versions of MySQL may refuse to start up unless: FR3.1: the redo log format tag is the current one FR3.2: the redo log format tag is the previous one, and the log is clean Example: 5.8 will refuse to start up on clean (and dirty) 5.6 redo log files. MySQL 5.7 will refuse to start up on 5.8 redo log files. 5.(n+1) must start up on clean 5.n redo log files when n>6. (5.7 must start up on clean 5.6 redo log files; 5.8 must start up on clean 5.7.9 redo log files.) Impact on future versions that use the file name "ib_logfile0": FV1: At byte offset 0..3 of the file "ib_logfile0" there must be a monotonically increasing 32-bit big-endian unsigned number that identifies the redo log format. FV2: At byte offset 16, there must be an ASCII buffer for identifying the log file creator to the user in case startup is refused due to incompatibility (FR1, FR2, FR3). This string must be NUL-terminated and should not exceed 32 bytes of length. If the length is exceeded, older versions may truncate the string to 32 bytes when displaying the incompatibility message. FV3: At byte offset 508..511 of the file "ib_logfile0", there must be a CRC-32C checksum of the bytes 0..508. Example: If a future version uses a different redo log block size than the current 512 bytes, it should include the CRC-32C checksum (FV3) for the first 508 bytes at the old location 508..511. It should also initialize the bytes 0..3 and write a NUL-terminated ASCII string at offset 16. Anything else on the page can be freely repurposed, and the new server version could put its own native checksum somewhere else. For the rest of the redo log file, only the native checksum could be used; old incompatible server versions would not read anything beyond the first 512 bytes.
WL#6501: InnoDB: Make Internal InnoDB TRUNCATE TABLE statement to be atomic for single tablespace
WL#7142: InnoDB: Simplify tablespace discovery during crash recovery
WL#7142: InnoDB: Simplify tablespace discovery during crash recovery
The configuration parameter innodb_log_checksum_algorithm is replaced with the Boolean parameter innodb_log_checksums, with the following mapping: innodb_log_checksums=ON (default) corresponds to innodb_log_checksum_algorithm=strict_crc32 innodb_log_checksums=OFF roughly corresponds to innodb_log_checksum_algorithm=none with the exception that when reading the redo log, the contents of the checksum field will be ignored entirely (it can be anything, not necessarily the magic 0xdeadbeef value nor a valid CRC-32C checksum). Code that dealt with the initial buggy byte-order-dependent (‘legacy’) CRC32-C computation is removed. Code that writes the slow and weak ‘innodb’ checksum on redo log record pages is removed, with the exception (3) below. The InnoDB redo log format is changed as follows: All 512-byte pages will by default use the CRC-32C computed on the first 508 bytes, stored at the last 4 bytes. On any other pages than 0,1,3, this checksum can be disabled (the bytes at 508..511 will be the constant 0xdeadbeef) if innodb_log_checksum=off. Page 0 (header page): This page was largely ignored by old versions of MySQL. The main use is the 32-byte field at offset 16 that can contain a 26-byte NUL-terminated string that causes InnoDB to display a message at startup if that string starts with "ibbackup ". We will keep this field at its current location and length. If the stored string is shorter than 32 bytes, it will have to be terminated by a NUL byte. When creating files, InnoDB will start writing a clear-text version identifier message in this field, so that when encountering an incompatible format-tagged redo log file, a post-WL#8845 server can display a user-readable message that identifies the server version that created the redo log. The actual string length stored inside the 32-byte buffer can be anything from 0 to 32 bytes (that is, if the string is not terminated by NUL, it will be truncated to 32 bytes when displayed). This string basically only matters for the purpose of displaying a message at startup. Before WL#8455, the string was only displayed if it started with the prefix "ibbackup ". Such strings would be 26 bytes long. Before WL#8455, the string could also be four spaces, or four spaces overwriting the start of the "ibbackup " prefix, e.g., " ckup ". With WL#8455, the string will be something like "MySQL 5.7.9", so the length would be 11 bytes (12 when we go into double-digit minor version numbers). On startup, this message would be displayed when refusing startup due to incompatible format-tagged redo log file, e.g., when 5.7 is started up on 5.8 redo log. The display code tolerates anything up to 32 bytes, and truncates anything longer to 32 bytes. The first 4 bytes were always written as 0. We will repurpose that for a version identifier, starting with the value 1: /** Log file header format identifier (32-bit unsigned big-endian integer). This used to be called LOG_GROUP_ID and always written as 0, because InnoDB never supported more than one copy of the redo log. */ #define LOG_HEADER_FORMAT 0 /** 4 unused (zero-initialized) bytes. In format version 0, the LOG_FILE_START_LSN started here, 4 bytes earlier. */ #define LOG_HEADER_PAD1 4 /** LSN of the start of data in this log file (with format version 1; in format version 0, it was called LOG_FILE_START_LSN and at offset 4). */ #define LOG_HEADER_START_LSN 8 /** A null-terminated string which will contain either the string 'ibbackup' and the creation time if the log file was created by mysqlbackup --restore, or the MySQL version that created the redo log file. */ #define LOG_HEADER_CREATOR 16 /** End of the log file creator field. */ #define LOG_HEADER_CREATOR_END (LOG_HEADER_CREATOR + 32) /** Contents of the LOG_HEADER_CREATOR field */ #define LOG_HEADER_CREATOR_CURRENT "MySQL " INNODB_VERSION_STR /** Current redo log format identifier. Stored in LOG_HEADER_FORMAT. */ #define LOG_HEADER_FORMAT_CURRENT 1 There used to be no checksum on page 0, and recovery does not really care about the page contents. A lot of unused fields were removed, and LOG_GROUP_START_LSN is moved. The unused bytes will be zero-initialized. It is safe to completely change the format of page 0, because the post-WL#8845 server will refuse to start up if the first 4 bytes are 0 (or anything else than LOG_GROUP_FORMAT_CURRENT). The first 4 bytes always were 0 in the past, and there was no CRC-32C checksum in the past, so we would very likely refuse startup if the first page was garbage. To prevent pre-WL#8845 servers from starting up with post-WL#8845 format-tagged redo log files, we must change the format of the two log checkpoint pages (pages 1 and 3) as well. The validity of these pages was checked based on a weak checksum on some bytes. These bytes will always be written as 0 in the WL#8845 format. Some fields will also be moved around, but that is safe for 2 reasons: (1) post-WL#8845 MySQL will not read the checkpoint pages if page 0 carries an incompatible format tag or an invalid checksum. (See exception (3) below, to allow startup with clean redo log files.) (2) pre-WL#8845 MySQL will find invalid checksums (all bytes at 288..295 are 0) on both checkpoint pages and refuse startup, because there is no valid checkpoint. The latter property would not hold in the improbable (2^-63 or 10^-19) case that both checksums happen to be valid (0) by accident on either redo log checksum page. In this improbable case, because WL#8845 will change the format of the checkpoint header fields as well, the redo log scanning should be started from a bogus LSN or byte offset, likely reporting corruption very soon, because already the first redo log record byte allows only some 20% of the possible 256 byte values. (3) MySQL with WL#8845 must allow startup if the pre-WL#8845 redo log is in clean state. This will happen when: (3a) the first 4 bytes of ib_logfile0 are 0 (3b) at least one of the 2 checksum pages has valid old-style checksum (3c) the redo log record page pointed to by the checkpoint has valid ‘innodb’ checksum and is empty If all these conditions are met, we will allow startup. The target is that WL#8845 goes to MySQL 5.7.9, and a follow-up of it goes to MySQL 5.8.0. In mysql-trunk-wl8845, the code for (3) will be removed. So, upgrading to 5.8 from earlier versions than 5.7.9 will require either a clean shutdown on that old server, followed by deleting the redo log files, or a prior upgrade to 5.7.9+ before upgrading to 5.8.
These are the changes to mysql-5.7. Some of these functions are renamed and replaced further for 5.8; search for 5.8 below. innodb_log_checksums_func_update(), innodb_log_checksums_update(): Update triggers for the new global Boolean variable innodb_log_checksums. innodb_log_checksum_func_update(), innodb_log_checksum_algorithm_update(): Removed along with the global parameter innodb_log_checksum_algorithm. Removed definitions: LOG_MAX_N_GROUPS log_group_read_checkpoint_info() log_checkpoint_get_nth_group_info() log_checkpoint_set_nth_group_info() log_block_calc_checksum_innodb() log_block_calc_checksum_crc32_legacy_big_endian() recv_check_cp_is_consistent() log_block_checksum_weak_validation() log_block_checksum_what_matches() log_block_checksum_fail_fatal() log_block_checksum_is_ok_or_old_format() LOG_CHECKPOINT_OFFSET_LOW32 LOG_CHECKPOINT_ARCHIVED_LSN LOG_CHECKPOINT_GROUP_ARRAY LOG_CHECKPOINT_ARCHIVED_FILE_NO LOG_CHECKPOINT_ARCHIVED_OFFSET LOG_CHECKPOINT_ARRAY_END LOG_CHECKPOINT_CHECKSUM_1 LOG_CHECKPOINT_CHECKSUM_2 LOG_CHECKPOINT_FSP_FREE_LIMIT LOG_CHECKPOINT_FSP_MAGIC_N LOG_CHECKPOINT_FSP_MAGIC_N_VAL LOG_CHECKPOINT_OFFSET_HIGH32 LOG_CHECKPOINT_SIZE LOG_GROUP_ID LOG_FILE_START_LSN LOG_FILE_NO LOG_FILE_WAS_CREATED_BY_HOT_BACKUP LOG_FILE_ARCH_COMPLETED LOG_FILE_END_LSN Changed definitions: LOG_CHECKPOINT_LOG_BUF_SIZE Added definitions: innodb_log_checksums: The current value of the SET GLOBAL variable. LOG_CHECKPOINT_OFFSET LOG_HEADER_FORMAT LOG_HEADER_PAD1 (unused 4 bytes, zero-initialized) LOG_HEADER_START_LSN LOG_HEADER_CREATOR (renamed from LOG_FILE_WAS_CREATED_BY_HOT_BACKUP) LOG_HEADER_CREATOR_END LOG_HEADER_CREATOR_CURRENT LOG_HEADER_FORMAT_CURRENT log_group_t::format log_block_calc_checksum_format_0(): Renamed from log_block_calc_checksum_innodb(). This is only used when upgrading the redo log from non-tagged format. recv_find_max_checkpoint_0(): New function, used when upgrading the redo log from non-tagged format. recv_log_format_0_recover(): New function, used when upgrading the redo log from non-tagged format. Checks if the redo log is clean. log_group_header_read(): Replaces log_group_read_checkpoint_info(). Also used for reading the log header page (page 0). recv_check_log_header_checksum(): Replaces recv_check_cp_is_consistent(). Also used for checking the log header page (page 0). log_block_checksum_is_ok(): Checks a log block checksum. It must either be CRC-32C, or we must have innodb_log_checksums=OFF. Changed functions: log_group_file_header_flush(): Always zero-initialize the buffer and initialize all LOG_HEADER_ fields. log_group_checkpoint(): Zero-initialize the checkpoint buffer, and write the checkpoint in the new format, with CRC-32C checksum. recv_find_max_checkpoint(): Support both the old log format (if the old-format redo log is logically empty) and the new format. recv_scan_log_recs(): Clean up the logic a bit. Display a message every time when encountering (and terminating log parsing due to) invalid log blocks. recv_recovery_from_checkpoint_start(): Replace the whole "ibbackup" label. Check if a log upgrade or a normal recovery is needed. srv_prepare_to_delete_redo_log_files(): Display a message about upgrading the redo log. Changes to startup: * If we are going to upgrade the redo log, we must avoid writing any new redo log records before we have replaced the redo log. Thus, dict_check_sys_tablespaces() and dict_check_sys_tables() must avoid updating SYS_DATAFILES if we are going to upgrade the redo log. * Likewise, dict_create_or_check_sys_virtual() must not modify anything if we are running in --innodb-read-only or --innodb-force-recovery=6 mode. ---- Changes in MySQL 5.8: MySQL 5.8 will refuse to start up with redo log from before MySQL 5.7.9. It will support startup from 5.7.9 when the redo log is clean. Removed definitions: log_block_calc_checksum_format_0() recv_find_max_checkpoint_0() Added definitions: recv_log_recover_5_7()
Copyright (c) 2000, 2020, Oracle Corporation and/or its affiliates. All rights reserved.