WL#8845: InnoDB: Redo log format version identifier

InnoDB has several times changed its redo log format by introducing
new redo log record types. Format changes would lead to misleading
reports of redo log corruption when processing an individual redo log record.

In the redo log header (start of the ib_logfile0 file), we will introduce
a format version identifier and textual representation of the software
version that created the redo log files.

Furthermore, we change the checksum of redo log checkpoint pages, so that
older versions of MySQL will refuse to start up on redo log files that
were created with MySQL 5.7.9 or 5.8 (which include this fix).

We will also remove a number of unused fields from the redo log
header and checkpoint pages (pages 0, 1, and 3).

Some tests will be expanded, because with this fix,
the 5.7.9 must refuse to start up with old (not format-tagged)
redo log files from MySQL 5.6 and earlier, unless those redo logs are clean.

The MySQL 5.8 server will refuse to start up if the redo log was not
created by MySQL 5.7.9 or later. If the redo log from MySQL 5.7.9 is not
clean, MySQL 5.8 will refuse startup. Downgrade from MySQL 5.8 to
MySQL 5.7.9 is refused, unless the user manually removes the ib_logfile*
files (hopefully, after a clean shutdown of MySQL 5.8).

We will also replace the configuration parameter
innodb_log_checksum_algorithm with the Boolean parameter
innodb_log_checksums.
We make CRC-32C the only checksum on the InnoDB redo log pages when
innodb_log_checksums=ON (the default). Checksums on the header page and the
checkpoint pages are never disabled.

FR0. Upgrade of one major version (e.g., 5.6 to 5.7, but not 5.6 to 5.8) with
respect to this feature will be possible after a clean shutdown
(innodb_fast_shutdown=1 or 0, but not 2). On upgrade, the clean redo log
will be automatically replaced with a new-format one.

FR1. Old versions of MySQL will refuse to start up if the redo log is
in a newer format, even if it is clean (after proper shutdown).
The error message about this will point to the documentation, e.g.,
http://dev.mysql.com/doc/refman/5.7/en/upgrading-downgrading.html
where we could add more advice, such as suggesting a slow shutdown
(innodb_fast_shutdown=0) of the newer server version, followed by
removal of ib_logfile*

FR2. Newer versions of MySQL will refuse to start up if the redo log
is dirty and in an older format, with a clear error message.

FR3. Post-WL8845 versions of MySQL may refuse to start up unless:
FR3.1: the redo log format tag is the current one
FR3.2: the redo log format tag is the previous one, and the log is clean

Example: 5.8 will refuse to start up on clean (and dirty) 5.6 redo log files.
MySQL 5.7 will refuse to start up on 5.8 redo log files.
5.(n+1) must start up on clean 5.n redo log files when n>6.
(5.7 must start up on clean 5.6 redo log files;
 5.8 must start up on clean 5.7.9 redo log files.)

Impact on future versions that use the file name "ib_logfile0":

FV1: At byte offset 0..3 of the file "ib_logfile0" there must be
a monotonically increasing 32-bit big-endian unsigned number that
identifies the redo log format.

FV2: At byte offset 16, there must be an ASCII buffer for identifying the log
file creator to the user in case startup is refused due to incompatibility
(FR1, FR2, FR3). This string must be NUL-terminated and should not exceed
32 bytes of length. If the length is exceeded, older versions may truncate
the string to 32 bytes when displaying the incompatibility message.

FV3: At byte offset 508..511 of the file "ib_logfile0", there must be a
CRC-32C checksum of the bytes 0..508.

Example: If a future version uses a different redo log block size than
the current 512 bytes, it should include the CRC-32C checksum (FV3)
for the first 508 bytes at the old location 508..511. It should also
initialize the bytes 0..3 and write a NUL-terminated ASCII string at
offset 16. Anything else on the page can be freely repurposed, and the
new server version could put its own native checksum somewhere else.
For the rest of the redo log file, only the native checksum could
be used; old incompatible server versions would not read anything
beyond the first 512 bytes.

The configuration parameter innodb_log_checksum_algorithm
is replaced with the Boolean parameter innodb_log_checksums,
with the following mapping:

innodb_log_checksums=ON (default) corresponds to
innodb_log_checksum_algorithm=strict_crc32

innodb_log_checksums=OFF roughly corresponds to
innodb_log_checksum_algorithm=none
with the exception that when reading the redo log, the contents of
the checksum field will be ignored entirely (it can be anything, not
necessarily the magic 0xdeadbeef value nor a valid CRC-32C checksum).

Code that dealt with the initial buggy byte-order-dependent (‘legacy’) CRC32-C
computation is removed.

Code that writes the slow and weak ‘innodb’ checksum on redo log record
pages is removed, with the exception (3) below.

The InnoDB redo log format is changed as follows:

All 512-byte pages will by default use the CRC-32C computed on the first
508 bytes, stored at the last 4 bytes. On any other pages than 0,1,3,
this checksum can be disabled (the bytes at 508..511 will be the
constant 0xdeadbeef) if innodb_log_checksum=off.

Page 0 (header page): This page was largely ignored by old versions of MySQL.
The main use is the 32-byte field at offset 16 that can contain a 26-byte
NUL-terminated string that causes InnoDB to display a message at startup if
that string starts with "ibbackup ". We will keep this field at its
current location and length. If the stored string is shorter than 32 bytes,
it will have to be terminated by a NUL byte. When creating files, InnoDB
will start writing a clear-text version identifier message in this field,
so that when encountering an incompatible format-tagged redo log file,
a post-WL#8845 server can display a user-readable message that identifies
the server version that created the redo log.

The actual string length stored inside the 32-byte buffer can be anything
from 0 to 32 bytes (that is, if the string is not terminated by NUL, it
will be truncated to 32 bytes when displayed). This string basically only
matters for the purpose of displaying a message at startup. Before WL#8455,
the string was only displayed if it started with the prefix "ibbackup ".
Such strings would be 26 bytes long. Before WL#8455, the string could also
be four spaces, or four spaces overwriting the start of the
"ibbackup " prefix, e.g., "    ckup ".

With WL#8455, the string will be something like "MySQL 5.7.9", so the length
would be 11 bytes (12 when we go into double-digit minor version numbers).
On startup, this message would be displayed when refusing startup due to
incompatible format-tagged redo log file, e.g., when 5.7 is started up on
5.8 redo log. The display code tolerates anything up to 32 bytes, and
truncates anything longer to 32 bytes.

The first 4 bytes were always written as 0. We will repurpose that for a
version identifier, starting with the value 1:

/** Log file header format identifier (32-bit unsigned big-endian integer).
This used to be called LOG_GROUP_ID and always written as 0,
because InnoDB never supported more than one copy of the redo log. */
#define LOG_HEADER_FORMAT	0
/** 4 unused (zero-initialized) bytes. In format version 0, the
LOG_FILE_START_LSN started here, 4 bytes earlier. */
#define LOG_HEADER_PAD1		4
/** LSN of the start of data in this log file (with format version 1;
in format version 0, it was called LOG_FILE_START_LSN and at offset 4). */
#define LOG_HEADER_START_LSN	8
/** A null-terminated string which will contain either the string 'ibbackup'
and the creation time if the log file was created by mysqlbackup --restore,
or the MySQL version that created the redo log file. */
#define LOG_HEADER_CREATOR	16
/** End of the log file creator field. */
#define LOG_HEADER_CREATOR_END	(LOG_HEADER_CREATOR + 32)
/** Contents of the LOG_HEADER_CREATOR field */
#define LOG_HEADER_CREATOR_CURRENT	"MySQL " INNODB_VERSION_STR

/** Current redo log format identifier. Stored in LOG_HEADER_FORMAT. */
#define LOG_HEADER_FORMAT_CURRENT	1

There used to be no checksum on page 0, and recovery does not really care
about the page contents. A lot of unused fields were removed, and
LOG_GROUP_START_LSN is moved. The unused bytes will be zero-initialized.

It is safe to completely change the format of page 0, because the post-WL#8845
server will refuse to start up if the first 4 bytes are 0 (or anything else than
LOG_GROUP_FORMAT_CURRENT). The first 4 bytes always were 0 in the past,
and there was no CRC-32C checksum in the past, so we would very likely refuse
startup if the first page was garbage.

To prevent pre-WL#8845 servers from starting up with post-WL#8845
format-tagged redo log files, we must change the format of the two
log checkpoint pages (pages 1 and 3) as well. The validity of these
pages was checked based on a weak checksum on some bytes. These bytes
will always be written as 0 in the WL#8845 format. Some fields will
also be moved around, but that is safe for 2 reasons:

(1) post-WL#8845 MySQL will not read the checkpoint pages
if page 0 carries an incompatible format tag or an invalid checksum.
(See exception (3) below, to allow startup with clean redo log files.)

(2) pre-WL#8845 MySQL will find invalid checksums
(all bytes at 288..295 are 0) on both checkpoint pages and refuse startup,
because there is no valid checkpoint.

The latter property would not hold in the improbable (2^-63 or 10^-19)
case that both checksums happen to be valid (0) by accident on either
redo log checksum page.
In this improbable case, because WL#8845 will change the format of the
checkpoint header fields as well, the redo log scanning should be
started from a bogus LSN or byte offset, likely reporting corruption
very soon, because already the first redo log record byte allows only
some 20% of the possible 256 byte values.

(3) MySQL with WL#8845 must allow startup if the pre-WL#8845 redo log
is in clean state. This will happen when:
(3a) the first 4 bytes of ib_logfile0 are 0
(3b) at least one of the 2 checksum pages has valid old-style checksum
(3c) the redo log record page pointed to by the checkpoint has valid ‘innodb’
checksum and is empty
If all these conditions are met, we will allow startup.

The target is that WL#8845 goes to MySQL 5.7.9, and a follow-up of it goes
to MySQL 5.8.0. In mysql-trunk-wl8845, the code for (3) will be
removed. So, upgrading to 5.8 from earlier versions than 5.7.9 will require
either a clean shutdown on that old server, followed by deleting the redo log
files, or a prior upgrade to 5.7.9+ before upgrading to 5.8.

These are the changes to mysql-5.7.
Some of these functions are renamed and replaced further for 5.8;
search for 5.8 below.

innodb_log_checksums_func_update(), innodb_log_checksums_update():
Update triggers for the new global Boolean variable innodb_log_checksums.

innodb_log_checksum_func_update(), innodb_log_checksum_algorithm_update():
Removed along with the global parameter innodb_log_checksum_algorithm.

Removed definitions:
LOG_MAX_N_GROUPS
log_group_read_checkpoint_info()
log_checkpoint_get_nth_group_info()
log_checkpoint_set_nth_group_info()
log_block_calc_checksum_innodb()
log_block_calc_checksum_crc32_legacy_big_endian()
recv_check_cp_is_consistent()
log_block_checksum_weak_validation()
log_block_checksum_what_matches()
log_block_checksum_fail_fatal()
log_block_checksum_is_ok_or_old_format()
LOG_CHECKPOINT_OFFSET_LOW32
LOG_CHECKPOINT_ARCHIVED_LSN
LOG_CHECKPOINT_GROUP_ARRAY
LOG_CHECKPOINT_ARCHIVED_FILE_NO
LOG_CHECKPOINT_ARCHIVED_OFFSET
LOG_CHECKPOINT_ARRAY_END
LOG_CHECKPOINT_CHECKSUM_1
LOG_CHECKPOINT_CHECKSUM_2
LOG_CHECKPOINT_FSP_FREE_LIMIT
LOG_CHECKPOINT_FSP_MAGIC_N
LOG_CHECKPOINT_FSP_MAGIC_N_VAL
LOG_CHECKPOINT_OFFSET_HIGH32
LOG_CHECKPOINT_SIZE
LOG_GROUP_ID
LOG_FILE_START_LSN
LOG_FILE_NO
LOG_FILE_WAS_CREATED_BY_HOT_BACKUP
LOG_FILE_ARCH_COMPLETED
LOG_FILE_END_LSN

Changed definitions:
LOG_CHECKPOINT_LOG_BUF_SIZE

Added definitions:
innodb_log_checksums: The current value of the SET GLOBAL variable.
LOG_CHECKPOINT_OFFSET
LOG_HEADER_FORMAT
LOG_HEADER_PAD1 (unused 4 bytes, zero-initialized)
LOG_HEADER_START_LSN
LOG_HEADER_CREATOR (renamed from LOG_FILE_WAS_CREATED_BY_HOT_BACKUP)
LOG_HEADER_CREATOR_END
LOG_HEADER_CREATOR_CURRENT
LOG_HEADER_FORMAT_CURRENT
log_group_t::format

log_block_calc_checksum_format_0(): Renamed from
log_block_calc_checksum_innodb(). This is only used when upgrading the
redo log from non-tagged format.

recv_find_max_checkpoint_0(): New function, used when upgrading the
redo log from non-tagged format.

recv_log_format_0_recover(): New function, used when upgrading the
redo log from non-tagged format. Checks if the redo log is clean.

log_group_header_read(): Replaces log_group_read_checkpoint_info().
Also used for reading the log header page (page 0).

recv_check_log_header_checksum(): Replaces recv_check_cp_is_consistent().
Also used for checking the log header page (page 0).

log_block_checksum_is_ok(): Checks a log block checksum. It must either
be CRC-32C, or we must have innodb_log_checksums=OFF.

Changed functions:

log_group_file_header_flush(): Always zero-initialize the buffer
and initialize all LOG_HEADER_ fields.

log_group_checkpoint(): Zero-initialize the checkpoint buffer,
and write the checkpoint in the new format, with CRC-32C checksum.

recv_find_max_checkpoint(): Support both the old log format
(if the old-format redo log is logically empty) and the new format.

recv_scan_log_recs(): Clean up the logic a bit. Display a message
every time when encountering (and terminating log parsing due to)
invalid log blocks.

recv_recovery_from_checkpoint_start(): Replace the whole "ibbackup" label.
Check if a log upgrade or a normal recovery is needed.

srv_prepare_to_delete_redo_log_files(): Display a message about
upgrading the redo log.

Changes to startup:

* If we are going to upgrade the redo log, we must avoid writing any
new redo log records before we have replaced the redo log.
Thus, dict_check_sys_tablespaces() and dict_check_sys_tables() must
avoid updating SYS_DATAFILES if we are going to upgrade the redo log.

* Likewise, dict_create_or_check_sys_virtual() must not modify anything
if we are running in --innodb-read-only or --innodb-force-recovery=6 mode.

----
Changes in MySQL 5.8:

MySQL 5.8 will refuse to start up with redo log from before MySQL 5.7.9.
It will support startup from 5.7.9 when the redo log is clean.

Removed definitions:

log_block_calc_checksum_format_0()
recv_find_max_checkpoint_0()

Added definitions:

recv_log_recover_5_7()