WL#9357: InnoDB: Upgrade steps for new data dictionary
Affects: Server-8.0 — Status: Complete — Priority: Medium
This WL describes the steps done by InnoDB on upgrade.
'''Functional Requirements''' 1. Upgrade must migrate all InnoDB tables to new DD 2. Upgrade must not corrupt any InnoDB table 3. Upgrade must operate only on 5.7 datadir 4. Upgrade should work on datadir with non-default paths (i.e innodb_data_home_dir, innodb_log_group_home_dir not using defaults) 5. Upgrade should work with datadir with remote and General tablespaces. 6. Upgrade must create InnoDB hidden tables, such as innodb_table_metadata, innodb_ddl_log and all FTS AUX tables in DD system tables 7. After successful upgrade, downgrade is not possible 8. If an incompatibility is detected during upgrade, upgrade should make the datadir usable by 5.7 server 9. Upgrade should work with non-empty undo from 5.7 (historical reasons) 10. Upgrade shouldn't work on a crashed 5.7 datadir. 11. Upgrade shouldn't work with innodb_fast_shutdown=2 on 5.7. This is equivalent to crash.
Upgrade from 5.7 ---------------- Major responsibilities of upgrade 1. Create new dictionary tables 2. Ability to read old dictionary from innodb 3. for each user table, create dictionary entries in new dictionary table 4. upgrading the redo & undo logs 5. Migrating table data from old to new dictionary tables like statistics, events, store procedures 6. Creating SDI for all tablespaces and storing it 7. Recovering from partial upgrade Overall architecture --------------------- 1. Server discovers the tables based on .frm. It knows the SE of each table from .frm 2. Server creates dictionary tables and in-memory dictionary objects (dd::Table, Tablespace etc) for each .frm 3. For a given in-memory dictionary table object, SE will provide the se_private_data 4. For tablespaces, since server doesn't have the list, InnoDB will register tablespace objects in dictionary 5. Server transfers data from old tables to new tables. For example, statistics, events, Store Procedures, etc 6. Server materializes dictionary entries (to disk). Flush modifications to disk 7. Server creates new undo tablespaces (new format). Enables undo & redo logging 9. Server creates SDI for each tablespace NOTE: On detecting incompatibilities, We would like to give user ability to start 5.7 server and fix incompatibilites. To achieve this, InnoDB will make sure redo is empty, change the redo version to 1. For undo, we make sure that old undo tablespaces are deleted only after successful table migration. Detailed Steps: --------------- 1. Start 8.0 server on 5.7 data directory 2. Create upgrade mode - mysql.ibd existence check -> Runtime Task - Server informs InnoDB that it is upgrade mode ->Runtime Task i.e. uses DICT_INIT_CREATE_MISSING_FILES as dictionary init mode 3. Boot InnoDB engine -> InnoDB Task - check if undo log is empty or not - creates mysql.ibd tablespace - creates dict_table_t objects for all SYS_* (keep 5.7 dict_boot() as is) 4. Create version Table -> Runtime Task 5. Insert Zero as version in dictionary to mark that we are upgrading -> Runtime Task 6. Create all *NEW* dictonary tables (mysql.columns, indexes, etc) 7. Create schema objects in new dictionary -> Runtime Task 8. Create tablespace entries in new dictionary - Since server has no knowledge about tablespaces, InnoDB has to do this - Introduce API to initiate this action -> Runtime Task(APIs to be defined) - InnoDB will fetch all tablespace object from SYS_TABLESPACES and register dd::Tablespace objects 9. For each InnoDB Table (as determined by .frm) - create in-memory dd::Table Object -> Runtime Task - this dd::Table object will have the following meta data filled indexes, columns, index fields. - Missing data is the se_private_data, and information stored in InnoDB dictionary (like Foreign Keys, Tablespaces) - Assign new table_id from in-memory counter. First 256 (lets say) table_ids are reserved for dictionary tables. So the table_id of user tables start from 257+) - call the API to fill se_private_data for Table, Indexes, etc -> Runtime Task (APIs to be defined) - Use the table_name to create InnoDB Table Object (dict_table_t) -> InnoDB Task dict_table_t for SYS_* exists. Example: call dict_load_open_by_name(t1) - From dict_table_t, fill all data required by server into dd::Table-> InnoDB Task - Write the in-memory dd::Table object to Dictionary tables (on-disk) -> Runtime Task - call new API which will flush dirty buffer pool pages to disk 10. Read from old statistics table and insert into new Statistics Table -> Runtime Task 11. Upgrade schema, i.e. create dictionary entry for schema. -> Runtime Task 12. Upgrade views without fixing view dependency. -> Runtime Task 13. Upgrade Events. -> Runtime Task 14. Upgrade SP/SF. -> Runtime Task 15. Fix view dependency. -> Runtime Task 16. Create a marking that we have migrated all meta data from other storage systems to dictionary. -> Runtime Task We will change .ibd files after this stage. Even if server is killed after this step, upgrade will only roll forward and complete only steps after this (step 17 onwards). - Inserts a new number into version Table? - call API to flush dirty buffer pool pages to disk -> RunTime & InnoDB 17. ask SE to upgrade and undo logs. Introduce new API. - API part - Runtime - upgrading redo log & create new undo tablespaces -> InnoDB 18. Iterate over all InnoDB tablespaces to create SDI entries in .ibd files. - Server now has all Tablespace information (step 9) - For each tablespace -> Runtime Task - ask SE to create SDI index -> Runtime Task - create SDI object (from dd::Table, Schema, Indexes, etc) -> Runtime Task - use sdi_set() to insert SDI object -> Runtime Task 19. Mark correct version number in dictionary to mark completion of upgrade for next restart. -> Runtime Task - call API to flush dirty buffer pool pages to disk (optional) -> RunTime
Copyright (c) 2000, 2019, Oracle Corporation and/or its affiliates. All rights reserved.