WL#9357: InnoDB: Upgrade steps for new data dictionary

This WL describes the steps done by InnoDB on upgrade.

'''Functional Requirements'''

1. Upgrade must migrate all InnoDB tables to new DD
2. Upgrade must not corrupt any InnoDB table
3. Upgrade must operate only on 5.7 datadir
4. Upgrade should work on datadir with non-default paths (i.e
   innodb_data_home_dir, innodb_log_group_home_dir not using defaults)
5. Upgrade should work with datadir with remote and General tablespaces.
6. Upgrade must create InnoDB hidden tables, such as innodb_table_metadata, 
   innodb_ddl_log and all FTS AUX tables in DD system tables
7. After successful upgrade, downgrade is not possible
8. If an incompatibility is detected during upgrade, upgrade should make
   the datadir usable by 5.7 server
9. Upgrade should work with non-empty undo from 5.7 (historical reasons)
10. Upgrade shouldn't work on a crashed 5.7 datadir.
11. Upgrade shouldn't work with innodb_fast_shutdown=2 on 5.7. This is equivalent
    to crash.

Upgrade from 5.7
----------------
Major responsibilities of upgrade

1. Create new dictionary tables

2. Ability to read old dictionary from innodb

3. for each user table, create dictionary entries in new dictionary table

4. upgrading the redo & undo logs

5. Migrating table data from old to new dictionary tables like statistics,
   events, store procedures

6. Creating SDI for all tablespaces and storing it

7. Recovering from partial upgrade

Overall architecture
---------------------

1. Server discovers the tables based on .frm. It knows the SE of each
   table from .frm

2. Server creates dictionary tables and in-memory dictionary objects
  (dd::Table, Tablespace etc) for each .frm

3. For a given in-memory dictionary table object, SE will provide the
   se_private_data

4. For tablespaces, since server doesn't have the list, InnoDB will
   register tablespace objects in dictionary

5. Server transfers data from old tables to new tables. For example,
   statistics, events, Store Procedures, etc

6. Server materializes dictionary entries (to disk). Flush modifications
   to disk

7. Server creates new undo tablespaces (new format). 
   Enables undo & redo logging

9. Server creates SDI for each tablespace

NOTE: On detecting incompatibilities, We would like to give user ability to
start 5.7 server and fix incompatibilites. To achieve this, InnoDB will make 
sure redo is empty, change the redo version to 1. For undo, we make sure that
old undo tablespaces are deleted only after successful table migration.

Detailed Steps:
---------------
1.  Start 8.0 server on 5.7 data directory

2.  Create upgrade mode
    - mysql.ibd existence check  -> Runtime Task
    - Server informs InnoDB that it is upgrade mode ->Runtime Task
      i.e. uses DICT_INIT_CREATE_MISSING_FILES as dictionary init
      mode

3.  Boot InnoDB engine -> InnoDB Task
    - check if undo log is empty or not
    - creates mysql.ibd tablespace
    - creates dict_table_t objects for all SYS_* (keep 5.7 dict_boot() as is)

4.  Create version Table -> Runtime Task

5.  Insert Zero as version in dictionary to mark that we are
    upgrading -> Runtime Task

6.  Create all *NEW* dictonary tables (mysql.columns, indexes, etc)

7.  Create schema objects in new dictionary  -> Runtime Task

8.  Create tablespace entries in new dictionary
    - Since server has no knowledge about tablespaces, InnoDB has to do
      this

    - Introduce API to initiate this action -> Runtime Task(APIs to be defined)

    -  InnoDB will fetch all tablespace object from SYS_TABLESPACES and
       register dd::Tablespace objects

9.  For each InnoDB Table (as determined by .frm)

    - create in-memory dd::Table Object  -> Runtime Task
      - this dd::Table object will have the following meta data filled
	indexes, columns, index fields.

      - Missing data is the se_private_data, and information stored in
        InnoDB dictionary (like Foreign Keys, Tablespaces)

      - Assign new table_id from in-memory counter. First 256 (lets say)
        table_ids are reserved for dictionary tables. So the table_id of
        user tables start from 257+)

    - call the API to fill se_private_data for Table,
      Indexes, etc -> Runtime Task (APIs to be defined)

    - Use the table_name to create InnoDB Table Object
      (dict_table_t) -> InnoDB Task

       dict_table_t for SYS_* exists. Example: call dict_load_open_by_name(t1)

    - From dict_table_t, fill all data required by server into
      dd::Table-> InnoDB Task

    - Write the in-memory dd::Table object to Dictionary tables
     (on-disk) -> Runtime Task
      - call new API which will flush dirty buffer pool pages to disk

10. Read from old statistics table and insert into new Statistics
    Table -> Runtime Task

11. Upgrade schema, i.e. create dictionary entry for schema. -> Runtime Task

12. Upgrade views without fixing view dependency.  -> Runtime Task

13. Upgrade Events.  -> Runtime Task

14. Upgrade SP/SF.  -> Runtime Task

15. Fix view dependency. -> Runtime Task

16. Create a marking that we have migrated all meta data from other storage
    systems to dictionary. -> Runtime Task
	
    We will change .ibd files after this stage. Even if server is killed after
    this step, upgrade will only roll forward and
    complete only steps after this (step 17 onwards).

    - Inserts a new number into version Table?

    - call API to flush dirty buffer pool pages to disk -> RunTime & InnoDB

17. ask SE to upgrade and undo logs. Introduce new API.

    - API part - Runtime
    - upgrading redo log & create new undo tablespaces -> InnoDB

18. Iterate over all InnoDB tablespaces to create SDI entries in .ibd files.
    - Server now has all Tablespace information (step 9)

    - For each tablespace	-> Runtime Task
      - ask SE to create SDI index -> Runtime Task

      - create SDI object (from dd::Table, Schema, Indexes, etc) -> Runtime Task

      - use sdi_set() to insert SDI object -> Runtime Task
     
19. Mark correct version number in dictionary to mark completion of upgrade for
    next restart. -> Runtime Task
    - call API to flush dirty buffer pool pages to disk (optional) -> RunTime