WL#6394: Bootstrap code for new DD

Affects: Server-8.0   —   Status: Complete   —   Priority: Medium

1. Introduction
===============

Bootstrapping is done when starting the mysqld server process. In the context of
this worklog, we focus on aspects related to the new data dictionary (DD). There
are four basic aspects of this:

i)   Creating the data dictionary while starting a new server instance.
ii)  (Re)starting a server with existing data.
iii) Upgrading from a previous version. 
iv)  Upgrading from 5.7 to 8.0, i.e., start using the new DD on already
     existing user data.

2. Related work
===============

- WL#6392: Upgrade and downgrade tool for new data dictionary.

3. Definitions
==============

DD: Data dictionary.

SE: Storage engine.

DDSE: Storage engine in which the DD tables are created.

Unconstrained table definition: A table definition without foreign key constraints.

Constrained table definition: A table definition with foreign key constraints.

System tables: The tables that are created by the MySQL server when the server
is initialized. E.g., the dictionary tables that store table definitions, the
help tables, time zone information, etc.

Dictionary (DD) tables: The system tables that are used for storing meta data
about database items. E.g., the table 'mysql.tables' is a dictionary table,
while the help tables are not.

DDSE tables: System tables that are created by the server on behalf of the
dictionary storage engine. These behave in the same way as the core tables (see
below), and are hard coded based on DD version, but their hard coding is in the
DDSE rather than in the SQL layer.

Table definition tables: The DD tables that are used for (re)storing an
arbitrary table definition. These tables are opened when a table definition is
to be read from, or written to, persistent storage.

Core tables: The table definition tables that are needed to store the
unconstrained definitions of the table definition tables.

Core table parents: System tables that are needed to enable creating the
constrained core tables, i.e., the tables which are the parents of foreign keys
of a core table.

Inert tables: Tables that have an inert, unalterable definition which can always
be assumed to be valid. Currently, this is only the 'version' table.

DD schema: The collection of all DD table definitions.

DD version: A unique version number associated with the dictionary schema. When
there is a change in the DD schema, the version number is increased. 

Supported DD version: A (set of) DD version(s) from which it is possible to upgrade.

Actual DD table definition: The meta data of a DD table that is actually stored
persistently within the DD tables.

Actual DD schema: The collection of all actual DD table definitions.

Actual DD version: The version of the actual DD schema. The actual DD version is
stored persistently in a dedicated DD table 'dd.version'.

Target DD table definition: The DD table definition which is created when a
MySQL server starts the first time. The actual- and target DD table definitions
are the same unless there is a dictionary upgrade. 

Target DD schema: The collection of all target DD table definitions.

Target DD version: The version of the target DD schema. The target DD version is
represented in the server source code. If the server restarts, and the actual DD
version (in the dd.version table) equals the target DD version, we know that
dictionary upgrade is not needed.

Hardcoded DD table definition: The DD table definition represented by a
hardcoded internal data structure. 

Synthesized DDL statement: An SQL DDL statement that is generated based on the
hardcoded definition. 

Synthesized TABLE_SHARE: A TABLE_SHARE structure that is generated based on the
internal definition. 

3.1 Table categorization
------------------------

With the definitions above, the following categorization may be applied:

INERT_TABLE:      Needed to start initialization. Immutable.
CORE_TABLE:       Needed to read/store unconstrained table definitions.
TABLE_DEF_TABLE:  Needed to read/store table definitions.
DDSE_TABLE:       Needed by DDSE.
DICTIONARY_TABLE: Needed to read/store database item meta data.
SYSTEM_TABLE:     Needed to read/store system data.

With the current (8.0.0) predefined tables, we may apply the categorization as
follows:

Inert tables
............
Version

Core tables
...........
Schemata
Tables
Columns
Column_type_elements
Indexes
Index_column_usage
Tablespaces
Tablespace_files

Table definition tables
.......................
Foreign_keys
Foreign_key_column_usage
Table_partitions
Table_partition_values
Index_partitions
View_table_usage

Dictionary tables
.................
Character_sets
Collations
Catalogs

DDSE_tables
...........
innodb_table_stats
innodb_index_stats
innodb_table_metadata

System tables
.............
...


4. Short term goals
===================

The short term goals of this WL are:

4.1 Create new DD tables
------------------------

The MySQL server should be able to create new DD tables during the bootstrap
procedure. We will do this internally in the server, not by means of executing
SQL scripts as has been done previously.

We suggest to use hard coded structures from which SQL DDL statements can be
syntesized, and then execute the DDL statements for two purposes:

i)  Create the DD tables in the DDSE.
ii) Generate the meta data representing the DD tables.

The DDL execution must be modified to avoid storing the generated meta data into
the DD tables (all of which are not yet created), and instead store the meta
data temporarily in memory until table creation is finished. Then, eventually,
the generated meta data can be stored persistently in the DD tables, when they
are all created in the DDSE.

Additionally, some foreign key constraints must be collected and added after the
tables are created due to circular foreign key relationships. 

4.2 Upgrade to a new DD version
-------------------------------

Upgrade to a new DD version must be supported. This means that it must be
possible to open the existing DD tables, provided their DD version is supported,
and alter them to match the target DD. Thus, we must be able to obtain the
following information for each of the DD tables:

i)   The target table definition.
ii)  For each supported version: 
     a) The corresponding table definition.
     b) The necessary changes to match the target schema.

Upgrade from the supported version to the target version may be done by
assembling and executing SQL statements. It may be possible to deduce these
statements based on the difference between i) and iia), so we may not need to
hard code all of the information listed above. However, this decision is more
of an implementation issue.

Downgrade is not supported. If it should be supported, this may be done within
the same framework as described above. However, downgrade is not only a
dictionary related procedure, so even if downgrade may be doable from a
dictionary point of view, there may be other obstacles preventing it from being
supported.

4.3 Read serialized meta data, execute SQL statements
-----------------------------------------------------

This is necessary in order to support upgrade from 5.7 to 8.0. A dedicated tool
will generate serialized dictionary information, which will be read by
the starting 8.0 server, which will: 

- Create the DD tables and initialize the MySQL server.
- De-serialize the serialized meta data, and store it in the DD tables.

Additionally, the upgrade tool will generate SQL statements necessary for e.g.
dropping the old and abandoned dictionary tables. These statements must be read
and executed by the server.

1. Use cases
============

From the data dictionary point of view, there are four basic bootstrapping use
cases: 

i)   Initialize a new server instance: Create the dictionary tables.
ii)  (Re)start an existing server instance: Verify that the DD table
     definitions are as expected.
iii) Upgrade an old server instance: Execute SQL statements to
     bring the DD table definitions up to date.
iv)  Upgrade from 5.7 to 8.0: First time initialization of the new DD,
     and population of the new DD with meta data based on existing data.

We would like to emphasize that in this context, we only consider the data
dictionary related aspects of bootstrapping. In general, bootstrapping involves
numerous additional components in the server.

2. Preconditions
================

Upgrade is not allowed unless there are no XA PREPARE transactions to commit or
roll back. To be more specific, upgrade should not be allowed unless the undo
log is empty. The undo log can be nonempty even when no XA PREPARE trx exist, if
no slow shutdown was done. Thus, the precondition is that both no XA PREPARE trx
exist, and that innodb_fast_shutdown=0 has been used.

3. Functional requirements
==========================

Based on the use cases above, this section goes into more detail to elaborate on
the required functionality. 

Upgrade from old to new DD strongly resembles starting a new server instance;
thus, these two use cases are described in the same subsection below. The same
applies to "post new DD" upgrade and ordinary restart; they resemble each other,
and are therefore described in the same subsection below. 

3.1 Initialize a new server instance 
------------------------------------

The main idea for initializing the new DD is to synthesize and execute CREATE
TABLE ... statements, and to intercept the execution in a way that allows
temporarily saving the generated meta data until all the core DD tables are
created, and then eventually write the meta data to persistent storage. 

In WL#7307, the --bootstrap option was deprecated and replaced by the
--initialize option. Thus, the --initialize option is also what will trigger the
initialization of the new DD.

The procedure below also describes upgrade from 5.7 to 8.0, when the new DD is
to be initialized for an existing server. This procedure resembles initial
start, unlike ordinary post 8.0 upgrade, which resembles normal restart. The
suggested procedure is as follows:

3.1.1  The server calls plugin::init() like before. For InnoDB, this 
       resolves to innobase_init(). This function will be stripped down to the 
       bare minimum. It should neither create nor open any files. It will parse 
       command line options and set system variables accordingly. 

3.1.2  The server calls handlerton::dict_init(). In broad terms, for InnoDB, 
       it will:

       i)   Check if the predefined files exist, depending on the submitted 
            options. This check must be skipped for upgrade from 5.7 to 8.0, 
            since we want to use the existing system tablespace.

       ii)  Create predefined files if they do not exist:

            - System tablespace
            - DD tablespace
            - Redo log
            - ...

            Get the names from the system variables, which were set by the 
            plugin:::init() function based on the command line parameters. 
            For upgrade from 5.7, the system tablespace will be kept, while
            the DD tablespace will be created. The DD tablespace must be
            created on a fixed, hard coded and unique se_private_id to avoid 
            conflict on 5.7 -> 8.0 upgrade.

       iii) The handlerton::dict_init() function returns meta data to be added 
            to the DD tables representing the predefined files that are 
            created, and the DD tables needed by InnoDB. The meta data for the 
            predefined files will be inserted to the appropriate DD tables, 
            while the meta data for the InnoDB DD tables will be used to 
            synthesize a CREATE TABLE statement that both creates the tables 
            and adds their meta data to the DD.

3.1.3  The meta data for the predefined tablespaces may now be created and added
       to the shared DD cache. This is necessary to make it possible to create 
       the DD tables. The tablespaces were already created physically in 
       dict_init().

3.1.4  Now, the DD database must be created, and the meta data representing it
       must be added to the shared DD cache. The DD database is created by
       executing a 'CREATE SCHEMA' statement, and the DDL execution is changed
       to only add the generated meta data to the shared DD cache when the DD
       database is created, instead of storing it into the DD tables (which do 
       not yet exist).

3.1.5  Then, the server can create the core DD tables. This must be done in a 
       fixed order using fixed and hard coded SE private ids in order to ensure 
       that the tables can be opened when the server is restarted. For InnoDB, 
       the important things are to make sure that: 

       - For a given DD table, the root page number of each index will always 
         have the same value. This is important to make sure we can re-open an 
         existing DD table when the server is restarted. (If data is inserted 
         before all tables with hard coded page numbers are created, we will 
         also have to take innodb_page_size into account when determining the 
         index root page number).
       - For each index of a given DD table, the id which is represented 
         in the b-trees (PAGE_INDEX_ID) always has the same value. 
       - The 'space_id', which is a foreign key of the dd.tablespaces table, 
         has the correct value, as given by the corresponding meta data 
         returned from 3.1.2.iii).

       The tables are created by synthesizing DDL statements that are executed. 
       During execution, the generated meta data is added to the shared DD 
       cache. As part of the DDL execution, the server calls
       handlerton::get_se_private_data(), which does the necessary updates to 
       the dd::Table structures that are passed as parameters, assigning the 
       se_private_ids, the root page numbers, etc. as appropriate. 

3.1.6  After all the tables in 3.1.5 are created in the DDSE, the server can 
       proceed to first call 'dict_recover()', and then store the generated 
       meta data representing the DD tables, as well as the predefined 
       tablespaces, into the DD tables. 

3.1.7  The server must also store additional meta data representing, e.g., the
       DD database, and the predefined tablespaces and tablespace files. The 
       meta data representing the DD database was temporarily saved in step 
       3.1.4, and the meta data representing the predefined tablespaces was 
       retrieved when the dict_init() function returned in step 3.1.2.iii, and 
       was added to the shared DD cache in step 3.1.3. 

3.1.8  Now, the DD tables that are neither core tables nor DDSE tables can be 
       created. This is done in the same way as in step 3.1.5, but now, the 
       meta data can be stored directly since the required DD tables for 
       storing table definitions are already in place. No special handling of 
       this is required neither on the server nor the DDSE side. The SQL DDL 
       statements for creating the tables are synthesized from hard coded 
       representations just like in step 3.1.5. 

3.1.9  The foreign keys are added as a separate step due to circular 
       dependencies. This is done by synthesizing ALTER TABLE ... ADD FOREIGN 
       KEY statements based on the hard coded table definitions. 

3.1.10 Finally, the version number of the target dictionary version is stored in
       the version table.

3.1.11 If this is 5.7 -> 8.0 upgrade, the server must take some additional  
       actions :

       i)   Read the submitted configuration file to obtain SDI generated by the
            upgrade tool. 
       ii)  De-serialize the SDI and merge it with the existing DD. 
       iii) Write back the SDI to the appropriate locations in the tablespaces, 
            unless the upgrade tool has written this already.
       iv)  Execute the submitted SQL script.

3.1.12 Now, the DD initialization is finished, and the server can continue the 
       normal initialization procedure, and eventually exit. For ordinary use, 
       it must be restarted without the <install-option> (or <upgrade-option>) 
       on the command line.

3.2 Restart or upgrade an existing server instance
--------------------------------------------------

Restart is the usual server startup scenario. Here, we assume that the
dictionary tablespace and database exist, and that all the DD tables exist. The
main idea is to synthesize dummy CREATE TABLE ... statements and execute them in
order to generate the structures that are required for opening the tables,
without creating the tables themselves.

For the normal restart scenario, we will have to check the dictionary version,
to determine whether upgrade is needed. Additionally, synchronization is needed
between the server and the DDSE to account for the merged dictionaries, and to
coordinate DDSE specific recovery. 

The procedure is outlined below, and also covers upgrade due to its tight
integration with normal restart:

3.2.1  The server calls plugin::init(). For InnoDB, this resolves to 
       innobase_init(). See 3.1.1 for additional information.

3.2.2  The server calls handlerton::dict_init(), which will verify that 
       the predefined files exist as expected (given by system variables that 
       were set by plugin::init() based on command line arguments). 

       In an upgrade scenario, the DDSE may do just the steps needed to read 
       the dd.version table, and have the version dependent initialization as a 
       separate step after the actual DD version has been read, i.e., 
       introducing a new handlerton function for doing this.

       For InnoDB being the DDSE, a new DD tablespace will be created on 
       upgrade. This is done to simplify the procedure of retrieving the root 
       page numbers for each index. If keeping the existing tablespace, the 
       indexes would get new root page numbers when undergoing an ALTER TABLE 
       ..., depending on the actual DD contents. The details of how to divide 
       the responsibility of rebuilding the DD tablespace, upgrading the DD 
       tables etc. will be worked out at a later stage.

3.2.3  The meta data representing the predefined tablespaces, retrieved from the
       DDSE, is added to the shared DD cache. This is necessary to be able to
       prepare the table meta data that we need to open the DD tables.

3.2.4  The server creates the structures necessary for reading from the 
       dd.version table. This is done by executing a CREATE TABLE statement to 
       generate the dd::Table structure. The DDL execution on the server layer 
       must be modified to keep the table from being created in the DDSE (i.e., 
       handler::create() will not be called, because the table already exists), 
       while the meta data must only be added temporarily to the shared DD 
       cache. This is volatile meta data that is just needed for opening the 
       dd.version table.

3.2.5  The server calls handlerton::get_se_private_data(table, version) where 
       'table' is the dd::Table from 3.2.4, and 'version' is the current target 
       dictionary version. Here, the DDSE updates the se_private_data (root 
       page number) and the se_private_id of the clustered index. These values 
       are needed to be able to read from the dd.version table, and the server 
       must be able to do this in order to determine which version of the DD 
       tables it should open, i.e., which DD schema it should expect the 
       existing tables to have. 

3.2.6  Now, we have the meta data necessary to open the dd.version table, and 
       can read the single entry from it. The server will validate the value 
       that was read: 

       i)   No value:                Error
       ii)  value > target version:  Error (downgrade not supported)
       iii) value < target version:  Error (unless <upgrade-option> is submitted
                                     and 'version' is supported vesrion)
       iv)  value == target version: OK

       At this stage, we know we are starting a server with either the target 
       DD table definitions, or with a different DD version for which we have 
       the hard coded table definitions available (that is what the phrase 
       "supported version" means). 

3.2.7  The server can create the structures needed for reading the core DD 
       tables using the appropriate DD schema, i.e., the schema that 
       corresponds to the version that was read in 3.2.6. The structures are 
       created by executing DDL statements for creating the tables, keeping the 
       tables from being created in the SE by modifying the DDL table 
       execution. The generated meta data is added to the shared DD cache. 

3.2.8  The required hard coded SE private data must be filled in by the DDSE. 
       This is done by calling handlerton::get_se_private_data(tables, version) 
       where the 'tables' are the dd::Table structures from f), and 'version' 
       is the actual dictionary version which was read in 3.2.6. Here, the DDSE 
       updates the se_private_data (root page number) and the se_private_id of 
       the indexes.

3.2.9  The server calls handlerton::dict_recover(version, phase) to make the 
       DDSE roll back incomplete transactions that performed DDL.

3.2.10 Now, the server can read the actual DD table definitions of all the DD 
       tables from the core DD tables, using the volatile meta data from 3.2.7.

3.2.11 At this point, if we are doing upgrade, the server must take some
       additional actions:

       i)   Perform a slow shutdown, but keeping the redo log open. This should 
            be implemented in the upgrading instance (requiring this to be done 
            when shutting down the old version may be cumbersome if the user 
            forgot to do it). This step ensures that InnoDB can drop support 
            for old versions of persistent internal data structures, and 
            guarantees that we can reassign dd.tables.se_private_id for the DD 
            tables in subsequent steps.
       ii)  Execute inplace alter SQL statements, CREATE/DROP etc. to change 
            the table definitions. Preferably use only one transaction if we 
            are able to relax auto commit.
       iii) Ensure cache consistency (e.g. dd cache vs. TDC). Re-read table 
            definitions and replace cache entries and re-generate serialized 
            dictionary information conforming to the updated DD schema.
       iv)  The stored SDI for the tables in the various tablespaces must be
            re-generated. The same applies to tables in other SEs than InnoDB.
       v)   When upgrade is done, the DD version number is stored in the 
            dd.version table. 

3.2.12 The server can initiate recovery of DML transactions by calling 
       handlerton::dict_recover(version, phase).

3.2.13 Now, the DD initialization is finished. For ordinary restart, the server 
       can continue the normal restart procedure. We may open and read any 
       table, both DD tables and user tables, as usual. 

       For upgrade, the server will eventually exit, and for ordinary use, it 
       must be restarted without the <upgrade-option> on the command line.

4. Error handling
-----------------

During upgrade, with InnoDB as the DDSE, the DD tablespace will be rebuilt.
Thus, in the event of a server crash in the middle of upgrade, the option is to
delete the new DD tablespace in the process of being rebuilt and restart the
upgrade. Deleting the leftover tablespace may be done automatically by the DDSE
when the upgrade is restarted.

While altering the DD tables as part of upgrade, it may make most sense to first
move the tables to the new DD tablespace, and then do the actual DD table
definition changes.

For a server crash in the middle of normal restart, the only option is to retry,
or possibly doing disaster recovery. 

5. General requirements
-----------------------

The use cases above have some functional requirements in common. 

5.1 Requirements implemented as of Phase one:
.............................................

FR-01  Refactor plugin initialization to allow the DD to be initialized after 
       the DDSE, but before the dynamic plugins.

FR-02  Refactor the existing bootstrap code to allow it to be used in various 
       contexts. Introduce a bootstrap function type for submitting function
       parameters where bootstrapping should start.

FR-03  Introduce a Bootstrapper singleton or name space in the DD module to 
       coordinate the initialization of the DD. 

FR-04  Provide a registry of all the DD table definitions.

FR-05  Represent the target definition of the DD tables by a hardcoded 
       structure.

FR-06  Generate the dd::Table object structures based on the hardcoded 
       structure.

FR-07  Create the table in the DDSE based on the hardcoded structure.

FR-08  Retrieve the temporarily stored, generated meta data 
       structures when a DD table is to be opened.

FR-09  Modify DDL execution to prohibit storing the meta data into the DD 
       tables, but instead save it temporarily. 

FR-10  Execute statements for populating the tables.

FR-11  Represent the current version number hard coded in the dictionary
       module.

FR-12  Define the dd.version table, add its table definition.

FR-13  It must be possible for the SE to ask for additional DD tables to be 
       created. This is the case for e.g. innodb_table_stats and 
       innodb_index_stats, which are  currently created from definitions 
       hardcoded within the server.

FR-14  Consistency between the SE perception of the DD tablespace and the
       corresponding meta data stored in the dd.tablespaces table must be 
       ensured before allowing the server to start.

[FR-15  The server must distinguish between the core and non-core tables. This 
       may limit the amount of hard coded data, but has the drawback of 
       complicationg the initialization and restart logic. This requirement
       is considered optional.]

FR-16  Create the meta data representing the predefined tablespaces and add
       this meta data to the shared DD cache.

FR-17  Transparent encryption will be implemented also for the shared 
       tablespaces such as the DD tablespace. We do not see that this will 
       cause any issues for bootstrapping, and therefore do not plan any 
       actions in this area; we assume that this will work out of the box.

5.2 Requirements related to upgrade (NOT implemented as part of this WL)
.............................................................................

These requirements will be implemented in the context where they become
relevant, e.g. within the scope of WL#6392, or when upgrade support is required.

FR-18  Support upgrade from 5.7 to 8.0, depending on WL#6392.

FR-19  Represent the definition of the previous DD version by a hardcoded 
       structure.

FR-20  Represent operations to upgrade from the previous to the current version.

FR-21  Remove the output parameters of dict_init() and move them to a new 
       method dict_internal_table_definitions(version, tables, artifacts), to 
       be invoked when the current dd.version is known.
1. Overall bootstrapping architecture
=====================================

We divide the bootstrapping architecture into various main areas that are
further decomposed below. 

1.1 Dictionary boostrapping, plugin- and server initialization
--------------------------------------------------------------

The current (5.7) server startup and bootstrapping implementation needs to be
refactored to allow for proper dictionary bootstrapping at the specific point
where it must be done. 

1.1.1 Refactor current bootstrap functions
..........................................

The code currently doing the bootstrapping spawns a thread (which is waited for)
which sets up a THD and reads, parses and executes SQL commands from a file
(e.g. stdin). 

This code has to be refactored to allow using the same THD from within the DD to
do the required bootstrapping there. Thus, the current bootstrap() function
accepts an additional paramter, which is a function pointer. If NULL, the
default bootstrap handler function is used, which reads SQL commands from the
submitted file and executes them. If non-NULL, the submitted function is called. 

1.1.2 Remove SQL statements from scripts/mysql_system_tables.sql
................................................................

This task is just deleting the creation of the new DD tables from the SQL script
that is used as a basis for creating tables when starting the server with the
--initialize option. The remainder of the script is left unchanged for now, but
the long term goal is to move it into the server in the same way as we are now
doing with the creation of the dictionary tables. 

1.1.3 Integrate DD bootstrapping with DD initialization
.......................................................

The bootstrapping is integrated with initialization of the dictionary. The DD
initialization function is changed to accept a parameter indicating whether the
server is started the first time or not. In dd::init(bool install), a new
Dictionary_impl instance is created, and the dictionary is made available
(assuming the tables already exist) or installed (creating new DD tables)
depending on the submitted argument. This is done by submitting the appropriate
bootstrap handler to the main bootstrap function.

There is a dedicated singleton which coordinates the bootstrapping. This
'Bootstrapper' implements the restart() and initialize() methods. The
Bootstrapper will be examined further below.

1.1.4 Hook up DD bootstrapping with mysqld startup
..................................................

Before this WL, the DD was initialized right before loading the plugins, because
the mysql.plugins table had to be present at that stage. The bootstrapping
itself was being done at a much later stage. 

When removing the .FRM files, and integrating the dictionary initialization,
installation and ordinary start, this has to be revised. The dictionary
bootstrapping must be done after InnoDB is initialized (because the DD is stored
in InnoDB), but before the dynamic plugins are loaded (because they assume that
the mysql.plugins table is present). Thus, the plugin_init() function is split
into four functions, two of which are externally visible. This allows
init_server_components() to first load and initialize the builtin plugins
(including the DDSE, i.e. InnoDB), then initialize the dictionary, and finally
to load and initialize the dynamic plugins.

1.1.5 Command line options
--------------------------

Before WL#7307, we needed a way to initialize the dictionary by submitting a
command line option. For this purpose, the '--install-server' option was
introduced. This option is still needed by external tools that do not use the
'--initialize' option introduced in WL#7307.

The '--install-server' option will be removed, and we will instead rely on the
'--initialize' option that is already introduced. However, this cannot be done
until mtr and other external tools all use the '--initialize' option.

1.1.6 Let the DDSE create additional tables
-------------------------------------------

The hanlerton function dict_init() is called to initialize the DDSE. This
function returns a list of descriptions of tables to be defined by the server on
behalf of the DDSE. 

1.1.7 Store meta data representing I_S tables
---------------------------------------------

The I_S tables are represented in source code only, there are no real tables
created. Thus, there is no meta data present in the DD tables reflecting the
existence of the I_S tables. 

We suggest to introduce an I_S initialization function in sql_show.cc, where the
table definition structures can be analyzed, and the corresponding dd.Table,
dd.Columns etc. can be created and stored to make the meta data avilable in the
DD tables. 

1.2 Representing DD table definitions
-------------------------------------

The DD objects that can be stored persistently are represented by objects having
an associated singleton representing the object table. E.g., the interface
dd::Table is implemented by dd::Table_impl, with an associated
dd::tables::Tables singleton representing the object table. The object table
class defines e.g. the table name, how to create keys for the table, etc. 

The definition of each DD table must be available in some form. We suggest to do
this in the form of SQL statements associated with the object table singelton.

1.2.1 New DD Object_table_definition class
..........................................

A new interface is introduced to support retrieving the SQL statements needed to
create the table, add foreign keys, and to populate it.

The interface is implemented by a class which also provides methods for adding
and manipulating the SQL definition of the table. The statement is broken up
into string fragments that may be combined to form a complete create table
statement: 

1.2.2 Add an Object_table_definition instance to Object_table
.............................................................

An instance m_target_def of the class Object_type_definition is added to the
Object_type_impl class to represent the target definition. Additional instances
may be added to keep track of previous supported versions of the table, which
may be needed in the case of upgrade. 

1.3 Modifying the SQL DDL
-------------------------

The MySQL server code which uses the dictionary must be changed in various ways
for this bootstrap approach to work.

DDL execution must be modified to recognize creation of DD tables, and to avoid
storing the created meta data (dd::Table instance) into the DD tables in these
situations, because the DD tables needed for storing the meta data may not be
created yet. Instead of storing the meta data, it is added to the shared DD
cache. This enable the cache to be accessed from elsewhere as usual, e.g. in
open_table_def().

In a similar fashion, the execution of schema DDL is modified to recognize the
creation of the DD schema.

Additionally, in mysql_create_table_no_lock(), we set a parameter for creating
handler files in create_table_impl() for DD tables only if we are doing initial
start. This is necessary to enable doing CREATE TABLE for DD tables at restart
for having the structures required for opening the tables generated, without
actually creating the table.

1.5 The new DD Bootstrapper class
---------------------------------

For coordinating bootsrapping of the dictionary, a dedicated singleton will be
used. Alternatively, this may be implemented by functions in a dedicated
namespace. The bootstrapper implements two main methods that are called when the
DD is initialized. 'initialize()' will create new DD tables, while 'restart()'
will assume that the DD already exists, and try opening the existing tables.

1. Extensions of the handlerton API
===================================

1.1 Initialization of the DDSE
------------------------------

enum dict_init_mode_t
{
  DICT_INIT_CREATE_FILES,         //< Create all required SE files
  DICT_INIT_CREATE_MISSING_FILES, //< Use files that already exist
  DICT_INIT_CHECK_FILES,          //< Verify existence of expected files
  DICT_INIT_IGNORE_FILES          //< Don't care about files at all
};


/**
  Initialize the SE for being used to store the DD tables. Create
  the required files according to the dict_init_mode. Create strings
  representing the required DDSE tables, i.e., tables that the DDSE
  expects to exist in the DD, and add them to the appropriate out
  parameter.

  @param dict_init_mode       How to initialize files
  @param version              Target DD version if a new
                              server is being installed.
                              0 if restarting an existing
                              server.
   @param [out] DDSE_tables    List of SQL DDL statements
                              for creating DD tables that
                              are needed by the DDSE.
  @param [out] DDSE_artifacts List of SQL DML statements
                              for adding meta data for DD
                              entities created by the DDSE,
                              e.g. inserting a row in the
                              dd.tablespaces table.

  @retval true                An error occurred.
  @retval false               Success - no errors.
 */

bool (*dict_init)(dict_init_mode_t dict_init_mode,
                  uint version,
                  List<const char> &DDSE_tables,
                  List<const char> &DDSE_artifacts);


1.2 Getting the se_private_data for the DD tables
-------------------------------------------------

/**
  Submit a list of dd::Table objects representing the core DD tables
  having hardcoded data to be filled in by the DDSE. This function can be
  used for retrieving the hard coded SE private data for the dd.version
  table, before creating for opening it (submitting version = 0), or for
  retrieving the hard coded SE private data for a list of core tables,
  before creating or opening them (submit version == the actual version
  which was read from the dd.version table).

  @param dd_tables[in,out]    List of dd::Table objects representing
                              the core DD tables.
  @param version              Actual version of the DD.

  @retval true                An error occurred.
  @retval false               Success - no errors.
 */

bool ha_get_se_private_data(List<dd::Table> dd_tables, uint version);