WL#6394: Bootstrap code for new DD
Affects: Server-8.0
—
Status: Complete
1. Introduction
===============
Bootstrapping is done when starting the mysqld server process. In the context of
this worklog, we focus on aspects related to the new data dictionary (DD). There
are four basic aspects of this:
i) Creating the data dictionary while starting a new server instance.
ii) (Re)starting a server with existing data.
iii) Upgrading from a previous version.
iv) Upgrading from 5.7 to 8.0, i.e., start using the new DD on already
existing user data.
2. Related work
===============
- WL#6392: Upgrade and downgrade tool for new data dictionary.
3. Definitions
==============
DD: Data dictionary.
SE: Storage engine.
DDSE: Storage engine in which the DD tables are created.
Unconstrained table definition: A table definition without foreign key constraints.
Constrained table definition: A table definition with foreign key constraints.
System tables: The tables that are created by the MySQL server when the server
is initialized. E.g., the dictionary tables that store table definitions, the
help tables, time zone information, etc.
Dictionary (DD) tables: The system tables that are used for storing meta data
about database items. E.g., the table 'mysql.tables' is a dictionary table,
while the help tables are not.
DDSE tables: System tables that are created by the server on behalf of the
dictionary storage engine. These behave in the same way as the core tables (see
below), and are hard coded based on DD version, but their hard coding is in the
DDSE rather than in the SQL layer.
Table definition tables: The DD tables that are used for (re)storing an
arbitrary table definition. These tables are opened when a table definition is
to be read from, or written to, persistent storage.
Core tables: The table definition tables that are needed to store the
unconstrained definitions of the table definition tables.
Core table parents: System tables that are needed to enable creating the
constrained core tables, i.e., the tables which are the parents of foreign keys
of a core table.
Inert tables: Tables that have an inert, unalterable definition which can always
be assumed to be valid. Currently, this is only the 'version' table.
DD schema: The collection of all DD table definitions.
DD version: A unique version number associated with the dictionary schema. When
there is a change in the DD schema, the version number is increased.
Supported DD version: A (set of) DD version(s) from which it is possible to upgrade.
Actual DD table definition: The meta data of a DD table that is actually stored
persistently within the DD tables.
Actual DD schema: The collection of all actual DD table definitions.
Actual DD version: The version of the actual DD schema. The actual DD version is
stored persistently in a dedicated DD table 'dd.version'.
Target DD table definition: The DD table definition which is created when a
MySQL server starts the first time. The actual- and target DD table definitions
are the same unless there is a dictionary upgrade.
Target DD schema: The collection of all target DD table definitions.
Target DD version: The version of the target DD schema. The target DD version is
represented in the server source code. If the server restarts, and the actual DD
version (in the dd.version table) equals the target DD version, we know that
dictionary upgrade is not needed.
Hardcoded DD table definition: The DD table definition represented by a
hardcoded internal data structure.
Synthesized DDL statement: An SQL DDL statement that is generated based on the
hardcoded definition.
Synthesized TABLE_SHARE: A TABLE_SHARE structure that is generated based on the
internal definition.
3.1 Table categorization
------------------------
With the definitions above, the following categorization may be applied:
INERT_TABLE: Needed to start initialization. Immutable.
CORE_TABLE: Needed to read/store unconstrained table definitions.
TABLE_DEF_TABLE: Needed to read/store table definitions.
DDSE_TABLE: Needed by DDSE.
DICTIONARY_TABLE: Needed to read/store database item meta data.
SYSTEM_TABLE: Needed to read/store system data.
With the current (8.0.0) predefined tables, we may apply the categorization as
follows:
Inert tables
............
Version
Core tables
...........
Schemata
Tables
Columns
Column_type_elements
Indexes
Index_column_usage
Tablespaces
Tablespace_files
Table definition tables
.......................
Foreign_keys
Foreign_key_column_usage
Table_partitions
Table_partition_values
Index_partitions
View_table_usage
Dictionary tables
.................
Character_sets
Collations
Catalogs
DDSE_tables
...........
innodb_table_stats
innodb_index_stats
innodb_table_metadata
System tables
.............
...
4. Short term goals
===================
The short term goals of this WL are:
4.1 Create new DD tables
------------------------
The MySQL server should be able to create new DD tables during the bootstrap
procedure. We will do this internally in the server, not by means of executing
SQL scripts as has been done previously.
We suggest to use hard coded structures from which SQL DDL statements can be
syntesized, and then execute the DDL statements for two purposes:
i) Create the DD tables in the DDSE.
ii) Generate the meta data representing the DD tables.
The DDL execution must be modified to avoid storing the generated meta data into
the DD tables (all of which are not yet created), and instead store the meta
data temporarily in memory until table creation is finished. Then, eventually,
the generated meta data can be stored persistently in the DD tables, when they
are all created in the DDSE.
Additionally, some foreign key constraints must be collected and added after the
tables are created due to circular foreign key relationships.
4.2 Upgrade to a new DD version
-------------------------------
Upgrade to a new DD version must be supported. This means that it must be
possible to open the existing DD tables, provided their DD version is supported,
and alter them to match the target DD. Thus, we must be able to obtain the
following information for each of the DD tables:
i) The target table definition.
ii) For each supported version:
a) The corresponding table definition.
b) The necessary changes to match the target schema.
Upgrade from the supported version to the target version may be done by
assembling and executing SQL statements. It may be possible to deduce these
statements based on the difference between i) and iia), so we may not need to
hard code all of the information listed above. However, this decision is more
of an implementation issue.
Downgrade is not supported. If it should be supported, this may be done within
the same framework as described above. However, downgrade is not only a
dictionary related procedure, so even if downgrade may be doable from a
dictionary point of view, there may be other obstacles preventing it from being
supported.
4.3 Read serialized meta data, execute SQL statements
-----------------------------------------------------
This is necessary in order to support upgrade from 5.7 to 8.0. A dedicated tool
will generate serialized dictionary information, which will be read by
the starting 8.0 server, which will:
- Create the DD tables and initialize the MySQL server.
- De-serialize the serialized meta data, and store it in the DD tables.
Additionally, the upgrade tool will generate SQL statements necessary for e.g.
dropping the old and abandoned dictionary tables. These statements must be read
and executed by the server.
1. Use cases
============
From the data dictionary point of view, there are four basic bootstrapping use
cases:
i) Initialize a new server instance: Create the dictionary tables.
ii) (Re)start an existing server instance: Verify that the DD table
definitions are as expected.
iii) Upgrade an old server instance: Execute SQL statements to
bring the DD table definitions up to date.
iv) Upgrade from 5.7 to 8.0: First time initialization of the new DD,
and population of the new DD with meta data based on existing data.
We would like to emphasize that in this context, we only consider the data
dictionary related aspects of bootstrapping. In general, bootstrapping involves
numerous additional components in the server.
2. Preconditions
================
Upgrade is not allowed unless there are no XA PREPARE transactions to commit or
roll back. To be more specific, upgrade should not be allowed unless the undo
log is empty. The undo log can be nonempty even when no XA PREPARE trx exist, if
no slow shutdown was done. Thus, the precondition is that both no XA PREPARE trx
exist, and that innodb_fast_shutdown=0 has been used.
3. Functional requirements
==========================
Based on the use cases above, this section goes into more detail to elaborate on
the required functionality.
Upgrade from old to new DD strongly resembles starting a new server instance;
thus, these two use cases are described in the same subsection below. The same
applies to "post new DD" upgrade and ordinary restart; they resemble each other,
and are therefore described in the same subsection below.
3.1 Initialize a new server instance
------------------------------------
The main idea for initializing the new DD is to synthesize and execute CREATE
TABLE ... statements, and to intercept the execution in a way that allows
temporarily saving the generated meta data until all the core DD tables are
created, and then eventually write the meta data to persistent storage.
In WL#7307, the --bootstrap option was deprecated and replaced by the
--initialize option. Thus, the --initialize option is also what will trigger the
initialization of the new DD.
The procedure below also describes upgrade from 5.7 to 8.0, when the new DD is
to be initialized for an existing server. This procedure resembles initial
start, unlike ordinary post 8.0 upgrade, which resembles normal restart. The
suggested procedure is as follows:
3.1.1 The server calls plugin::init() like before. For InnoDB, this
resolves to innobase_init(). This function will be stripped down to the
bare minimum. It should neither create nor open any files. It will parse
command line options and set system variables accordingly.
3.1.2 The server calls handlerton::dict_init(). In broad terms, for InnoDB,
it will:
i) Check if the predefined files exist, depending on the submitted
options. This check must be skipped for upgrade from 5.7 to 8.0,
since we want to use the existing system tablespace.
ii) Create predefined files if they do not exist:
- System tablespace
- DD tablespace
- Redo log
- ...
Get the names from the system variables, which were set by the
plugin:::init() function based on the command line parameters.
For upgrade from 5.7, the system tablespace will be kept, while
the DD tablespace will be created. The DD tablespace must be
created on a fixed, hard coded and unique se_private_id to avoid
conflict on 5.7 -> 8.0 upgrade.
iii) The handlerton::dict_init() function returns meta data to be added
to the DD tables representing the predefined files that are
created, and the DD tables needed by InnoDB. The meta data for the
predefined files will be inserted to the appropriate DD tables,
while the meta data for the InnoDB DD tables will be used to
synthesize a CREATE TABLE statement that both creates the tables
and adds their meta data to the DD.
3.1.3 The meta data for the predefined tablespaces may now be created and added
to the shared DD cache. This is necessary to make it possible to create
the DD tables. The tablespaces were already created physically in
dict_init().
3.1.4 Now, the DD database must be created, and the meta data representing it
must be added to the shared DD cache. The DD database is created by
executing a 'CREATE SCHEMA' statement, and the DDL execution is changed
to only add the generated meta data to the shared DD cache when the DD
database is created, instead of storing it into the DD tables (which do
not yet exist).
3.1.5 Then, the server can create the core DD tables. This must be done in a
fixed order using fixed and hard coded SE private ids in order to ensure
that the tables can be opened when the server is restarted. For InnoDB,
the important things are to make sure that:
- For a given DD table, the root page number of each index will always
have the same value. This is important to make sure we can re-open an
existing DD table when the server is restarted. (If data is inserted
before all tables with hard coded page numbers are created, we will
also have to take innodb_page_size into account when determining the
index root page number).
- For each index of a given DD table, the id which is represented
in the b-trees (PAGE_INDEX_ID) always has the same value.
- The 'space_id', which is a foreign key of the dd.tablespaces table,
has the correct value, as given by the corresponding meta data
returned from 3.1.2.iii).
The tables are created by synthesizing DDL statements that are executed.
During execution, the generated meta data is added to the shared DD
cache. As part of the DDL execution, the server calls
handlerton::get_se_private_data(), which does the necessary updates to
the dd::Table structures that are passed as parameters, assigning the
se_private_ids, the root page numbers, etc. as appropriate.
3.1.6 After all the tables in 3.1.5 are created in the DDSE, the server can
proceed to first call 'dict_recover()', and then store the generated
meta data representing the DD tables, as well as the predefined
tablespaces, into the DD tables.
3.1.7 The server must also store additional meta data representing, e.g., the
DD database, and the predefined tablespaces and tablespace files. The
meta data representing the DD database was temporarily saved in step
3.1.4, and the meta data representing the predefined tablespaces was
retrieved when the dict_init() function returned in step 3.1.2.iii, and
was added to the shared DD cache in step 3.1.3.
3.1.8 Now, the DD tables that are neither core tables nor DDSE tables can be
created. This is done in the same way as in step 3.1.5, but now, the
meta data can be stored directly since the required DD tables for
storing table definitions are already in place. No special handling of
this is required neither on the server nor the DDSE side. The SQL DDL
statements for creating the tables are synthesized from hard coded
representations just like in step 3.1.5.
3.1.9 The foreign keys are added as a separate step due to circular
dependencies. This is done by synthesizing ALTER TABLE ... ADD FOREIGN
KEY statements based on the hard coded table definitions.
3.1.10 Finally, the version number of the target dictionary version is stored in
the version table.
3.1.11 If this is 5.7 -> 8.0 upgrade, the server must take some additional
actions :
i) Read the submitted configuration file to obtain SDI generated by the
upgrade tool.
ii) De-serialize the SDI and merge it with the existing DD.
iii) Write back the SDI to the appropriate locations in the tablespaces,
unless the upgrade tool has written this already.
iv) Execute the submitted SQL script.
3.1.12 Now, the DD initialization is finished, and the server can continue the
normal initialization procedure, and eventually exit. For ordinary use,
it must be restarted without the (or )
on the command line.
3.2 Restart or upgrade an existing server instance
--------------------------------------------------
Restart is the usual server startup scenario. Here, we assume that the
dictionary tablespace and database exist, and that all the DD tables exist. The
main idea is to synthesize dummy CREATE TABLE ... statements and execute them in
order to generate the structures that are required for opening the tables,
without creating the tables themselves.
For the normal restart scenario, we will have to check the dictionary version,
to determine whether upgrade is needed. Additionally, synchronization is needed
between the server and the DDSE to account for the merged dictionaries, and to
coordinate DDSE specific recovery.
The procedure is outlined below, and also covers upgrade due to its tight
integration with normal restart:
3.2.1 The server calls plugin::init(). For InnoDB, this resolves to
innobase_init(). See 3.1.1 for additional information.
3.2.2 The server calls handlerton::dict_init(), which will verify that
the predefined files exist as expected (given by system variables that
were set by plugin::init() based on command line arguments).
In an upgrade scenario, the DDSE may do just the steps needed to read
the dd.version table, and have the version dependent initialization as a
separate step after the actual DD version has been read, i.e.,
introducing a new handlerton function for doing this.
For InnoDB being the DDSE, a new DD tablespace will be created on
upgrade. This is done to simplify the procedure of retrieving the root
page numbers for each index. If keeping the existing tablespace, the
indexes would get new root page numbers when undergoing an ALTER TABLE
..., depending on the actual DD contents. The details of how to divide
the responsibility of rebuilding the DD tablespace, upgrading the DD
tables etc. will be worked out at a later stage.
3.2.3 The meta data representing the predefined tablespaces, retrieved from the
DDSE, is added to the shared DD cache. This is necessary to be able to
prepare the table meta data that we need to open the DD tables.
3.2.4 The server creates the structures necessary for reading from the
dd.version table. This is done by executing a CREATE TABLE statement to
generate the dd::Table structure. The DDL execution on the server layer
must be modified to keep the table from being created in the DDSE (i.e.,
handler::create() will not be called, because the table already exists),
while the meta data must only be added temporarily to the shared DD
cache. This is volatile meta data that is just needed for opening the
dd.version table.
3.2.5 The server calls handlerton::get_se_private_data(table, version) where
'table' is the dd::Table from 3.2.4, and 'version' is the current target
dictionary version. Here, the DDSE updates the se_private_data (root
page number) and the se_private_id of the clustered index. These values
are needed to be able to read from the dd.version table, and the server
must be able to do this in order to determine which version of the DD
tables it should open, i.e., which DD schema it should expect the
existing tables to have.
3.2.6 Now, we have the meta data necessary to open the dd.version table, and
can read the single entry from it. The server will validate the value
that was read:
i) No value: Error
ii) value > target version: Error (downgrade not supported)
iii) value < target version: Error (unless is submitted
and 'version' is supported vesrion)
iv) value == target version: OK
At this stage, we know we are starting a server with either the target
DD table definitions, or with a different DD version for which we have
the hard coded table definitions available (that is what the phrase
"supported version" means).
3.2.7 The server can create the structures needed for reading the core DD
tables using the appropriate DD schema, i.e., the schema that
corresponds to the version that was read in 3.2.6. The structures are
created by executing DDL statements for creating the tables, keeping the
tables from being created in the SE by modifying the DDL table
execution. The generated meta data is added to the shared DD cache.
3.2.8 The required hard coded SE private data must be filled in by the DDSE.
This is done by calling handlerton::get_se_private_data(tables, version)
where the 'tables' are the dd::Table structures from f), and 'version'
is the actual dictionary version which was read in 3.2.6. Here, the DDSE
updates the se_private_data (root page number) and the se_private_id of
the indexes.
3.2.9 The server calls handlerton::dict_recover(version, phase) to make the
DDSE roll back incomplete transactions that performed DDL.
3.2.10 Now, the server can read the actual DD table definitions of all the DD
tables from the core DD tables, using the volatile meta data from 3.2.7.
3.2.11 At this point, if we are doing upgrade, the server must take some
additional actions:
i) Perform a slow shutdown, but keeping the redo log open. This should
be implemented in the upgrading instance (requiring this to be done
when shutting down the old version may be cumbersome if the user
forgot to do it). This step ensures that InnoDB can drop support
for old versions of persistent internal data structures, and
guarantees that we can reassign dd.tables.se_private_id for the DD
tables in subsequent steps.
ii) Execute inplace alter SQL statements, CREATE/DROP etc. to change
the table definitions. Preferably use only one transaction if we
are able to relax auto commit.
iii) Ensure cache consistency (e.g. dd cache vs. TDC). Re-read table
definitions and replace cache entries and re-generate serialized
dictionary information conforming to the updated DD schema.
iv) The stored SDI for the tables in the various tablespaces must be
re-generated. The same applies to tables in other SEs than InnoDB.
v) When upgrade is done, the DD version number is stored in the
dd.version table.
3.2.12 The server can initiate recovery of DML transactions by calling
handlerton::dict_recover(version, phase).
3.2.13 Now, the DD initialization is finished. For ordinary restart, the server
can continue the normal restart procedure. We may open and read any
table, both DD tables and user tables, as usual.
For upgrade, the server will eventually exit, and for ordinary use, it
must be restarted without the on the command line.
4. Error handling
-----------------
During upgrade, with InnoDB as the DDSE, the DD tablespace will be rebuilt.
Thus, in the event of a server crash in the middle of upgrade, the option is to
delete the new DD tablespace in the process of being rebuilt and restart the
upgrade. Deleting the leftover tablespace may be done automatically by the DDSE
when the upgrade is restarted.
While altering the DD tables as part of upgrade, it may make most sense to first
move the tables to the new DD tablespace, and then do the actual DD table
definition changes.
For a server crash in the middle of normal restart, the only option is to retry,
or possibly doing disaster recovery.
5. General requirements
-----------------------
The use cases above have some functional requirements in common.
5.1 Requirements implemented as of Phase one:
.............................................
FR-01 Refactor plugin initialization to allow the DD to be initialized after
the DDSE, but before the dynamic plugins.
FR-02 Refactor the existing bootstrap code to allow it to be used in various
contexts. Introduce a bootstrap function type for submitting function
parameters where bootstrapping should start.
FR-03 Introduce a Bootstrapper singleton or name space in the DD module to
coordinate the initialization of the DD.
FR-04 Provide a registry of all the DD table definitions.
FR-05 Represent the target definition of the DD tables by a hardcoded
structure.
FR-06 Generate the dd::Table object structures based on the hardcoded
structure.
FR-07 Create the table in the DDSE based on the hardcoded structure.
FR-08 Retrieve the temporarily stored, generated meta data
structures when a DD table is to be opened.
FR-09 Modify DDL execution to prohibit storing the meta data into the DD
tables, but instead save it temporarily.
FR-10 Execute statements for populating the tables.
FR-11 Represent the current version number hard coded in the dictionary
module.
FR-12 Define the dd.version table, add its table definition.
FR-13 It must be possible for the SE to ask for additional DD tables to be
created. This is the case for e.g. innodb_table_stats and
innodb_index_stats, which are currently created from definitions
hardcoded within the server.
FR-14 Consistency between the SE perception of the DD tablespace and the
corresponding meta data stored in the dd.tablespaces table must be
ensured before allowing the server to start.
[FR-15 The server must distinguish between the core and non-core tables. This
may limit the amount of hard coded data, but has the drawback of
complicationg the initialization and restart logic. This requirement
is considered optional.]
FR-16 Create the meta data representing the predefined tablespaces and add
this meta data to the shared DD cache.
FR-17 Transparent encryption will be implemented also for the shared
tablespaces such as the DD tablespace. We do not see that this will
cause any issues for bootstrapping, and therefore do not plan any
actions in this area; we assume that this will work out of the box.
5.2 Requirements related to upgrade (NOT implemented as part of this WL)
.............................................................................
These requirements will be implemented in the context where they become
relevant, e.g. within the scope of WL#6392, or when upgrade support is required.
FR-18 Support upgrade from 5.7 to 8.0, depending on WL#6392.
FR-19 Represent the definition of the previous DD version by a hardcoded
structure.
FR-20 Represent operations to upgrade from the previous to the current version.
FR-21 Remove the output parameters of dict_init() and move them to a new
method dict_internal_table_definitions(version, tables, artifacts), to
be invoked when the current dd.version is known.
WL#6381: Handler API changes for new dictionary
WL#6390: Use new DD API for handling non-partitioned tables
WL#6390: Use new DD API for handling non-partitioned tables
1. Overall bootstrapping architecture ===================================== We divide the bootstrapping architecture into various main areas that are further decomposed below. 1.1 Dictionary boostrapping, plugin- and server initialization -------------------------------------------------------------- The current (5.7) server startup and bootstrapping implementation needs to be refactored to allow for proper dictionary bootstrapping at the specific point where it must be done. 1.1.1 Refactor current bootstrap functions .......................................... The code currently doing the bootstrapping spawns a thread (which is waited for) which sets up a THD and reads, parses and executes SQL commands from a file (e.g. stdin). This code has to be refactored to allow using the same THD from within the DD to do the required bootstrapping there. Thus, the current bootstrap() function accepts an additional paramter, which is a function pointer. If NULL, the default bootstrap handler function is used, which reads SQL commands from the submitted file and executes them. If non-NULL, the submitted function is called. 1.1.2 Remove SQL statements from scripts/mysql_system_tables.sql ................................................................ This task is just deleting the creation of the new DD tables from the SQL script that is used as a basis for creating tables when starting the server with the --initialize option. The remainder of the script is left unchanged for now, but the long term goal is to move it into the server in the same way as we are now doing with the creation of the dictionary tables. 1.1.3 Integrate DD bootstrapping with DD initialization ....................................................... The bootstrapping is integrated with initialization of the dictionary. The DD initialization function is changed to accept a parameter indicating whether the server is started the first time or not. In dd::init(bool install), a new Dictionary_impl instance is created, and the dictionary is made available (assuming the tables already exist) or installed (creating new DD tables) depending on the submitted argument. This is done by submitting the appropriate bootstrap handler to the main bootstrap function. There is a dedicated singleton which coordinates the bootstrapping. This 'Bootstrapper' implements the restart() and initialize() methods. The Bootstrapper will be examined further below. 1.1.4 Hook up DD bootstrapping with mysqld startup .................................................. Before this WL, the DD was initialized right before loading the plugins, because the mysql.plugins table had to be present at that stage. The bootstrapping itself was being done at a much later stage. When removing the .FRM files, and integrating the dictionary initialization, installation and ordinary start, this has to be revised. The dictionary bootstrapping must be done after InnoDB is initialized (because the DD is stored in InnoDB), but before the dynamic plugins are loaded (because they assume that the mysql.plugins table is present). Thus, the plugin_init() function is split into four functions, two of which are externally visible. This allows init_server_components() to first load and initialize the builtin plugins (including the DDSE, i.e. InnoDB), then initialize the dictionary, and finally to load and initialize the dynamic plugins. 1.1.5 Command line options -------------------------- Before WL#7307, we needed a way to initialize the dictionary by submitting a command line option. For this purpose, the '--install-server' option was introduced. This option is still needed by external tools that do not use the '--initialize' option introduced in WL#7307. The '--install-server' option will be removed, and we will instead rely on the '--initialize' option that is already introduced. However, this cannot be done until mtr and other external tools all use the '--initialize' option. 1.1.6 Let the DDSE create additional tables ------------------------------------------- The hanlerton function dict_init() is called to initialize the DDSE. This function returns a list of descriptions of tables to be defined by the server on behalf of the DDSE. 1.1.7 Store meta data representing I_S tables --------------------------------------------- The I_S tables are represented in source code only, there are no real tables created. Thus, there is no meta data present in the DD tables reflecting the existence of the I_S tables. We suggest to introduce an I_S initialization function in sql_show.cc, where the table definition structures can be analyzed, and the corresponding dd.Table, dd.Columns etc. can be created and stored to make the meta data avilable in the DD tables. 1.2 Representing DD table definitions ------------------------------------- The DD objects that can be stored persistently are represented by objects having an associated singleton representing the object table. E.g., the interface dd::Table is implemented by dd::Table_impl, with an associated dd::tables::Tables singleton representing the object table. The object table class defines e.g. the table name, how to create keys for the table, etc. The definition of each DD table must be available in some form. We suggest to do this in the form of SQL statements associated with the object table singelton. 1.2.1 New DD Object_table_definition class .......................................... A new interface is introduced to support retrieving the SQL statements needed to create the table, add foreign keys, and to populate it. The interface is implemented by a class which also provides methods for adding and manipulating the SQL definition of the table. The statement is broken up into string fragments that may be combined to form a complete create table statement: 1.2.2 Add an Object_table_definition instance to Object_table ............................................................. An instance m_target_def of the class Object_type_definition is added to the Object_type_impl class to represent the target definition. Additional instances may be added to keep track of previous supported versions of the table, which may be needed in the case of upgrade. 1.3 Modifying the SQL DDL ------------------------- The MySQL server code which uses the dictionary must be changed in various ways for this bootstrap approach to work. DDL execution must be modified to recognize creation of DD tables, and to avoid storing the created meta data (dd::Table instance) into the DD tables in these situations, because the DD tables needed for storing the meta data may not be created yet. Instead of storing the meta data, it is added to the shared DD cache. This enable the cache to be accessed from elsewhere as usual, e.g. in open_table_def(). In a similar fashion, the execution of schema DDL is modified to recognize the creation of the DD schema. Additionally, in mysql_create_table_no_lock(), we set a parameter for creating handler files in create_table_impl() for DD tables only if we are doing initial start. This is necessary to enable doing CREATE TABLE for DD tables at restart for having the structures required for opening the tables generated, without actually creating the table. 1.5 The new DD Bootstrapper class --------------------------------- For coordinating bootsrapping of the dictionary, a dedicated singleton will be used. Alternatively, this may be implemented by functions in a dedicated namespace. The bootstrapper implements two main methods that are called when the DD is initialized. 'initialize()' will create new DD tables, while 'restart()' will assume that the DD already exists, and try opening the existing tables.
1. Extensions of the handlerton API
===================================
1.1 Initialization of the DDSE
------------------------------
enum dict_init_mode_t
{
DICT_INIT_CREATE_FILES, //< Create all required SE files
DICT_INIT_CREATE_MISSING_FILES, //< Use files that already exist
DICT_INIT_CHECK_FILES, //< Verify existence of expected files
DICT_INIT_IGNORE_FILES //< Don't care about files at all
};
/**
Initialize the SE for being used to store the DD tables. Create
the required files according to the dict_init_mode. Create strings
representing the required DDSE tables, i.e., tables that the DDSE
expects to exist in the DD, and add them to the appropriate out
parameter.
@param dict_init_mode How to initialize files
@param version Target DD version if a new
server is being installed.
0 if restarting an existing
server.
@param [out] DDSE_tables List of SQL DDL statements
for creating DD tables that
are needed by the DDSE.
@param [out] DDSE_artifacts List of SQL DML statements
for adding meta data for DD
entities created by the DDSE,
e.g. inserting a row in the
dd.tablespaces table.
@retval true An error occurred.
@retval false Success - no errors.
*/
bool (*dict_init)(dict_init_mode_t dict_init_mode,
uint version,
List &DDSE_tables,
List &DDSE_artifacts);
1.2 Getting the se_private_data for the DD tables
-------------------------------------------------
/**
Submit a list of dd::Table objects representing the core DD tables
having hardcoded data to be filled in by the DDSE. This function can be
used for retrieving the hard coded SE private data for the dd.version
table, before creating for opening it (submitting version = 0), or for
retrieving the hard coded SE private data for a list of core tables,
before creating or opening them (submit version == the actual version
which was read from the dd.version table).
@param dd_tables[in,out] List of dd::Table objects representing
the core DD tables.
@param version Actual version of the DD.
@retval true An error occurred.
@retval false Success - no errors.
*/
bool ha_get_se_private_data(List dd_tables, uint version);
Copyright (c) 2000, 2025, Oracle Corporation and/or its affiliates. All rights reserved.