WL#3574: Online backup: Service API for Metadata Objects II

Rationale
---------
To make a alpha-quality interface between server kernel and backup kernel.
Stabilizing this interface is WL#4264.

Overview
--------
The online backup system requires server functionality to permit the access to 
metadata concerning database objects (e.g., CREATE DATABASE). This 
functionality is not unique to online backup and therefore shall be designed to 
support access to metadata using a generalized interface.

Background
----------
One of the driving goals of the online backup system is to minimize reliance on 
functionality in the MySQL server proper. That is, to abstract the backup code 
from the server code as much as possible in order to avoid conditions where 
changes in one breaks the other. While a complete abstraction is not possible 
(the backup code needs to communicate with the server in a variety of ways), it 
is possible to isolate the affects of changes to either the online backup or 
server code by using an intermediate layer. The term "service interface" has 
been given to this layer to indicate its goal: to provide a buffer between the 
services needed by online backup and how those services are provided by the 
server.

Requirements
------------
The purpose of this worklog is to establish a service interface that can be 
used to gather metadata about database objects, iterate through the various 
types of objects supported, and to create database objects on restore. The 
deliverable for this worklog is a service interface that implements these 
features (at a minimum) for the following database objects.

* Trigger (WL#3582) [40]
* View              [37]
* Stored Procedure  [38] 
* Stored Function   [39]
* Event             [41]

Note: The [] references a number assigned to each object described in WL#3713. 
There is a complete list of objects described in WL#3713. 

The service interface shall be designed to allow the following operations 
concerning metadata.

* The retrieval of metadata concerning a specific object.
* The recreation of a specific object using the retrieved metadata.
* Enumeration for each object supported and for all objects of a given type.
* Abstractions of the DDL blocker to permit isolation of DDL blocker.
* Dependency checking for objects to discover objects that need to be
  created/exist before a given object is created.

The service interface shall be designed using abstractions to represent the 
above functionality. 

The retrieval of metadata shall be called 'serialize' and the output of the 
serialize() method called the serialization of the object.

The serialization of the object can be used to create an instance of the 
service interface class and this process shall be called materialize.

The process of using the materialized serialization and creating a new object 
on the server shall be referred as executing the serialization.

Thus, an object can be serialized for storage in the online backup and 
materialized on restore and the serialization executed to create the database 
object on the server.

For example, the serialization, e.g. db->serialize() of a database shall 
produce a serialization that can be stored in the online backup image file. 
Upon restore, the service interface can be used to materialize this data using 
db->materialize(serialized_data). To create the object on the server (to create 
the database), the service interface would allow db->execute() to create the 
database 'xyz'. All other objects supported in this worklog shall use this 
mechanism.

Architecture
------------
The high-level architecture is given in the following diagram.

+-----------+     +---------------+
|    |     | online backup |
|-----------+     +---------------+
      |                   |
      |                   |
      +--------+----------+
               |
               |
      +--------+----------+
      | service interface |
      +--------+----------+
               |
               |
       +-------+---------+
       | server (mysqld) |
       +-------+---------+

This diagram illustrates that the service interface provides an abstraction of 
server functionality for calling applications such as mysqldump or online 
backup. The advantage is that the server interface can be used to isolate 
changes to the server from the applications and vice-versa. Furthermore, the 
service interface can be improved independently of either caller or callee.

Decision
--------
A decision was made to serialize the object metadata using the currently 
available show_create_* methods. This implementation of the serialization shall 
produce a string obtained in the same way as in SHOW CREATE statements. 
However, there is a problem related to the fact that output of SHOW CRATE 
statement depends on the setting of sql_mode variable (see BUG#33570 and 
BUG#33571). If the mode is anything else than the "native" mysql mode then 
CREATE statements can miss some important clauses (like "ENGINE=xxx" for a 
table). Also, when object is re-created using the full CREATE statement for it, 
sql_mode must be set to "native" as otherwise not all clauses will be 
understood by the server. All this must be taken into account when implementing 
serialization via CREATE statements.

Note: In the future, the serialization may not be based on CREATE statements 
and therefore the caller of the service interface should make no assumptions 
about the content of the serialization other than what has been stated in the 
above sections.

Known Limitation
----------------
When objects are described by CREATE statements, they are created by executing 
these statements. Currently this is done by passing CREATE statement to 
mysql_parse() function. However, this function can not be called from inside a 
stored routine - if it happens, an assertion is violated and server crashes 
(see BUG#33563). Therefore, it has been decided to not permit the inclusion of 
the BACKUP or RESTORE command inside a stored routing. See solution for 
BUG#33563 for more details.

Future Work
-----------
It has been suggested that the ownership of the service interface could be the 
runtime team. If ownership of this interface is transferred to the runtime 
team, it is requested that they refactor the implementation of the service 
interface however they see fit. It was suggested that the 
serialize/materialize/execute mechanism is sufficient to allow the 
implementation to be changed to use something other than CREATE SQL statements. 
In such a case, the design shall incorporate a mechanism to allow 
identification of versions of the service interface.

It has been decided to leave out the complete implementation of dependency 
checking for views and other objects. Additional work is needed to understand 
all of the requirements for dependency checking WRT backup and restore. See 
WL#4211 and WL#4225.

The service interface shall be implemented in the ./sql source folder and named 
si_objects.*. These source files shall comprise the interface specification and 
code for the following requirements.

Object Class
------------
The service interface shall implement a general object class named Obj. This 
class shall have, at a minimum, the following methods. This shall become an 
abstract class from which each database object support (trigger, event, etc.) 
shall derive.

class Obj 
{ 
public:
  virtual bool serialize(THD *thd, String *serialialization) = 0;
  virtual const String *get_name() = 0;
  virtual const String *get_db_name() = 0;
  virtual bool execute(THD *thd) = 0;
  virtual ~Obj() { }

private:
  virtual bool materialize(uint serialization_version,
                           const String *serialialization) = 0;
  virtual bool drop(THD *thd) = 0;

};

In addition to the serialize(), materialize(), and execute() methods discussed 
in the high level specification, the methods get_name() and get_db_name() are 
provided to allow identification of the object instance. The method drop() is 
included to permit a general call to destroy an object on the server. For 
example to drop a database, a child class could be materialized for the 
database then drop() can be called to execute DROP DATABASE ... on the server.

The serialization mechanism shall be implemented to support versioning of the 
interface. The initial version shall be 0 and it shall not be considered in 
future revisions for backward compatibility. The reason is version 0 implements 
serialization using CREATE commands which may not be the best implementation 
for future plans for the server. Thus, starting with version 1, the service 
interface shall require the version number present when services are requested 
and shall provide backward compatibility in future versions of the service 
interface.

Iterators
---------
The iterators for the database objects shall be a simple class that permits the 
instantiation of the iterator and a forward-only increment mechanism. The 
abstract class for the iterator functionality is shown below.

class ObjIterator
{
public:
  ObjIterator() { }
  virtual Obj *next() = 0;

public:
  virtual ~ObjIterator() { }
};

Versioning of Service Interface
-------------------------------
The service interface shall use as its versioning the version number of the 
server. When a caller uses the serialization mechanism, she must record the 
version number of the server and store it with the serialization data. Upon 
materialization, the caller passes the version number in the method call as 
shown.

materialize(uint serialization_version,
            const String *serialialization);

This way, the service interface shall be capable of supporting services for 
multiple releases. Note that the first release of this interface shall use 
0 in place of the version number as described above.

Convenience Methods
-------------------
The service interface shall implement convenience methods to reduce the burden 
of the caller to create instances of objects, iterators, and materialization.

A method for each database object supported shall be created to allow the 
instantiation of a single object shall be provided using the following or 
similar method.

Obj *get_database(const String *db_name);

A method for each database object supported shall be created to allow the 
creation of an object iterator. For example, to get an iterator for all of the 
databases on a server, the service interface shall implement the following or 
similar method.

ObjIterator *get_databases(THD *thd);

A method for each database object supported shall be created to allow the 
materialization of a single object. The following or similar method shall be 
implemented.

Obj *materialize_database(const String *db_name,
                          uint serialization_version,
                          const String *serialialization);

DDL Blocker
-----------
The following methods shall be implemented to abstract the DDL blocker methods 
already implemented in the server. These methods are necessary to allow the 
replacement of the DDL blocker code in the future without affecting the code in 
the caller.

bool ddl_blocker_enable(THD *thd);
void ddl_blocker_disable();
void ddl_blocker_exception_on(THD *thd);
void ddl_blocker_exception_off(THD *thd);

Helper Methods
--------------
Aside from the methods listed above, the following helper methods shall be 
implemented in the service interface. Their function is shown above each.

/*
   Check to see if db_name is an internal database such as mysql
   of information_schema.
*/
bool is_internal_db_name(const String *db_name);

/*
  Check if the given directory actually exists.
*/
bool check_db_existence(const String *db_name);