WL#5457: Add an LRU list to the data dictionary

Status: Complete   —   Priority: Medium

Close and remove non-system tables from the InnoDB data dictionary. We also
make an exception for tables that are either referenced by another table
or refer to another table in a foreign key relationship.

This should address BUG#20877.
For tables that have a reference count of zero, are not referenced by any
foreign key and don't refer to any table (FK) are removed from the dictionary
cache if it reaches an upper bound. The count is controlled by the parameter

We only make a best effort to remove, there is nothing stopping the data
dictionary from going beyond the upper limit.

Added functions:


Rename  dict_table_decrement_handle_count() to dict_table_close()
Rename dict_table_get_on_id() to  dict_table_open_on_id()
Rename dict_table_get() to dict_table_open_on_name()

Fields added to dict_sys_t:

   UT_LIST_BASE_NODE_T(dict_table_t)     table_non_LRU
                               /*!< List of tables that can't be evicted from
the cache. */

Fields added to dict_table_t:
       unsigned        can_be_evicted:1;
                               /*!< TRUE if it's not an InnoDB system table
                               or a table that has no FK relationships */

       ulint           n_rec_locks;
                               /*!< Count of the number of record locks on
                               this table. We use this to determine whether
                               we can evict the table from the dictionary
                               cache. It is protected by the lock mutex. */

       ulint           n_ref_count;
                               /*!< count of how many handles are opened
                               to this table; dropping of the table is
                               NOT allowed until this count gets to zero;
                               MySQL does NOT itself check the number of
                               open handles at drop */

Added following enumerations to sym_tab_entry.

       SYM_UNSET,       /*!< Unset entry. */
                                /*!< database table name, ref counted. Must
                               be closed explicitly. */

Added new global variable srv_dict_table_cache_size. This is tied to the
--table-cache-size parameter.

With this change all InnoDB tables should be opened using the public interface
of dict0dict.c and not the low-level equivalent functions, this applies to
internal InnoDB calls too. I've made the low level variants static and private
to the dict/ source files. The tables must be closed after use. We use this to
reference count all opens including open requests from MySQL layer. Previously
we only reference counted open requests from the MySQL layer. Several changes
were made to existing code to conform to the new behavior. The changes are too
numerous to list individually.

When opening or closing a table the only important requirement is whether the
caller has the dict_sys_t::mutex locked or not. This state is passed to the
open/close functions using a flag. Which is set to TRUE if the caller has locked
the dict mutex or FALSE if it is not locked.

The internal SQL parser has been updated to use the new interfaces too. A new
symbol type is introduced for reference counted tables. In fact we don't open
the table multiple times as we did in the past.

The master thread will attempt to keep the dictionary table cache within limits
periodically. Because this check can be expensive we limit it to once every 60
seconds and scan only 50% of the LRU list when the server is active and 100%
when the server is idle.

Before a table is dropped we move it from the LRU list to the non-LRU list this
is to ensure that the table is not asynchronously evicted from the cache.

Added several debug assertions that check whether a table has any kind of lock
on the table. There was a bug in existing code that resulted in a table being
dropped while it had record locks on the table. The record locks were implicit
locks of recovered transactions that had been converted to explicit locks.

See rb://379.