MySQL 8.0.32
Source Code Documentation
ddl Namespace Reference

The general architecture is that the work is done in two phases, roughly the read and write phase. More...

Classes

struct  Aligned_buffer
 Buffer used for reading and writing to the temporary files. More...
 
struct  Builder
 For loading indexes. More...
 
struct  Compare_key
 Compare the keys of an index. More...
 
struct  Context
 DDL context/configuration. More...
 
struct  Copy_ctx
 Context for copying cluster index row for the index to being created. More...
 
struct  Cursor
 Cursor for reading the data. More...
 
struct  Dup
 Structure for reporting duplicate records. More...
 
struct  Fetch_sequence
 Fetch the document ID from the table. More...
 
struct  File_cursor
 For loading a Btree index from a file. More...
 
struct  File_reader
 Read rows from the temporary file. More...
 
struct  file_t
 Information about temporary files used in merge sort. More...
 
struct  FTS
 Full text search index builder. More...
 
struct  Gen_sequence
 Generate the next document ID using a monotonic sequence. More...
 
struct  Index_defn
 Definition of an index being created. More...
 
struct  Index_field
 Index field definition. More...
 
struct  Insert
 Structure stores information needed for the insertion phase of FTS parallel sort. More...
 
struct  Key_sort_buffer
 Buffer for sorting in main memory. More...
 
struct  Key_sort_buffer_cursor
 For loading an index from a sorted buffer. More...
 
struct  Load_cursor
 
class  Loader
 Build indexes on a table by reading a clustered index, creating a temporary file containing index entries, merge sorting these index entries and inserting sorted index entries to indexes. More...
 
struct  Merge_cursor
 Merge the sorted files. More...
 
struct  Merge_file_sort
 Merge the blocks in the file. More...
 
struct  Parallel_cursor
 Cursor used for parallel reads. More...
 
struct  PFS_buffer
 For DDL memory allocations that use the mem_key_ddl handle. More...
 
struct  Row
 Physical row context. More...
 
class  RTree_inserter
 Class that caches RTree index tuples made from a single cluster index page scan, and then insert into corresponding index tree. More...
 
struct  Sequence
 Generate the next autoinc based on a snapshot of the session auto_increment_increment and auto_increment_offset variables. More...
 
struct  Token
 Row fts token for plugin parser. More...
 
struct  Tokenize_ctx
 Structure stores information from string tokenization operation. More...
 
class  Unique_os_file_descriptor
 Captures ownership and manages lifetime of an already opened OS file descriptor. More...
 

Typedefs

using mrec_buf_t = byte[UNIV_PAGE_SIZE_MAX]
 Secondary buffer for I/O operations of merge records. More...
 
using mrec_t = byte
 Merge record in Aligned_buffer. More...
 
using IO_buffer = std::pair< byte *, os_offset_t >
 Block size for DDL I/O operations. More...
 
using Latch_release = std::function< dberr_t()>
 Called when a log free check is required. More...
 
using Builders = std::vector< Builder *, ut::allocator< Builder * > >
 
using Merge_offsets = std::deque< os_offset_t, ut::allocator< os_offset_t > >
 Start offsets in the file, from where to merge records. More...
 

Enumerations

enum class  Thread_state : uint8_t { UNKNOWN , COMPLETE , EXITING , ABORT }
 status bit used for communication between parent and child thread More...
 

Functions

template<typename T , typename Compare >
void merge_sort (T *arr, T *aux_arr, const size_t low, const size_t high, Compare compare)
 Merge sort a given array. More...
 
static void index_build_failed (dict_index_t *index) noexcept
 Note that an index build has failed. More...
 
dberr_t pread (os_fd_t fd, void *ptr, size_t len, os_offset_t offset) noexcept
 Read a merge block from the file system. More...
 
dberr_t pwrite (os_fd_t fd, void *ptr, size_t size, os_offset_t offset) noexcept
 Write a merge block to the file system. More...
 
Unique_os_file_descriptor file_create_low (const char *path) noexcept
 Create temporary merge files in the given parameter path, and if UNIV_PFS_IO defined, register the file descriptor with Performance Schema. More...
 
bool file_create (file_t *file, const char *path) noexcept
 Create a merge file int the given location. More...
 
dict_index_tcreate_index (trx_t *trx, dict_table_t *table, const Index_defn *index_def, const dict_add_v_col_t *add_v) noexcept
 Create the index and load in to the dictionary. More...
 
dberr_t drop_table (trx_t *trx, dict_table_t *table) noexcept
 Drop a table. More...
 
dberr_t lock_table (trx_t *trx, dict_table_t *table, lock_mode mode) noexcept
 Sets an exclusive lock on a table, for the duration of creating indexes. More...
 
static void mark_secondary_indexes (trx_t *trx, dict_table_t *table) noexcept
 We will have to drop the secondary indexes later, when the table is in use, unless the the DDL has already been externalized. More...
 
static void drop_secondary_indexes (trx_t *trx, dict_table_t *table) noexcept
 Invalidate all row_prebuilt_t::ins_graph that are referring to this table. More...
 
void drop_indexes (trx_t *trx, dict_table_t *table, bool locked) noexcept
 Drop those indexes which were created before an error occurred. More...
 
std::ostream & operator<< (std::ostream &out, const Merge_file_sort::Range &range) noexcept
 Print range. More...
 

Variables

long fill_factor
 Innodb B-tree index fill factor for bulk load. More...
 
ulong fts_parser_threads = 2
 Variable specifying the number of FTS parser threads to use. More...
 
constexpr size_t IO_BLOCK_SIZE = 4 * 1024
 Minimum IO buffer size. More...
 
static constexpr size_t SERVER_CLUSTER_INDEX_ID = 0
 Cluster index ID (always the first index). More...
 

Detailed Description

The general architecture is that the work is done in two phases, roughly the read and write phase.

The scanner pushes the document to a read handler queue for processing.

Phase I: Start several parsing/tokenization threads that read the document from a queue, parse the document, tokenize the document, add them to a buffer, sort the rows in the buffer and then write the buffer to a temporary file. There is one file per auxiliary table per parser instance. So, if you have 2 parse threads you will end up with:

2 x FTS_NUM_AUX_INDEX files.

Phase 2: The temporary files generated during phase I are not closed but passed to the second (write) phase so that these temporary files can be merged and the rows inserted into the new FTS index. Using the example from above, create FTS_NUM_AUX_INDEX threads and each thread will merge 2 files.

Typedef Documentation

◆ Builders

using ddl::Builders = typedef std::vector<Builder *, ut::allocator<Builder *> >

◆ IO_buffer

using ddl::IO_buffer = typedef std::pair<byte *, os_offset_t>

Block size for DDL I/O operations.

The minimum is UNIV_PAGE_SIZE, or page_get_free_space_of_empty() rounded to a power of 2.

◆ Latch_release

using ddl::Latch_release = typedef std::function<dberr_t()>

Called when a log free check is required.

◆ Merge_offsets

using ddl::Merge_offsets = typedef std::deque<os_offset_t, ut::allocator<os_offset_t> >

Start offsets in the file, from where to merge records.

◆ mrec_buf_t

Secondary buffer for I/O operations of merge records.

This buffer is used for writing or reading a record that spans two Aligned_buffer. Thus, it must be able to hold one merge record, whose maximum size is the same as the minimum size of Aligned_buffer.

◆ mrec_t

using ddl::mrec_t = typedef byte

Merge record in Aligned_buffer.

The format is the same as a record in ROW_FORMAT=COMPACT with the exception that the REC_N_NEW_EXTRA_BYTES are omitted.

Enumeration Type Documentation

◆ Thread_state

enum class ddl::Thread_state : uint8_t
strong

status bit used for communication between parent and child thread

Enumerator
UNKNOWN 
COMPLETE 
EXITING 
ABORT 

Function Documentation

◆ create_index()

dict_index_t * ddl::create_index ( trx_t trx,
dict_table_t table,
const Index_defn index_def,
const dict_add_v_col_t add_v 
)
noexcept

Create the index and load in to the dictionary.

Parameters
[in,out]trxTrx (sets error_state)
[in,out]tableThe index is on this table
[in]index_defThe index definition
[in]add_vNew virtual columns added along with add index call
Returns
index, or nullptr on error

◆ drop_indexes()

void ddl::drop_indexes ( trx_t trx,
dict_table_t table,
bool  locked 
)
noexcept

Drop those indexes which were created before an error occurred.

The data dictionary must have been locked exclusively by the caller, because the transaction will not be committed.

Parameters
[in,out]trxTransaction
[in]tableTable to lock.
[in]lockedtrue=table locked, false=may need to do a lazy drop

◆ drop_secondary_indexes()

static void ddl::drop_secondary_indexes ( trx_t trx,
dict_table_t table 
)
staticnoexcept

Invalidate all row_prebuilt_t::ins_graph that are referring to this table.

That is, force row_get_prebuilt_insert_row() to rebuild prebuilt->ins_node->entry_list).

Parameters
[in,out]trxTransaction
[in]tableTable that owns the indexes.

◆ drop_table()

dberr_t ddl::drop_table ( trx_t trx,
dict_table_t table 
)
noexcept

Drop a table.

The caller must have ensured that the background stats thread is not processing the table. This can be done by calling dict_stats_wait_bg_to_stop_using_table() after locking the dictionary and before calling this function.

Parameters
[in,out]trxTransaction
[in,out]tableTable to drop.
Returns
DB_SUCCESS or error code

◆ file_create()

bool ddl::file_create ( file_t file,
const char *  path 
)
noexcept

Create a merge file int the given location.

Parameters
[out]fileTemporary generated during DDL.
[in]pathLocation for creating temporary file
Returns
true if file is created successfully

◆ file_create_low()

Unique_os_file_descriptor ddl::file_create_low ( const char *  path)
noexcept

Create temporary merge files in the given parameter path, and if UNIV_PFS_IO defined, register the file descriptor with Performance Schema.

Parameters
[in]pathLocation for creating temporary merge files.
Returns
File descriptor

◆ index_build_failed()

static void ddl::index_build_failed ( dict_index_t index)
staticnoexcept

Note that an index build has failed.

Parameters
[in,out]indexIndex that failed to build.

◆ lock_table()

dberr_t ddl::lock_table ( trx_t trx,
dict_table_t table,
lock_mode  mode 
)
noexcept

Sets an exclusive lock on a table, for the duration of creating indexes.

Parameters
[in,out]trxTransaction
[in]tableTable to lock.
[in]modeLock mode LOCK_X or LOCK_S
Returns
error code or DB_SUCCESS

◆ mark_secondary_indexes()

static void ddl::mark_secondary_indexes ( trx_t trx,
dict_table_t table 
)
staticnoexcept

We will have to drop the secondary indexes later, when the table is in use, unless the the DDL has already been externalized.

Mark the indexes as incomplete and corrupted, so that other threads will stop using them. Let dict_table_close() or crash recovery or the next invocation of prepare_inplace_alter_table() take care of dropping the indexes.

Parameters
[in,out]trxTransaction
[in]tableTable that owns the indexes.

◆ merge_sort()

template<typename T , typename Compare >
void ddl::merge_sort ( T *  arr,
T *  aux_arr,
const size_t  low,
const size_t  high,
Compare  compare 
)
inline

Merge sort a given array.

Parameters
[in,out]arrArray to sort.
[in,out]aux_arrAuxiliary space to use for sort.
[in]lowFirst element (inclusive).
[in]highNumber of elements to sort from low.
[in]compareFunction to compare two elements.

◆ operator<<()

std::ostream & ddl::operator<< ( std::ostream &  out,
const Merge_file_sort::Range range 
)
noexcept

Print range.

Parameters
[in,out]outOutput stream.
[in]rangeRange to print.
Returns
output stream.

◆ pread()

dberr_t ddl::pread ( os_fd_t  fd,
void *  ptr,
size_t  len,
os_offset_t  offset 
)
noexcept

Read a merge block from the file system.

Parameters
[in]fdfile descriptor.
[out]ptrBuffer to read into.
[in]lenNumber of bytes to read.
[in]offsetByte offset to start reading from.
Returns
DB_SUCCESS or error code

◆ pwrite()

dberr_t ddl::pwrite ( os_fd_t  fd,
void *  ptr,
size_t  size,
os_offset_t  offset 
)
noexcept

Write a merge block to the file system.

Parameters
[in]fdFile descriptor
[in]ptrBuffer to write.
[in]sizeNumber of bytes to write.
[in]offsetByte offset where to write.
Returns
DB_SUCCESS or error code

Variable Documentation

◆ fill_factor

long ddl::fill_factor

Innodb B-tree index fill factor for bulk load.

◆ fts_parser_threads

ulong ddl::fts_parser_threads = 2
extern

Variable specifying the number of FTS parser threads to use.

Parallel sort degree, must be a power of 2.

◆ IO_BLOCK_SIZE

constexpr size_t ddl::IO_BLOCK_SIZE = 4 * 1024
constexpr

Minimum IO buffer size.

◆ SERVER_CLUSTER_INDEX_ID

constexpr size_t ddl::SERVER_CLUSTER_INDEX_ID = 0
staticconstexpr

Cluster index ID (always the first index).