MySQL 9.0.0
Source Code Documentation
Fil_system Class Reference

The tablespace memory cache. More...

Public Types

using Fil_shards = std::vector< Fil_shard * >
 

Public Member Functions

 Fil_system (size_t n_shards, size_t max_open)
 Constructor. More...
 
 ~Fil_system ()
 Destructor. More...
 
fil_space_tspace_acquire (space_id_t space_id, bool silent)
 Acquire a tablespace when it could be dropped concurrently. More...
 
Tablespace_dirs::Result get_scanned_filename_by_space_id (space_id_t space_id)
 Fetch the file names opened for a space_id during recovery. More...
 
Tablespace_dirs::Result get_scanned_filename_by_space_num (space_id_t space_num, space_id_t &space_id)
 Fetch the file name opened for an undo space number. More...
 
bool get_file_by_space_id (space_id_t space_id, std::string &name)
 Fetch the file name opened for a space_id from the file map. More...
 
bool get_file_by_space_num (space_id_t space_num, space_id_t &space_id, std::string &name)
 Fetch the file name opened for an undo space number. More...
 
bool erase_path (space_id_t space_id)
 Erase a tablespace ID and its mapping from the scanned files. More...
 
void add_old_file (const std::string &file_path)
 Add file to old file list. More...
 
void rename_partition_files (bool revert)
 Rename partition files during upgrade. More...
 
void clear_old_files ()
 Clear all accumulated old files. More...
 
const std::string & get_root (const std::string &path) const
 Get the top level directory where this filename was found. More...
 
dberr_t prepare_open_for_business (bool read_only_mode)
 Update the DD if any files were moved to a new location. More...
 
void flush_file_spaces ()
 Flush to disk the writes in file spaces possibly cached by the OS (note: spaces of type FIL_TYPE_TEMPORARY are skipped) More...
 
void purge ()
 Clean up the shards. More...
 
size_t count_undo_deleted (space_id_t undo_num)
 Count how many truncated undo space IDs are still tracked in the buffer pool and the file_system cache. More...
 
bool is_deleted (space_id_t space_id) noexcept
 Check if a particular undo space_id for a page in the buffer pool has been deleted recently. More...
 
fil_space_tget_space_by_name (const char *name)
 Fetch the fil_space_t instance that maps to the name. More...
 
bool is_greater_than_max_id (space_id_t space_id) const
 Check a space ID against the maximum known tablespace ID. More...
 
void set_maximum_space_id (const fil_space_t *space)
 Update the maximum known tablespace ID. More...
 
void update_maximum_space_id (space_id_t space_id)
 Update the maximum known space ID if it's smaller than max_id. More...
 
bool assign_new_space_id (space_id_t *space_id)
 Assigns a new space id for a new single-table tablespace. More...
 
void wait_while_ios_in_progress () const
 Allows other threads to advance work while we wait for I/Os to complete. More...
 
bool close_file_in_all_LRU ()
 Tries to close a file in all the LRU lists. More...
 
void open_all_system_tablespaces ()
 Opens all system tablespace data files in all shards. More...
 
void close_all_files ()
 Close all open files. More...
 
size_t get_minimum_limit_for_open_files (size_t n_files_not_belonging_in_lru) const
 Returns minimum open files limit to be set to allow the specified number of non-LRU files opened. More...
 
bool set_open_files_limit (size_t &new_max_open_files)
 Changes the maximum opened files limit. More...
 
size_t get_open_files_limit () const
 Returns maximum number of allowed opened files. More...
 
dberr_t iterate (Fil_iterator::Function &f)
 Iterate through all persistent tablespace files (FIL_TYPE_TABLESPACE) returning the nodes via callback function cbk. More...
 
size_t encryption_rotate ()
 Rotate the tablespace keys by new master key. More...
 
void encryption_reencrypt (std::vector< space_id_t > &sid_vector)
 Reencrypt tablespace keys by current master key. More...
 
void space_detach (fil_space_t *space)
 Detach a space object from the tablespace memory cache. More...
 
space_id_t get_max_space_id () const
 
bool lookup_for_recovery (space_id_t space_id)
 Lookup the tablespace ID. More...
 
dberr_t open_for_recovery (space_id_t space_id)
 Open a tablespace that has a redo log record to apply. More...
 
bool check_missing_tablespaces ()
 This function should be called after recovery has completed. More...
 
void moved (dd::Object_id object_id, space_id_t space_id, const char *space_name, const std::string &old_path, const std::string &new_path, bool before_90000)
 Note that a file has been relocated. More...
 
bool check_path (const std::string &path) const
 Check if a path is known to InnoDB. More...
 
std::string get_dirs () const
 Get the list of directories that InnoDB knows about. More...
 
void set_scan_dir (const std::string &directory, bool is_undo_dir)
 Normalize and save a directory to scan for IBD and IBU datafiles before recovery. More...
 
void set_scan_dirs (const std::string &directories)
 Normalize and save a list of directories to scan for IBD and IBU datafiles before recovery. More...
 
dberr_t scan ()
 Scan the directories to build the tablespace ID to file name mapping table. More...
 
Fil_shardshard_by_id (space_id_t space_id) const
 Fil_shard by space ID. More...
 
void mutex_acquire_all () const
 Acquire all the mutexes. More...
 
void mutex_release_all () const
 Release all the mutexes. More...
 
bool validate () const
 Checks the consistency of the tablespace cache. More...
 
bool mutex_owned_all () const
 Check if all mutexes are owned. More...
 
dberr_t rename_tablespace_name (space_id_t space_id, const char *old_name, const char *new_name)
 Rename a tablespace. More...
 
void free_scanned_files ()
 Free the data structures required for recovery. More...
 

Static Public Member Functions

static size_t get_limit_for_non_lru_files (size_t open_files_limit)
 Returns maximum number of allowed non-LRU files opened for a specified open files limit. More...
 
static bool space_belongs_in_LRU (const fil_space_t *space)
 Determines if a file belongs to the least-recently-used list. More...
 
static space_id_t get_tablespace_id (const std::string &filename)
 Get the tablespace ID from an .ibd and/or an undo tablespace. More...
 

Private Member Functions

fil_load_status ibd_open_for_recovery (space_id_t space_id, const std::string &path, fil_space_t *&space)
 Open an ibd tablespace and add it to the InnoDB data structures. More...
 
 Fil_system (Fil_system &&)=delete
 
 Fil_system (const Fil_system &)=delete
 
Fil_systemoperator= (const Fil_system &)=delete
 

Private Attributes

Fil_shards m_shards
 Fil_shards managed. More...
 
fil::detail::Open_files_limit m_open_files_limit
 
space_id_t m_max_assigned_id
 Maximum space id in the existing tables, or assigned during the time mysqld has been up; at an InnoDB startup we scan the data dictionary and set here the maximum of the space id's of the tables there. More...
 
bool m_space_id_reuse_warned
 true if fil_space_create() has issued a warning about potential space_id reuse More...
 
dd_fil::Tablespaces m_moved
 List of tablespaces that have been relocated. More...
 
Tablespace_dirs m_dirs
 Tablespace directories scanned at startup. More...
 
std::vector< std::string > m_old_paths
 Old file paths during 5.7 upgrade. More...
 
std::atomic_size_t m_next_shard_to_close_from_LRU {}
 Next index (modulo number of shards) to try to close a file from the LRU list to distribute closures evenly between the shards. More...
 
std::atomic_size_t m_n_files_not_belonging_in_lru {}
 Current number of files that are not belonging in LRU. More...
 
ib::Throttler m_MANY_NON_LRU_FILES_OPENED_throttler {}
 Throttles messages about high files not belonging in LRU count, the warning ER_IB_WARN_MANY_NON_LRU_FILES_OPENED. More...
 
ib::Throttler m_TRYING_TO_OPEN_FILE_FOR_LONG_TIME_throttler {}
 Throttles messages about long waiting for opened files limit, the warning ER_IB_MSG_TRYING_TO_OPEN_FILE_FOR_LONG_TIME. More...
 
ib::Throttler m_ACCESSING_NONEXISTINC_SPACE_throttler {}
 Throttles messages about accessing space that was already removed, the warning ACCESSING_NONEXISTINC_SPACE. More...
 

Friends

class Fil_shard
 

Detailed Description

The tablespace memory cache.

Member Typedef Documentation

◆ Fil_shards

using Fil_system::Fil_shards = std::vector<Fil_shard *>

Constructor & Destructor Documentation

◆ Fil_system() [1/3]

Fil_system::Fil_system ( size_t  n_shards,
size_t  max_open 
)

Constructor.

Parameters
[in]n_shardsNumber of shards to create
[in]max_openMaximum number of open files

◆ ~Fil_system()

Fil_system::~Fil_system ( )

Destructor.

◆ Fil_system() [2/3]

Fil_system::Fil_system ( Fil_system &&  )
privatedelete

◆ Fil_system() [3/3]

Fil_system::Fil_system ( const Fil_system )
privatedelete

Member Function Documentation

◆ add_old_file()

void Fil_system::add_old_file ( const std::string &  file_path)
inline

Add file to old file list.

The list is used during 5.7 upgrade failure to revert back the modified file names. We modify partitioned file names to lower case.

Parameters
[in]file_pathold file name with path

◆ assign_new_space_id()

bool Fil_system::assign_new_space_id ( space_id_t space_id)

Assigns a new space id for a new single-table tablespace.

This works simply by incrementing the global counter. If 4 billion ids is not enough, we may need to recycle ids.

Parameters
[out]space_idSet this to the new tablespace ID
Returns
true if assigned, false if not

◆ check_missing_tablespaces()

bool Fil_system::check_missing_tablespaces ( )

This function should be called after recovery has completed.

Check for tablespace files for which we did not see any MLOG_FILE_DELETE or MLOG_FILE_RENAME record. These could not be recovered.

Returns
true if there were some filenames missing for which we had to ignore redo log records during the apply phase

◆ check_path()

bool Fil_system::check_path ( const std::string &  path) const
inline

Check if a path is known to InnoDB.

Parameters
[in]pathPath to check
Returns
true if path is known to InnoDB

◆ clear_old_files()

void Fil_system::clear_old_files ( )
inline

Clear all accumulated old files.

◆ close_all_files()

void Fil_system::close_all_files ( )

Close all open files.

◆ close_file_in_all_LRU()

bool Fil_system::close_file_in_all_LRU ( )

Tries to close a file in all the LRU lists.

The caller must hold the mutex.

Returns
true if success, false if should retry later

◆ count_undo_deleted()

size_t Fil_system::count_undo_deleted ( space_id_t  undo_num)
inline

Count how many truncated undo space IDs are still tracked in the buffer pool and the file_system cache.

Parameters
[in]undo_numundo tablespace number.
Returns
number of undo tablespaces that are still in memory.

◆ encryption_reencrypt()

void Fil_system::encryption_reencrypt ( std::vector< space_id_t > &  sid_vector)

Reencrypt tablespace keys by current master key.

◆ encryption_rotate()

size_t Fil_system::encryption_rotate ( )

Rotate the tablespace keys by new master key.

Returns
the number of tablespaces that failed to rotate.

◆ erase_path()

bool Fil_system::erase_path ( space_id_t  space_id)
inline

Erase a tablespace ID and its mapping from the scanned files.

Parameters
[in]space_idTablespace ID to erase
Returns
true if successful

◆ flush_file_spaces()

void Fil_system::flush_file_spaces ( )

Flush to disk the writes in file spaces possibly cached by the OS (note: spaces of type FIL_TYPE_TEMPORARY are skipped)

◆ free_scanned_files()

void Fil_system::free_scanned_files ( )
inline

Free the data structures required for recovery.

◆ get_dirs()

std::string Fil_system::get_dirs ( ) const
inline

Get the list of directories that InnoDB knows about.

Returns
the list of directories 'dir1;dir2;....;dirN'

◆ get_file_by_space_id()

bool Fil_system::get_file_by_space_id ( space_id_t  space_id,
std::string &  name 
)
inline

Fetch the file name opened for a space_id from the file map.

Parameters
[in]space_idtablespace ID
[out]namethe scanned filename
Returns
true if the space_id is found. The name is set to an empty string if the space_id is not found.

◆ get_file_by_space_num()

bool Fil_system::get_file_by_space_num ( space_id_t  space_num,
space_id_t space_id,
std::string &  name 
)
inline

Fetch the file name opened for an undo space number.

Parameters
[in]space_numundo tablespace number
[out]space_idtablespace ID
[out]namethe scanned filename
Returns
true if the space_id is found. The name is set to an empty string if the space_id is not found.

◆ get_limit_for_non_lru_files()

size_t Fil_system::get_limit_for_non_lru_files ( size_t  open_files_limit)
static

Returns maximum number of allowed non-LRU files opened for a specified open files limit.

◆ get_max_space_id()

space_id_t Fil_system::get_max_space_id ( ) const
inline
Returns
the maximum assigned ID so far

◆ get_minimum_limit_for_open_files()

size_t Fil_system::get_minimum_limit_for_open_files ( size_t  n_files_not_belonging_in_lru) const

Returns minimum open files limit to be set to allow the specified number of non-LRU files opened.

This is inverse function for the get_limit_for_non_lru_files.

◆ get_open_files_limit()

size_t Fil_system::get_open_files_limit ( ) const
inline

Returns maximum number of allowed opened files.

◆ get_root()

const std::string & Fil_system::get_root ( const std::string &  path) const

Get the top level directory where this filename was found.

Parameters
[in]pathPath to look for.
Returns
the top level directory under which this file was found.

◆ get_scanned_filename_by_space_id()

Tablespace_dirs::Result Fil_system::get_scanned_filename_by_space_id ( space_id_t  space_id)
inline

Fetch the file names opened for a space_id during recovery.

Parameters
[in]space_idTablespace ID to lookup
Returns
pair of top level directory scanned and names that map to space_id or nullptr if not found.

◆ get_scanned_filename_by_space_num()

Tablespace_dirs::Result Fil_system::get_scanned_filename_by_space_num ( space_id_t  space_num,
space_id_t space_id 
)
inline

Fetch the file name opened for an undo space number.

Parameters
[in]space_numundo tablespace numb er to lookup
[out]space_idTablespace ID found
Returns
pair of top level directory scanned and name that maps to the space_num or nullptr if not found.

◆ get_space_by_name()

fil_space_t * Fil_system::get_space_by_name ( const char *  name)
inline

Fetch the fil_space_t instance that maps to the name.

Parameters
[in]nameTablespace name to lookup
Returns
tablespace instance or nullptr if not found.

◆ get_tablespace_id()

space_id_t Fil_system::get_tablespace_id ( const std::string &  filename)
static

Get the tablespace ID from an .ibd and/or an undo tablespace.

If the ID is 0 on the first page then try finding the ID with Datafile::find_space_id().

Parameters
[in]filenameFile name to check
Returns
s_invalid_space_id if not found, otherwise the space ID

◆ ibd_open_for_recovery()

fil_load_status Fil_system::ibd_open_for_recovery ( space_id_t  space_id,
const std::string &  path,
fil_space_t *&  space 
)
private

Open an ibd tablespace and add it to the InnoDB data structures.

This is similar to fil_ibd_open() except that it is used while processing the redo log, so the data dictionary is not available and very little validation is done. The tablespace name is extracted from the dbname/tablename.ibd portion of the filename, which assumes that the file is a file-per-table tablespace. Any name will do for now. General tablespace names will be read from the dictionary after it has been recovered. The tablespace flags are read at this time from the first page of the file in validate_for_recovery().

Parameters
[in]space_idtablespace ID
[in]pathpath/to/databasename/tablename.ibd
[out]spacethe tablespace, or nullptr on error
Returns
status of the operation

◆ is_deleted()

bool Fil_system::is_deleted ( space_id_t  space_id)
inlinenoexcept

Check if a particular undo space_id for a page in the buffer pool has been deleted recently.

Its space_id will be found in Fil_shard::m_deleted_spaces until Fil:shard::checkpoint removes the fil_space_t from Fil_system.

Parameters
[in]space_idTablespace ID to check.
Returns
true if this space_id is in the list of recently deleted spaces.

◆ is_greater_than_max_id()

bool Fil_system::is_greater_than_max_id ( space_id_t  space_id) const
inline

Check a space ID against the maximum known tablespace ID.

Parameters
[in]space_idTablespace ID to check
Returns
true if it is > than maximum known tablespace ID.

◆ iterate()

dberr_t Fil_system::iterate ( Fil_iterator::Function f)

Iterate through all persistent tablespace files (FIL_TYPE_TABLESPACE) returning the nodes via callback function cbk.

Parameters
[in]fCallback
Returns
any error returned by the callback function.

◆ lookup_for_recovery()

bool Fil_system::lookup_for_recovery ( space_id_t  space_id)

Lookup the tablespace ID.

Lookup the tablespace ID for recovery and DDL log apply.

Parameters
[in]space_idTablespace ID to lookup
Returns
true if the space ID is known.

◆ moved()

void Fil_system::moved ( dd::Object_id  object_id,
space_id_t  space_id,
const char *  space_name,
const std::string &  old_path,
const std::string &  new_path,
bool  before_90000 
)
inline

Note that a file has been relocated.

Parameters
[in]object_idServer DD tablespace ID
[in]space_idInnoDB tablespace ID
[in]space_nameTablespace name
[in]old_pathPath to the old location
[in]new_pathPath scanned from disk
[in]before_90000The move has happened before 90000 and didn't update the DD_TABLE_DATA_DIRECTORY flag.

◆ mutex_acquire_all()

void Fil_system::mutex_acquire_all ( ) const
inline

Acquire all the mutexes.

◆ mutex_owned_all()

bool Fil_system::mutex_owned_all ( ) const
inline

Check if all mutexes are owned.

Returns
true if all owned.

◆ mutex_release_all()

void Fil_system::mutex_release_all ( ) const
inline

Release all the mutexes.

◆ open_all_system_tablespaces()

void Fil_system::open_all_system_tablespaces ( )

Opens all system tablespace data files in all shards.

◆ open_for_recovery()

dberr_t Fil_system::open_for_recovery ( space_id_t  space_id)

Open a tablespace that has a redo log record to apply.

Parameters
[in]space_idTablespace ID
Returns
DB_SUCCESS if the open was successful

◆ operator=()

Fil_system & Fil_system::operator= ( const Fil_system )
privatedelete

◆ prepare_open_for_business()

dberr_t Fil_system::prepare_open_for_business ( bool  read_only_mode)

Update the DD if any files were moved to a new location.

Free the Tablespace_files instance.

Parameters
[in]read_only_modetrue if InnoDB is started in read only mode.
Returns
DB_SUCCESS if all OK

◆ purge()

void Fil_system::purge ( )
inline

Clean up the shards.

◆ rename_partition_files()

void Fil_system::rename_partition_files ( bool  revert)

Rename partition files during upgrade.

Parameters
[in]revertif true, revert to old names

◆ rename_tablespace_name()

dberr_t Fil_system::rename_tablespace_name ( space_id_t  space_id,
const char *  old_name,
const char *  new_name 
)

Rename a tablespace.

Use the space_id to find the shard.

Parameters
[in]space_idtablespace ID
[in]old_nameold tablespace name
[in]new_namenew tablespace name
Returns
DB_SUCCESS on success

◆ scan()

dberr_t Fil_system::scan ( )
inline

Scan the directories to build the tablespace ID to file name mapping table.

◆ set_maximum_space_id()

void Fil_system::set_maximum_space_id ( const fil_space_t space)
inline

Update the maximum known tablespace ID.

Parameters
[in]spaceTablespace instance

◆ set_open_files_limit()

bool Fil_system::set_open_files_limit ( size_t &  new_max_open_files)

Changes the maximum opened files limit.

Parameters
[in,out]new_max_open_filesNew value for the open files limit. If the limit cannot be changed, the value is changed to a minimum value recommended. If there are any concurrent calls to set_open_files_limit in progress, setting the limit will fail and the new_max_open_files will be set to 0.
Returns
true if the new limit was set.

◆ set_scan_dir()

void Fil_system::set_scan_dir ( const std::string &  directory,
bool  is_undo_dir 
)
inline

Normalize and save a directory to scan for IBD and IBU datafiles before recovery.

Parameters
[in]directoryDirectory to scan
[in]is_undo_dirtrue for an undo directory

◆ set_scan_dirs()

void Fil_system::set_scan_dirs ( const std::string &  directories)
inline

Normalize and save a list of directories to scan for IBD and IBU datafiles before recovery.

Parameters
[in]directoriesDirectories to scan

◆ shard_by_id()

Fil_shard * Fil_system::shard_by_id ( space_id_t  space_id) const
inline

Fil_shard by space ID.

Parameters
[in]space_idTablespace ID
Returns
reference to the shard

◆ space_acquire()

fil_space_t * Fil_system::space_acquire ( space_id_t  space_id,
bool  silent 
)

Acquire a tablespace when it could be dropped concurrently.

Used by background threads that do not necessarily hold proper locks for concurrency control.

Parameters
[in]space_idTablespace ID
[in]silentWhether to silently ignore missing tablespaces
Returns
the tablespace, or nullptr if missing or being deleted

◆ space_belongs_in_LRU()

bool Fil_system::space_belongs_in_LRU ( const fil_space_t space)
static

Determines if a file belongs to the least-recently-used list.

Parameters
[in]spaceTablespace to check
Returns
true if the file belongs to fil_system->m_LRU mutex.

◆ space_detach()

void Fil_system::space_detach ( fil_space_t space)

Detach a space object from the tablespace memory cache.

Closes the tablespace files but does not delete them. There must not be any pending I/O's or flushes on the files.

Parameters
[in,out]spacetablespace

◆ update_maximum_space_id()

void Fil_system::update_maximum_space_id ( space_id_t  space_id)
inline

Update the maximum known space ID if it's smaller than max_id.

Parameters
[in]space_idValue to set if it's greater

◆ validate()

bool Fil_system::validate ( ) const

Checks the consistency of the tablespace cache.

Returns
true if ok

◆ wait_while_ios_in_progress()

void Fil_system::wait_while_ios_in_progress ( ) const
inline

Allows other threads to advance work while we wait for I/Os to complete.

Friends And Related Function Documentation

◆ Fil_shard

friend class Fil_shard
friend

Member Data Documentation

◆ m_ACCESSING_NONEXISTINC_SPACE_throttler

ib::Throttler Fil_system::m_ACCESSING_NONEXISTINC_SPACE_throttler {}
private

Throttles messages about accessing space that was already removed, the warning ACCESSING_NONEXISTINC_SPACE.

◆ m_dirs

Tablespace_dirs Fil_system::m_dirs
private

Tablespace directories scanned at startup.

◆ m_MANY_NON_LRU_FILES_OPENED_throttler

ib::Throttler Fil_system::m_MANY_NON_LRU_FILES_OPENED_throttler {}
private

Throttles messages about high files not belonging in LRU count, the warning ER_IB_WARN_MANY_NON_LRU_FILES_OPENED.

◆ m_max_assigned_id

space_id_t Fil_system::m_max_assigned_id
private

Maximum space id in the existing tables, or assigned during the time mysqld has been up; at an InnoDB startup we scan the data dictionary and set here the maximum of the space id's of the tables there.

◆ m_moved

dd_fil::Tablespaces Fil_system::m_moved
private

List of tablespaces that have been relocated.

We need to update the DD when it is safe to do so.

◆ m_n_files_not_belonging_in_lru

std::atomic_size_t Fil_system::m_n_files_not_belonging_in_lru {}
private

Current number of files that are not belonging in LRU.

This includes redo and temporary tablespaces, but not files that were temporarily removed from the LRU for I/O.

◆ m_next_shard_to_close_from_LRU

std::atomic_size_t Fil_system::m_next_shard_to_close_from_LRU {}
private

Next index (modulo number of shards) to try to close a file from the LRU list to distribute closures evenly between the shards.

◆ m_old_paths

std::vector<std::string> Fil_system::m_old_paths
private

Old file paths during 5.7 upgrade.

◆ m_open_files_limit

fil::detail::Open_files_limit Fil_system::m_open_files_limit
private

◆ m_shards

Fil_shards Fil_system::m_shards
private

Fil_shards managed.

◆ m_space_id_reuse_warned

bool Fil_system::m_space_id_reuse_warned
private

true if fil_space_create() has issued a warning about potential space_id reuse

◆ m_TRYING_TO_OPEN_FILE_FOR_LONG_TIME_throttler

ib::Throttler Fil_system::m_TRYING_TO_OPEN_FILE_FOR_LONG_TIME_throttler {}
private

Throttles messages about long waiting for opened files limit, the warning ER_IB_MSG_TRYING_TO_OPEN_FILE_FOR_LONG_TIME.


The documentation for this class was generated from the following file: