Class used to recover binary / relay log file. More...

#include <log_sanitizer.h>

Inheritance diagram for binlog::Log_sanitizer:

Public Member Functions
	Log_sanitizer ()
	Ctor. More...

virtual	~Log_sanitizer ()=default
	Dtor. More...

my_off_t	get_valid_pos () const
	Retrieves the position of the last binlog/relay log event that ended a transaction or position after the RLE/FDE/SE that comes from the source. More...

std::pair< my_off_t, bool >	get_valid_source_pos () const
	Retrieves the last valid source position of an event in read from the binary log / relay log file, which may be: More...

std::pair< std::string, bool >	get_valid_source_file () const
	Retrieves the updated name of the binlog source file. More...

bool	is_log_malformed () const
	Retrieves whether or not the log was correctly processed in full. More...

std::string const &	get_failure_message () const
	Retrieves the textual representation of the encontered failure, if any. More...

std::string	get_valid_file () const

bool	is_log_truncation_needed () const
	Checks whether a valid sanitized log file needs truncation of the last, partially written transaction or events that cannot be safely read. More...

bool	is_fatal_error () const
	Checks whether the fatal error occurred during log sanitization (OOM / decompression error which we cannot handle) More...

Protected Member Functions
virtual PSI_memory_key &	get_memory_key () const =0
	Function used to obtain memory key for derived classes. More...

template<class Type_reader >
void	process_logs (Type_reader &reader)
	This function goes through the opened file and searches for a valid position in a binary log file. More...

template<class Type_reader >
void	process_logs (Type_reader &reader, const std::list< std::string > &list_of_files, MYSQL_BIN_LOG &log)
	This function goes iterates over the relay log files in the 'list_of_files' container, starting from the most recent one. More...

template<class Type_reader >
void	process_logs (Type_reader &reader, MYSQL_BIN_LOG &log)
	This function will obtain the list of relay log files using the object of MYSQL_BIN_LOG class and iterate over them to find the last valid position within a relay log file. More...

template<class Type_reader >
bool	process_one_log (Type_reader &reader, const std::string &filename)
	Reads and validates one log file. More...

void	process_query_event (Query_log_event const &ev)
	Invoked when a `Query_log_event` is read from the binary log file reader. More...

void	process_xid_event (Xid_log_event const &ev)
	Invoked when a `Xid_log_event` is read from the binary log file reader. More...

void	process_xa_prepare_event (XA_prepare_log_event const &ev)
	Invoked when a `XA_prepare_log_event` is read from the binary log file reader. More...

void	process_start ()
	Invoked when a `BEGIN` or an ‘XA START’ is found in a `Query_log_event`. More...

void	process_commit ()
	Invoked when a `COMMIT` is found in a `Query_log_event`. More...

void	process_rollback ()
	Invoked when a `ROLLBACK` is found in a `Query_log_event`. More...

void	process_atomic_ddl (Query_log_event const &ev)
	Invoked when a DDL is found in a `Query_log_event`. More...

void	process_xa_commit (std::string const &query)
	Invoked when an `XA COMMIT` is found in a `Query_log_event`. More...

void	process_xa_rollback (std::string const &query)
	Invoked when an `XA ROLLBACK` is found in a `Query_log_event`. More...

void	add_external_xid (std::string const &query, enum_ha_recover_xa_state state)
	Parses the provided string for an XID and adds it to the externally coordinated transactions map, along side the provided state. More...

Protected Attributes
bool	m_validation_started {true}
	Indicates whether validation has started. More...

my_off_t	m_valid_pos {0}
	Position of the last binlog/relay log event that ended a transaction. More...

my_off_t	m_valid_source_pos {0}
	Position of the last binlog event that ended a transaction (source position which corresponds to m_valid_pos) More...

std::string	m_valid_source_file {""}
	Currently processed binlog file set in case source rotation event is encountered. More...

std::string	m_valid_file {""}
	Last log file containing finished transaction. More...

bool	m_in_transaction {false}
	Whether or not the event being processed is within a transaction. More...

bool	m_is_malformed {false}
	Whether or not the binary log is malformed/corrupted or error occurred. More...

bool	m_fatal_error {false}
	Whether or not the binary log has a fatal error. More...

std::string	m_failure_message {""}
	Textual representation of the encountered failure. More...

MEM_ROOT	m_mem_root
	Memory pool to use for the XID lists. More...

Mem_root_allocator< my_xid >	m_set_alloc
	Memory pool allocator to use with the normal transaction list. More...

Mem_root_allocator< std::pair< const XID, XID_STATE::xa_states > >	m_map_alloc
	Memory pool allocator to use with the XA transaction list. More...

Xid_commit_list	m_internal_xids
	List of normal transactions fully written to the binary log. More...

Xa_state_list::list	m_external_xids
	List of XA transactions and states that appear in the binary log. More...

bool	m_is_log_truncation_needed {false}
	Information on whether log needs to be truncated, i.e. More...

bool	m_has_valid_pos {false}
	Indicator on whether a valid position has been found in the log file. More...

bool	m_has_valid_source_pos {false}
	Indicator on whether a valid source position has been found in the log file. More...

my_off_t	m_last_file_size {0}
	Last opened file size. More...

Detailed Description

Class used to recover binary / relay log file.

This base class is responsible for finding the last valid position of a relay log / binary log file, meaning, the position of the last finished event which occurs outside of transaction boundary. Validation starts when first reliable position has been found, i.e.:

source rotation event
source FDE
source STOP event
first finished transaction:
- Query log event with: COMMIT / ROLLBACK / XA COMMIT / XA ROLLBACK / atomic DDL
- XID Log event Validation ends at the end of the binlog file / relay log file or in case further reading is not possible. Binary log recovery: Binary log file always start with an FDE which is the first and valid position within a file. Binary log files are never removed by a log sanitizer. Relay log recovery: If no valid position has been found in any of the relay log files, Log sanitizer will keep all of the relay log files. In case a valid position has been found in any of the first relay log files, relay log files that do not contain a valid position outside of a transaction boundary, will be removed.

Constructor & Destructor Documentation

◆ Log_sanitizer()

binlog::Log_sanitizer::Log_sanitizer ( )

Ctor.

◆ ~Log_sanitizer()

virtual binlog::Log_sanitizer::~Log_sanitizer ( )

virtualdefault

Dtor.

Member Function Documentation

◆ add_external_xid()

void binlog::Log_sanitizer::add_external_xid	(	std::string const &	query,
		enum_ha_recover_xa_state	state
	)

protected

Parses the provided string for an XID and adds it to the externally coordinated transactions map, along side the provided state.

Parameters

query	The query to search and retrieve the XID from
state	The state to add to the map, along side the XID

◆ get_failure_message()

std::string const & binlog::Log_sanitizer::get_failure_message ( ) const

Retrieves the textual representation of the encontered failure, if any.

Returns: the string containing the textual representation of the failure, an empty string otherwise.

◆ get_memory_key()

virtual PSI_memory_key & binlog::Log_sanitizer::get_memory_key ( ) const

protectedpure virtual

Function used to obtain memory key for derived classes.

Returns: Reference to a memory key

Implemented in binlog::Binlog_recovery, and rpl::Relay_log_sanitizer.

◆ get_valid_file()

std::string binlog::Log_sanitizer::get_valid_file ( ) const

inline

◆ get_valid_pos()

my_off_t binlog::Log_sanitizer::get_valid_pos ( ) const

Retrieves the position of the last binlog/relay log event that ended a transaction or position after the RLE/FDE/SE that comes from the source.

Returns: The position of the last binlog event that ended a transaction

◆ get_valid_source_file()

std::pair< std::string, bool > binlog::Log_sanitizer::get_valid_source_file ( ) const

Retrieves the updated name of the binlog source file.

Returns: Updated source file or empty string; indicator equal to true in case filename is valid

◆ get_valid_source_pos()

std::pair< my_off_t, bool > binlog::Log_sanitizer::get_valid_source_pos ( ) const

Retrieves the last valid source position of an event in read from the binary log / relay log file, which may be:

source position of the event ending a transaction
source position written in the source RLE
Returns
The position of the last binlog event that ended a transaction and indicator whether this position is valid

◆ is_fatal_error()

bool binlog::Log_sanitizer::is_fatal_error ( ) const

Checks whether the fatal error occurred during log sanitization (OOM / decompression error which we cannot handle)

Returns: true in case fatal error occurred, false otherwise

◆ is_log_malformed()

bool binlog::Log_sanitizer::is_log_malformed ( ) const

Retrieves whether or not the log was correctly processed in full.

Returns: true if the log processing ended with errors, false otherwise.

◆ is_log_truncation_needed()

bool binlog::Log_sanitizer::is_log_truncation_needed ( ) const

Checks whether a valid sanitized log file needs truncation of the last, partially written transaction or events that cannot be safely read.

Returns: true in case log file needs to be truncated, false otherwise

◆ process_atomic_ddl()

void binlog::Log_sanitizer::process_atomic_ddl ( Query_log_event const & ev )

protected

Invoked when a DDL is found in a Query_log_event.

Actions taken to process the statement:

If m_in_transaction flag is set to true, m_is_malformed is set to true, indicating that the binary log is malformed.
The XID of the transaction is extracted and added to the list of internally coordinated transactions m_internal_xids.
If the XID already exists in the list, m_is_malformed is set to true, indicating that the binary log is malformed.
Parameters

ev The Query_log_event to process

◆ process_commit()

void binlog::Log_sanitizer::process_commit ( )

protected

Invoked when a COMMIT is found in a Query_log_event.

Actions taken to process the statement:

If m_in_transaction flag is set to false, m_is_malformed is set to true, indicating that the binary log is malformed.
The m_in_transaction flag is set to false, indicating that the event starts a transaction.

◆ process_logs() [1/3]

template<class Type_reader >

void binlog::Log_sanitizer::process_logs ( Type_reader & reader )

protected

This function goes through the opened file and searches for a valid position in a binary log file.

It also gathers information about XA transactions which will be used during the binary log recovery

Parameters

reader Log reader, must be opened

◆ process_logs() [2/3]

template<class Type_reader >

void binlog::Log_sanitizer::process_logs	(	Type_reader &	reader,
		const std::list< std::string > &	list_of_files,
		MYSQL_BIN_LOG &	log
	)

protected

This function goes iterates over the relay log files in the 'list_of_files' container, starting from the most recent one.

It gathers information about XA transactions and performs a small validation of the log files. Validation starts in case a first reliable position has been found (FDE/RLE/SE from the source or the end of a transaction), and proceeds till the end of file or until a read error has occurred. In case a valid position has been found within a file, relay log files that were created after this file will be removed. In case no valid position has been found within a file, sanitizer will iterate over events in the previous (older) relay log file. In case no valid position has been found in any of the files listed in the 'list_of_files' container, relay log files won't be removed. It may happen e.g. in case we cannot decrypt events.

Parameters

reader	Relay log file reader object
list_of_files	The list of relay logs we know, obtained from the relay log index
log	MYSQL_BIN_LOG object used to manipulate relay log files

◆ process_logs() [3/3]

template<class Type_reader >

void binlog::Log_sanitizer::process_logs	(	Type_reader &	reader,
		MYSQL_BIN_LOG &	log
	)

protected

This function will obtain the list of relay log files using the object of MYSQL_BIN_LOG class and iterate over them to find the last valid position within a relay log file.

It will remove relay log files that contain only parts of the last, partially written transaction

Parameters

reader	Relay log file reader object
log	MYSQL_BIN_LOG object used to manipulate relay log files

◆ process_one_log()

template<class Type_reader >

bool binlog::Log_sanitizer::process_one_log	(	Type_reader &	reader,
		const std::string &	filename
	)

protected

Reads and validates one log file.

Parameters

[in]	filename	Name of the log file to process
[in]	reader	Reference to reader able to read processed log file

Returns: true if processed log contains a valid log position outside of transaction boundaries

◆ process_query_event()

void binlog::Log_sanitizer::process_query_event ( Query_log_event const & ev )

protected

Invoked when a Query_log_event is read from the binary log file reader.

The underlying query string is inspected to determine if the SQL command starts or ends a transaction. The following commands are searched for:

BEGIN
COMMIT
ROLLBACK
DDL
XA START
XA COMMIT
XA ROLLBACK Check below for the description of the action that is taken for each.
Parameters

ev The Query_log_event to process

◆ process_rollback()

void binlog::Log_sanitizer::process_rollback ( )

protected

Invoked when a ROLLBACK is found in a Query_log_event.

Actions taken to process the statement:

If m_in_transaction flag is set to false, m_is_malformed is set to true, indicating that the binary log is malformed.
The m_in_transaction flag is set to false, indicating that the event starts a transaction.

◆ process_start()

void binlog::Log_sanitizer::process_start ( )

protected

Invoked when a BEGIN or an ‘XA START’ is found in a Query_log_event.

Actions taken to process the statement:

If m_in_transaction flag is set to true, m_is_malformed is set to true, indicating that the binary log is malformed.
The m_in_transaction flag is set to true, indicating that the event starts a transaction.

◆ process_xa_commit()

void binlog::Log_sanitizer::process_xa_commit ( std::string const & query )

protected

Invoked when an XA COMMIT is found in a Query_log_event.

Actions taken to process the statement:

If m_in_transaction flag is set to true, m_is_malformed is set to true, indicating that the binary log is malformed.
The m_in_transaction flag is set to false, indicating that the event ends a transaction.
The XID of the transaction is extracted and added to the list of externally coordinated transactions m_external_xids, alongside the state COMMITTED.
If the XID already exists in the list associated with a state other than PREPARED, m_is_malformed is set to true, indicating that the binary log is malformed.
Parameters

query The query string to process

◆ process_xa_prepare_event()

void binlog::Log_sanitizer::process_xa_prepare_event ( XA_prepare_log_event const & ev )

protected

Invoked when a XA_prepare_log_event is read from the binary log file reader.

Actions taken to process the event:

If m_in_transaction flag is set to false, m_is_malformed is set to true, indicating that the binary log is malformed.
The m_in_transaction flag is set to false, indicating that the event ends a transaction.
The XID of the transaction is extracted and added to the list of externally coordinated transactions m_external_xids, along side the state COMMITTED if the event represents an XA COMMIT ONE_PHASE or PREPARED if not.
If the XID already exists in the list associated with a state other than COMMITTED or ROLLEDBACK, m_is_malformed is set to true, indicating that the binary log is malformed.
Parameters

ev The XA_prepare_log_event to process

◆ process_xa_rollback()

void binlog::Log_sanitizer::process_xa_rollback ( std::string const & query )

protected

Invoked when an XA ROLLBACK is found in a Query_log_event.

Actions taken to process the statement:

If m_in_transaction flag is set to true, m_is_malformed is set to true, indicating that the binary log is malformed.
The m_in_transaction flag is set to false, indicating that the event ends a transaction.
The XID of the transaction is extracted and added to the list of externally coordinated transactions m_external_xids, along side the state ROLLEDBACK.
If the XID already exists in the list associated with a state other than PREPARED, m_is_malformed is set to true, indicating that the binary log is malformed.
Parameters

query The query string to process

◆ process_xid_event()

void binlog::Log_sanitizer::process_xid_event ( Xid_log_event const & ev )

protected

Invoked when a Xid_log_event is read from the binary log file reader.

Actions taken to process the event:

If m_in_transaction flag is set to false, m_is_malformed is set to true, indicating that the binary log is malformed.
The m_in_transaction flag is set to false, indicating that the event ends a transaction.
The XID of the transaction is extracted and added to the list of internally coordinated transactions m_internal_xids.
If the XID already exists in the list, m_is_malformed is set to true, indicating that the binary log is malformed.
Parameters

ev The Xid_log_event to process

Member Data Documentation

◆ m_external_xids

Xa_state_list::list binlog::Log_sanitizer::m_external_xids

protected

List of XA transactions and states that appear in the binary log.

◆ m_failure_message

std::string binlog::Log_sanitizer::m_failure_message {""}

protected

Textual representation of the encountered failure.

◆ m_fatal_error

bool binlog::Log_sanitizer::m_fatal_error {false}

protected

Whether or not the binary log has a fatal error.

◆ m_has_valid_pos

bool binlog::Log_sanitizer::m_has_valid_pos {false}

protected

Indicator on whether a valid position has been found in the log file.

◆ m_has_valid_source_pos

bool binlog::Log_sanitizer::m_has_valid_source_pos {false}

protected

Indicator on whether a valid source position has been found in the log file.

◆ m_in_transaction

bool binlog::Log_sanitizer::m_in_transaction {false}

protected

Whether or not the event being processed is within a transaction.

◆ m_internal_xids

Xid_commit_list binlog::Log_sanitizer::m_internal_xids

protected

List of normal transactions fully written to the binary log.

◆ m_is_log_truncation_needed

bool binlog::Log_sanitizer::m_is_log_truncation_needed {false}

protected

Information on whether log needs to be truncated, i.e.

log is not ending at transaction boundary or we cannot read it till the end

◆ m_is_malformed

bool binlog::Log_sanitizer::m_is_malformed {false}

protected

Whether or not the binary log is malformed/corrupted or error occurred.

◆ m_last_file_size

my_off_t binlog::Log_sanitizer::m_last_file_size {0}

protected

Last opened file size.

◆ m_map_alloc

Mem_root_allocator<std::pair<const XID, XID_STATE::xa_states> > binlog::Log_sanitizer::m_map_alloc

protected

Memory pool allocator to use with the XA transaction list.

◆ m_mem_root

MEM_ROOT binlog::Log_sanitizer::m_mem_root

protected

Memory pool to use for the XID lists.

◆ m_set_alloc

Mem_root_allocator<my_xid> binlog::Log_sanitizer::m_set_alloc

protected

Memory pool allocator to use with the normal transaction list.

◆ m_valid_file

std::string binlog::Log_sanitizer::m_valid_file {""}

protected

Last log file containing finished transaction.

◆ m_valid_pos

my_off_t binlog::Log_sanitizer::m_valid_pos {0}

protected

Position of the last binlog/relay log event that ended a transaction.

◆ m_valid_source_file

std::string binlog::Log_sanitizer::m_valid_source_file {""}

protected

Currently processed binlog file set in case source rotation event is encountered.

◆ m_valid_source_pos

my_off_t binlog::Log_sanitizer::m_valid_source_pos {0}

protected

Position of the last binlog event that ended a transaction (source position which corresponds to m_valid_pos)

◆ m_validation_started

bool binlog::Log_sanitizer::m_validation_started {true}

protected

Indicates whether validation has started.

In case of relay log sanitization, we start validation when we are sure that we are at transaction boundary and we are able to recover source position, meaning, when we detect:

first encountered Rotation Event, that comes from the source
end of a transaction (Xid event, QLE containing COMMIT/ROLLBACK/XA COMMIT/XA ROLLBACK)
an atomic DDL transaction Since binary logs always start at transaction boundary, when doing a binary log recovery, we start validation right away. By default, we are assuming that we are in the binary log recovery procedure

The documentation for this class was generated from the following files:

sql/binlog/log_sanitizer.h
sql/binlog/log_sanitizer.cc
sql/binlog/log_sanitizer_impl.hpp

Public Member Functions

Protected Member Functions

Protected Attributes

Detailed Description

Constructor & Destructor Documentation

◆ Log_sanitizer()

◆ ~Log_sanitizer()

Member Function Documentation

◆ add_external_xid()

◆ get_failure_message()

◆ get_memory_key()

◆ get_valid_file()

◆ get_valid_pos()

◆ get_valid_source_file()

◆ get_valid_source_pos()

◆ is_fatal_error()

◆ is_log_malformed()

◆ is_log_truncation_needed()

◆ process_atomic_ddl()

◆ process_commit()

◆ process_logs() [1/3]

◆ process_logs() [2/3]

◆ process_logs() [3/3]

◆ process_one_log()

◆ process_query_event()

◆ process_rollback()

◆ process_start()

◆ process_xa_commit()

◆ process_xa_prepare_event()

◆ process_xa_rollback()

◆ process_xid_event()

Member Data Documentation

◆ m_external_xids

◆ m_failure_message

◆ m_fatal_error

◆ m_has_valid_pos

◆ m_has_valid_source_pos

◆ m_in_transaction

◆ m_internal_xids

◆ m_is_log_truncation_needed

◆ m_is_malformed

◆ m_last_file_size

◆ m_map_alloc

◆ m_mem_root

◆ m_set_alloc

◆ m_valid_file

◆ m_valid_pos

◆ m_valid_source_file

◆ m_valid_source_pos

◆ m_validation_started