MaterializeIterator Class Reference

Handles materialization; the first call to Init() will scan the given iterator to the end, store the results in a temporary table (optionally with deduplication), and then Read() will allow you to read that table repeatedly without the cost of executing the given subquery many times (unless you ask for rematerialization). More...

#include <composite_iterators.h>

Inheritance diagram for MaterializeIterator:

Classes
struct	Invalidator
struct	QueryBlock

Public Types
using	keeps_own_timing = void

Public Member Functions
	MaterializeIterator (THD *thd, Mem_root_array< QueryBlock > query_blocks_to_materialize, TABLE *table, unique_ptr_destroy_only< RowIterator > table_iterator, Common_table_expr *cte, Query_expression *unit, JOIN *join, int ref_slice, bool rematerialize, ha_rows limit_rows, bool reject_multiple_rows)
bool	Init () override
	Initialize or reinitialize the iterator. More...
int	Read () override
	Read a single row. More...
void	SetNullRowFlag (bool is_null_row) override
	Mark the current row buffer as containing a NULL row or not, so that if you read from it and the flag is true, you'll get only NULLs no matter what is actually in the buffer (typically some old leftover row). More...
void	StartPSIBatchMode () override
	Start performance schema batch mode, if supported (otherwise ignored). More...
void	EndPSIBatchModeIfStarted () override
	Ends performance schema batch mode, if started. More...
void	UnlockRow () override
void	AddInvalidator (const CacheInvalidatorIterator *invalidator)
	Add a cache invalidator that must be checked on every Init(). More...
uint64_t	num_init_calls () const
uint64_t	num_rows () const
Public Member Functions inherited from TableRowIterator
	TableRowIterator (THD *thd, TABLE *table)
void	UnlockRow () override
	The default implementation of unlock-row method of RowIterator, used in all access methods except EQRefIterator. More...
void	SetNullRowFlag (bool is_null_row) override
	Mark the current row buffer as containing a NULL row or not, so that if you read from it and the flag is true, you'll get only NULLs no matter what is actually in the buffer (typically some old leftover row). More...
void	StartPSIBatchMode () override
	Start performance schema batch mode, if supported (otherwise ignored). More...
void	EndPSIBatchModeIfStarted () override
	Ends performance schema batch mode, if started. More...
Public Member Functions inherited from RowIterator
	RowIterator (THD *thd)
virtual	~RowIterator ()=default
	RowIterator (const RowIterator &)=delete
	RowIterator (RowIterator &&)=default
virtual std::string	TimingString () const
virtual RowIterator *	real_iterator ()
	If this iterator is wrapping a different iterator (e.g. More...
virtual const RowIterator *	real_iterator () const

Private Member Functions
bool	doing_hash_deduplication () const
	Whether we are deduplicating using a hash field on the temporary table. More...
bool	doing_deduplication () const
	Whether we are deduplicating, whether through a hash field or a regular unique index. More...
bool	MaterializeRecursive ()
	Recursive materialization happens much like regular materialization, but some steps are repeated multiple times. More...
bool	MaterializeQueryBlock (const QueryBlock &query_block, ha_rows *stored_rows)

Private Attributes
Mem_root_array< QueryBlock >	m_query_blocks_to_materialize
unique_ptr_destroy_only< RowIterator >	m_table_iterator
Common_table_expr *	m_cte
	If we are materializing a CTE, points to it (otherwise nullptr). More...
Query_expression *	m_query_expression
	The query expression we are materializing. More...
JOIN *const	m_join
	See constructor. More...
const int	m_ref_slice
	The slice to set when accessing temporary table; used if anything upstream (e.g. More...
const bool	m_rematerialize
	If true, we need to materialize anew for each Init() (because the contents of the table will depend on some outer non-constant value). More...
const bool	m_reject_multiple_rows
	See constructor. More...
const ha_rows	m_limit_rows
	See constructor. More...
Mem_root_array< Invalidator >	m_invalidators
uint64_t	m_num_materializations = 0
uint64_t	m_num_materialized_rows = 0

Additional Inherited Members
Protected Member Functions inherited from TableRowIterator
int	HandleError (int error)
void	PrintError (int error)
TABLE *	table () const
Protected Member Functions inherited from RowIterator
THD *	thd () const

Detailed Description

Handles materialization; the first call to Init() will scan the given iterator to the end, store the results in a temporary table (optionally with deduplication), and then Read() will allow you to read that table repeatedly without the cost of executing the given subquery many times (unless you ask for rematerialization).

When materializing, MaterializeIterator takes care of evaluating any items that need so, and storing the results in the fields of the outgoing table – which items is governed by the temporary table parameters.

Conceptually (although not performance-wise!), the MaterializeIterator is a no-op if you don't ask for deduplication, and in some cases (e.g. when scanning a table only once), we elide it. However, it's not necessarily straightforward to do so by just not inserting the iterator, as the optimizer will have set up everything (e.g., read sets, or what table upstream items will read from) assuming the materialization will happen, so the realistic option is setting up everything as if materialization would happen but not actually write to the table; see StreamingIterator for details.

MaterializeIterator conceptually materializes iterators, not JOINs or Query_expressions. However, there are many details that leak out (e.g., setting performance schema batch mode, slices, reusing CTEs, etc.), so we need to send them in anyway.

Member Typedef Documentation

keeps_own_timing

using MaterializeIterator::keeps_own_timing = void

Constructor & Destructor Documentation

MaterializeIterator()

MaterializeIterator::MaterializeIterator	(	THD *	thd,
		Mem_root_array< QueryBlock >	query_blocks_to_materialize,
		TABLE *	table,
		unique_ptr_destroy_only< RowIterator >	table_iterator,
		Common_table_expr *	cte,
		Query_expression *	unit,
		JOIN *	join,
		int	ref_slice,
		bool	rematerialize,
		ha_rows	limit_rows,
		bool	reject_multiple_rows
	)

Parameters

thd	Thread handler.
query_blocks_to_materialize	List of query blocks to materialize.
table	Handle to table to materialize into.
table_iterator	Iterator used for scanning the temporary table after materialization.
cte	If materializing a CTE, points to it (see m_cte), otherwise nullptr.
unit	The query expression we are materializing (see m_query_expression).
join	When materializing within the same JOIN (e.g., into a temporary table before sorting), as opposed to a derived table or a CTE, we may need to change the slice on the join before returning rows from the result table. If so, join and ref_slice would need to be set, and query_blocks_to_materialize should contain only one member, with the same join.
ref_slice	See join. If we are materializing across JOINs, e.g. derived tables, ref_slice should be left at -1.
rematerialize	true if rematerializing on every Init() call (e.g., because we have a dependency on a value from outside the query block).
limit_rows	Used for when pushing LIMIT down to MaterializeIterator; this is more efficient than having a LimitOffsetIterator above the MaterializeIterator, since we can stop materializing when there are enough rows. (This is especially important for recursive CTEs.) Note that we cannot have a LimitOffsetIterator below the MaterializeIterator, as that would count wrong if we have deduplication, and would not work at all for recursive CTEs. Set to HA_POS_ERROR for no limit.
reject_multiple_rows	true if this is the top level iterator for a materialized derived table transformed from a scalar subquery which needs run-time cardinality check.

Member Function Documentation

AddInvalidator()

void MaterializeIterator::AddInvalidator

(

const CacheInvalidatorIterator *

invalidator

)

Add a cache invalidator that must be checked on every Init().

If its generation has increased since last materialize, we need to rematerialize even if m_rematerialize is false.

doing_deduplication()

bool MaterializeIterator::doing_deduplication ( ) const

private

Whether we are deduplicating, whether through a hash field or a regular unique index.

doing_hash_deduplication()

bool MaterializeIterator::doing_hash_deduplication ( ) const

inlineprivate

Whether we are deduplicating using a hash field on the temporary table.

(This condition mirrors check_unique_constraint().) If so, we compute a hash value for every row, look up all rows with the same hash and manually compare them to the row we are trying to insert.

Note that this is not the common way of deduplicating as we go. The common method is to have a regular index on the table over the right columns, and in that case, ha_write_row() will fail with an ignorable error, so that the row is ignored even though check_unique_constraint() is not called. However, B-tree indexes have limitations, in particular on length, that sometimes require us to do this instead. See create_tmp_table() for details.

EndPSIBatchModeIfStarted()

void MaterializeIterator::EndPSIBatchModeIfStarted ( )

overridevirtual

Ends performance schema batch mode, if started.

It's always safe to call this.

Iterators that have children (composite iterators) must forward the EndPSIBatchModeIfStarted() call to every iterator they could conceivably have called StartPSIBatchMode() on. This ensures that after such a call to on the root iterator, all handlers are out of batch mode.

Reimplemented from RowIterator.

Init()

bool MaterializeIterator::Init ( )

overridevirtual

Initialize or reinitialize the iterator.

You must always call Init() before trying a Read() (but Init() does not imply Read()).

You can call Init() multiple times; subsequent calls will rewind the iterator (or reposition it, depending on whether the iterator takes in e.g. a TABLE_REF) and allow you to read the records anew.

Implements RowIterator.

MaterializeQueryBlock()

bool MaterializeIterator::MaterializeQueryBlock ( const QueryBlock &  query_block, ha_rows *  stored_rows  )

private

MaterializeRecursive()

bool MaterializeIterator::MaterializeRecursive ( )

private

Recursive materialization happens much like regular materialization, but some steps are repeated multiple times.

Our general strategy is:

1. Materialize all non-recursive query blocks, once.
2. Materialize all recursive query blocks in turn.
3. Repeat #2 until no query block writes any more rows (ie., we have converged) – for UNION DISTINCT queries, rows removed by deduplication do not count. Each materialization sees only rows that were newly added since the previous iteration; see FollowTailIterator for more details on the implementation.

Note that the result table is written to while other iterators are still reading from it; again, see FollowTailIterator. This means that each run of #2 can potentially run many actual CTE iterations – possibly the entire query to completion if we have only one query block.

This is not how the SQL standard specifies recursive CTE execution (it assumes building up the new result set from scratch for each iteration, using the previous iteration's results), but it is equivalent, and more efficient for the class of queries we support, since we don't need to re-create the same rows over and over again.

num_init_calls()

uint64_t MaterializeIterator::num_init_calls ( ) const

inline

num_rows()

uint64_t MaterializeIterator::num_rows ( ) const

inline

Read()

int MaterializeIterator::Read ( )

overridevirtual

Read a single row.

The row data is not actually returned from the function; it is put in the table's (or tables', in case of a join) record buffer, ie., table->records[0].

Return values

0	OK
-1	End of records
1	Error

Implements RowIterator.

SetNullRowFlag()

void MaterializeIterator::SetNullRowFlag ( bool  is_null_row )

inlineoverridevirtual

Mark the current row buffer as containing a NULL row or not, so that if you read from it and the flag is true, you'll get only NULLs no matter what is actually in the buffer (typically some old leftover row).

This is used for outer joins, when an iterator hasn't produced any rows and we need to produce a NULL-complemented row. Init() or Read() won't necessarily reset this flag, so if you ever set is to true, make sure to also set it to false when needed.

Note that this can be called without Init() having been called first. For example, NestedLoopIterator can hit EOF immediately on the outer iterator, which means the inner iterator doesn't get an Init() call, but will still forward SetNullRowFlag to both inner and outer iterators.

TODO: We shouldn't need this. See the comments on AggregateIterator for a bit more discussion on abstracting out a row interface.

Implements RowIterator.

StartPSIBatchMode()

void MaterializeIterator::StartPSIBatchMode ( )

inlineoverridevirtual

Start performance schema batch mode, if supported (otherwise ignored).

PFS batch mode is a mitigation to reduce the overhead of performance schema, typically applied at the innermost table of the entire join. If you start it before scanning the table and then end it afterwards, the entire set of handler calls will be timed only once, as a group, and the costs will be distributed evenly out. This reduces timer overhead.

If you start PFS batch mode, you must also take care to end it at the end of the scan, one way or the other. Do note that this is true even if the query ends abruptly (LIMIT is reached, or an error happens). The easiest workaround for this is to simply call EndPSIBatchModeIfStarted() on the root iterator at the end of the scan. See the PFSBatchMode class for a useful helper.

The rules for starting batch and ending mode are:

1. If you are an iterator with exactly one child (FilterIterator etc.), forward any StartPSIBatchMode() calls to it.
2. If you drive an iterator (read rows from it using a for loop or similar), use PFSBatchMode as described above.
3. If you have multiple children, ignore the call and do your own handling of batch mode as appropriate. For materialization, #2 would typically apply. For joins, it depends on the join type (e.g., NestedLoopIterator applies batch mode only when scanning the innermost table).

The upshot of this is that when scanning a single table, batch mode will typically be activated for that table (since we call StartPSIBatchMode() on the root iterator, and it will trickle all the way down to the table iterator), but for a join, the call will be ignored and the join iterator will activate batch mode by itself as needed.

Reimplemented from RowIterator.

UnlockRow()

void MaterializeIterator::UnlockRow ( )

inlineoverridevirtual

Implements RowIterator.

Member Data Documentation

m_cte

Common_table_expr* MaterializeIterator::m_cte

private

If we are materializing a CTE, points to it (otherwise nullptr).

Used so that we see if some other iterator already materialized the table, avoiding duplicate work.

m_invalidators

Mem_root_array<Invalidator> MaterializeIterator::m_invalidators

private

m_join

JOIN* const MaterializeIterator::m_join

private

See constructor.

m_limit_rows

const ha_rows MaterializeIterator::m_limit_rows

private

See constructor.

m_num_materializations

uint64_t MaterializeIterator::m_num_materializations = 0

private

m_num_materialized_rows

uint64_t MaterializeIterator::m_num_materialized_rows = 0

private

m_query_blocks_to_materialize

Mem_root_array<QueryBlock> MaterializeIterator::m_query_blocks_to_materialize

private

m_query_expression

Query_expression* MaterializeIterator::m_query_expression

private

The query expression we are materializing.

For derived tables, we materialize the entire query expression; for materialization within a query expression (e.g. for sorting or for windowing functions), we materialize only parts of it. Used to clear correlated CTEs within the unit when we rematerialize, since they depend on values from outside the query expression, and those values may have changed since last materialization.

m_ref_slice

const int MaterializeIterator::m_ref_slice

private

The slice to set when accessing temporary table; used if anything upstream (e.g.

WHERE, HAVING) wants to evaluate values based on its contents. See constructor.

m_reject_multiple_rows

const bool MaterializeIterator::m_reject_multiple_rows

private

See constructor.

m_rematerialize

const bool MaterializeIterator::m_rematerialize

private

If true, we need to materialize anew for each Init() (because the contents of the table will depend on some outer non-constant value).

m_table_iterator

unique_ptr_destroy_only<RowIterator> MaterializeIterator::m_table_iterator

private

---

The documentation for this class was generated from the following files:

• sql/iterators/composite_iterators.h
• sql/iterators/composite_iterators.cc