There are special engines used for optimization purposes. These engines do not have a full range of features. They can only fetch data. The normal engine can be replaced with such special engines only during the optimization process.

Now we have two such engines:

subselect_uniquesubquery_engine used for:

left_expression IN (SELECT primary_key FROM table WHERE conditions)

This looks for the given value once in a primary index, checks the WHERE condition, and returns was it found or not?

left_expression IN (SELECT any_key FROM table WHERE conditions)

This first looks up the value of the left expression in an index (checking the WHERE condition), then if value was not found, it checks for NULL values so that it can return NULL correctly (only if a NULL result makes sense, for example if an IN subquery is the top item of the WHERE clause then NULL will not be sought)

The decision about replacement of the engine happens in JOIN::optimize, after calling make_join_readinfo, when we know what the best index choice is.