WL#6642: InnoDB: multiple page_cleaner threads

Status: Complete   —   Priority: Medium

Currently there is a single page_cleaner thread which is tasked with flushing 
dirty pages from the buffer pool[s]. In case of multiple buffer pool instances, 
which is the default in 5.6, each buffer pool has its own free list, flush list 
and LRU list. The page_cleaner thread works on LRU and flush list. The idea of 
this worklog is to have one page_cleaner thread per buffer pool instance.

* One of the major motivation is scalability. As buffer pool instances are 
neatly isolated from each other it makes much more sense to have a separate 
background thread for each instance which works solely for that instance. That 
will give us parallel flushing.

* Another reason is that in certain cases having a single background thread can 
actually cause regression with multiple buffer pool instances. When flushing 
from the flush list up to a certain LSN we scan all instances one by one and 
dispatch batches of flush. The flush lists are ordered by LSN in that each flush 
list tail will have the oldest LSN page but it is quite possible that the 
overall oldest page is found in the flush list of last buffer pool instance. For 
example, if the oldest page is at the tail of buffer pool instance number 64 
then all the batches that we do in the first 63 instances won't be able to move 
the flush LSN. This can lead to user threads waiting and eventually entering 
into sync flushing aka 'furious flushing'. This was unearthed by Dimitri in his 
benchmarks.

'innodb_page_cleaners' option is added.
Variable Scope   | Global
Dynamic Variable | No
Type             | numeric
Default          | 1
Range            | 1 .. 64

innodb_page_cleaners = 1 is intended to keep current behavior.

And page_cleaners are not dedicated to the each buffer pool instances now.
The task control for each buffer pool is implemented as slot array,
and each page_cleaner threads take the task until all finished.
page_cleaners are not dedicated to the each buffer pool instances now.
The task control for each buffer pool is implemented as slot array,
and each page_cleaner threads take the task until all finished.

The coordinator thread requests tasks through the slot array.
And the all page cleaner threads (coordinator and workers) do the tasks.

The new option 'innodb_page_cleaners' decides the number of the page cleaner
threads (coordinator and workers). So, '1' means the coordinator thread only.