WL#6642: InnoDB: multiple page_cleaner threads
Status: Complete — Priority: Medium
Currently there is a single page_cleaner thread which is tasked with flushing dirty pages from the buffer pool[s]. In case of multiple buffer pool instances, which is the default in 5.6, each buffer pool has its own free list, flush list and LRU list. The page_cleaner thread works on LRU and flush list. The idea of this worklog is to have one page_cleaner thread per buffer pool instance. * One of the major motivation is scalability. As buffer pool instances are neatly isolated from each other it makes much more sense to have a separate background thread for each instance which works solely for that instance. That will give us parallel flushing. * Another reason is that in certain cases having a single background thread can actually cause regression with multiple buffer pool instances. When flushing from the flush list up to a certain LSN we scan all instances one by one and dispatch batches of flush. The flush lists are ordered by LSN in that each flush list tail will have the oldest LSN page but it is quite possible that the overall oldest page is found in the flush list of last buffer pool instance. For example, if the oldest page is at the tail of buffer pool instance number 64 then all the batches that we do in the first 63 instances won't be able to move the flush LSN. This can lead to user threads waiting and eventually entering into sync flushing aka 'furious flushing'. This was unearthed by Dimitri in his benchmarks.
'innodb_page_cleaners' option is added. Variable Scope | Global Dynamic Variable | No Type | numeric Default | 1 Range | 1 .. 64 innodb_page_cleaners = 1 is intended to keep current behavior. And page_cleaners are not dedicated to the each buffer pool instances now. The task control for each buffer pool is implemented as slot array, and each page_cleaner threads take the task until all finished.
page_cleaners are not dedicated to the each buffer pool instances now. The task control for each buffer pool is implemented as slot array, and each page_cleaner threads take the task until all finished. The coordinator thread requests tasks through the slot array. And the all page cleaner threads (coordinator and workers) do the tasks. The new option 'innodb_page_cleaners' decides the number of the page cleaner threads (coordinator and workers). So, '1' means the coordinator thread only.
Copyright (c) 2000, 2018, Oracle Corporation and/or its affiliates. All rights reserved.