WL#5579: Add 'page_cleaner' a separate thread to flush dirty pages

Status: Complete

In Trondheim we decided that it makes sense to move the flushing activity in 
innodb to a separate thread.

Currently flushing activity happens in either the master thread or in the user 

There are two types of flushing activities i.e.: flush_list and LRU_list. More 
details about this can be found in the following presentation: 

This WL will trace the following activities:

* Have a separate thread for flushing. We'll call it page_cleaner. No strict 
opinion on name. If you have something better I'll change it.
* Move the flushing of dirty pages from the master thread to the page_cleaner
* Move the flushing when log->max_modified_age_async is reached to the 

The following won't be touched by this WL:
* Flushing that happens in the user thread when log->max_modified_age_sync is 
* LRU flushing (WL#5580)

* After tweaking with adaptive flushing heuristics I have realized that it is 
better left as a separate piece of work. The scope of this WL is now to have a 
page_cleaner thread and move async flushing to the background. WL#5666 is for
adaptive flushing.

Changes in innodb status output:

                "Log sequence number %llu\n"
                "Log flushed up to   %llu\n"
+               "Pages flushed up to %llu\n"
                "Last checkpoint at  %llu\n",
+               log_buf_pool_get_oldest_modification(),

-       fprintf(file, "srv_master_thread loops: %lu 1_second, %lu sleeps, "
-               "%lu 10_second, %lu background, %lu flush\n",
-               srv_main_1_second_loops, srv_main_sleeps,
-               srv_main_10_second_loops, srv_main_background_loops,
-               srv_main_flush_loops);
+       fprintf(file, "srv_master_thread loops: %lu srv_active, "
+               "%lu srv_shutdown, %lu srv_idle\n",
+               srv_main_active_loops,
+               srv_main_shutdown_loops,
+               srv_main_idle_loops);
We'll have a separate page_cleaner thread.
- As a first cut I'll have a single page_cleaner. Eventually we can move to have 
multiple such threads.
- For now I'll not use the SRV_WORKER thread mechanism to dispatch discreet 
tasks to the page_cleaner. Sunny's MT purge has the basic infrastructure to use 
multiple worker threads. We can port this patch to eventually use that once MT 
purge makes it to the trunk.

page_cleaner tasks:
- flushing of dirty pages based on:
  * max_dirty_ratio
  * adaptive_flushing heuristic
  * async preflushing when nearing checkpoint age
- flushing of dirty pages in periods of inactivity
- flushing of dirty pages at shutdown

What page_cleaner will not do (still remain with master or other threads)
- Syncing of log buffer every second
- making checkpoints
- ibuf merge
- purge

page_cleaner and LRU flushing:
This WL won't touch LRU flushing. Eventually LRU flushing will be moved to the 
page_cleaner as well. At that point we may opt to have multiple page_cleaner if 

- page_cleaner will be spawned after IO threads and before any other threads 
during the server startup
- page_cleaner will exit after all other threads (including purge thread/s) have 
exited and before master thread has exited (maybe we can allow master thread to 
exit before page_cleaner and only need IO helper threads to alive and kicking).

polling or signalling:
We have a choice to either have the master thread (or other user threads) signal 
the page_cleaner that some work is needed or else we can use polling.
For the sake of simplicity I'd start with polling mechanism which is closer to 
what master thread has been doing i.e.: doing one second loops and ten seconds 
Note that page_cleaner will have to eventually wait on some event when there is 
no activity at all. And that suspension will end with a signal. But generally 
the page_cleaner will check state of the system itself to decide how much work 
to do and when to do it.

page_cleaner activities:

while (srv_shutdown_state == SRV_SHUTDOWN_NONE) {
  sleep if needed;
        do async_flushing
        do max_dirty_flushing
        do adaptive_flushing
     do idle_flushing

while(other threads active)
  keep doing shutdown_flushing


REF: rb://481