WL#5758: InnoDB: Improve Thread Scheduling

Status: Complete   —   Priority: Medium

The InnoDB thread scheduling has in the past been called the throttling code. And 
roughly that's exactly what it does. All threads are given a configurable number 
of  tickets and each time they enter InnoDB they use up a ticket. Users can set a 
configuration parameter to control how many threads can concurrently enter InnoDB 
at any given time. The code uses mutexes and events to synchronise the operation.
A thread waiting inside InnoDB code on a table or record lock is deemed to be not 
inside InnoDB. The inside InnoDB counter is decremented. When that thread is woken 
up we increment the inside counter again. User's can disable the throttling code 
by setting innodb_thread_concurrency to 0. This has served us well in the past 
when the number of threads inside InnoDB was limited to approximately < 16. With 
plans to scale beyond 1024 threads we need a new mechanism that takes into 
consideration other kinds of waits, e.g., IO waits. Also, the overhead of using 
mutexes and events to manage the queue has now become an unacceptable overhead.

The plan is to use:

  1. Atomics where possible to manage the scheduling

  2. Take into consideration all waits including IO.
Introduce new config variable:

  innodb_adaptive_max_sleep_delay := INT [0 ... 1000000] in micro seconds.

If > 0 then it is enabled, value of 0 disables adaptive sleep. If enabled then
the variable innodb_thread_sleep_delay will be adjusted dynamically based on the
system load. The aim is to make it converge on the optimal sleep delay when
concurrency thread limit is active.