WL#8423: InnoDB: Remove the buffer pool mutex

Affects: Server-8.0   —   Status: Complete   —   Priority: Medium

Buffer pool mutex protects several data structures at once. It may become hot in 
some workloads. Increasing the number of buffer pool instances does not always 
help, as some buffer pool instances (the ones that hot pages hash to) are 
naturally hotter than the others.

The main idea is to split buffer pool mutex into several mutexes: separate mutex 
for free_list, LRU_list, zip_free, and zip_hash.

The original patch has been contributed by Percona. See BUG#75534.
Functional requirements:
F-1: Buffer pool with split mutexes should work/function as one buffer pool 
mutex.

Non-Functional requirements:
NF-1: Implicit requirements: No new SQL needed, work on all  platforms, do not 
break replication, backup, partitioning, FK, or any  other exiting features.

NF-2: No change in semantics expected.
Removes the buffer pool mutex.

1. Introduces several new list/hash protecting mutexes, and access without any 
mutex to several variables. 
  The new mutexes are
  - LRU_list_mutex for the LRU_list;
  - zip_free mutex for the zip_free arrays;
  - zip_hash mutex for the zip_hash hash and in_zip_hash flag;
  - free_list_mutex for the free_list and withdraw list.
  - flush_state_mutex for init_flush, n_flush, no_flush arrays.

2. The variables switched from buffer pool mutex protection to atomic operations 
and/or os_rmb/os_wmb. Particularly the uses of latter might be very debatable.
  - srv_buf_pool_old_size,
    srv_buf_pool_size,
    srv_buf_pool_curr_size,
    srv_buf_pool_base_size
  - buf_pool->buddy_stat[i].used
  - buf_pool->curr_size, n_chunks_new 
1. Introduces several new list/hash protecting mutexes, and access without any 
mutex to several variables.

storage/innobase/include/buf0buf.h
struct buf_pool_t{

        /** @name General fields */
        /* @{ */
        BufListMutex    LRU_list_mutex; /*!< LRU list mutex */
        BufListMutex    free_list_mutex;/*!< free and withdraw list mutex */
        BufListMutex    zip_free_mutex; /*!< buddy allocator mutex */
        BufListMutex    zip_hash_mutex; /*!< zip_hash mutex */
        ib_mutex_t      flush_state_mutex;/*!< Flush state protection
                                        mutex */

2. The variables switched from buffer pool mutex protection to atomic operations 
and/or os_rmb/os_wmb.
   functions:
   btr_search_enable(),
   buf_resize_thread(), 
   buf_get_withdraw_depth()
   ...

3. Exploits the fact that freed pages must have no pointers to them from the 
buffer pool nor from any other thread except for the freeing one to remove 
redundant locking. The same applies to freshly allocated pages before any 
pointers to them are published. This however necessitates removing some of the 
debug checks that scan buffer pool chunks directly, as they don't have a way to 
freeze such blocks.

  storage/innobase/buf/buf0buf.cc
  buf_block_align()

4. buf_buddy_alloc() rewritten not to require the buffer pool mutex at the 
start, which then might be released, and this fact propagated to the caller to 
make decisions to re-check things. It is now called with mutexes unlocked, and 
the caller buf_page_init_for_read algorithm has been simplified. All its 
allocations now happen with mutexes unlocked.

5. buf_flush_LRU_list_batch() uses mutex_enter_nowait to skip over any 
currently-locked blocks.

6. Avoid unnecessary block mutex enter.

   storage/innobase/include/buf0buf.ic
   buf_page_get_io_fix_unlocked()