WL#7195: Offload connection authentication and thd/net initialization from accept thread to tp worker threads for thread pool plugin

THD Creation, vio/net initialization and connection authentication happens in the acceptor thread which accepts the connection. THD creation, vio/net initialization and connection authentication involves acquiring locks, memory allocation of various structures, network i/o. These tasks are both compute-bound and blocking.The acceptor thread is an event loop which waits for new connection events from clients.

To maximize the number of connections which can be handled per unit of time, the acceptor thread should spend as much of it's time listening for new connections. This means that the thd creation and other tasks should not be handled as part of accept event loop. Instead these tasks delegated to thread pool worker threads that handle the client connections. It is to be noted that connection authentication should be done as quickly as possible so that connect latency is not increased and it should happen within the default connect timeout settings in the server.

1. NF1: The worklog shall offload the work of vio & net initialization along with connection authentication to thread pool worker threads which shall result in an optimal connection throughput.

2. FR1: A new field CONNECT_THREAD_COUNT shall be introduced in INFORMATION_SCHEMA.TP_THREAD_GROUP_STATE. This shall indicate the number of connect threads that process connection events or awaiting to process connection events. There will be 1 connect thread available per thread group to process connection and there can be a maximum of 4 connection threads per thread group.

Connection authentication, THD creation & vio/net initialization happens in acceptor thread when server uses thread pool plugin for connection handling. Connection authentication involves exchanging handshake packets with client , reading from ACL tables, encryption & decryption etc. These tasks are both compute bound and blocking. Thus client connections can get queued in accept queue resulting in increased connect latency and decreased connection throughput. In the worst case, some connections could be dropped.

In addition, THD creation and vio/net initialization also happens in accept thread for thread pool plugin. THD creation and vio/net initialization involves acquiring locks, memory allocation of various structures and other system & library calls. The accept thread is an event loop which waits for new connection events from clients. To maximize the number of connections that can be handled per unit of time as well as improve on connect latency, acceptor thread should spend as much of it’s time listening for new connections.

The worklog details high-level architecture and design for offloading connection authentication, THD creation & vio/net initialization for thread pool plugin.

In the new design, each thread group shall have a queue of connection_context objects, which represent new connections that need to be processed by the thread group connect handler threads. The connection_context is composed of a lightweight channel_info object and list pointers for a intrusive queue implementation. In addition, we have a dedicated thread per thread group to handle the connection events.

Whenever a new connection arrives, a connection context object is allocated and added to the connection context queue of the thread group to which the connection is assigned. The dedicated thread per thread group for handling the connection is woken to handle the connection event as quickly as possible. If the dedicated thread is processing an already existing connection initialization/authentication task on the thread group, then a new thread shall be created to handle the connection initialization and authentication. The number of connect handler threads is limited to a maximum of four. After a time interval of 60 seconds, all extra threads that are created for connect handling task shall be retired if no new connection is available for processing. Accordingly the worker thread loop is modified to check for connection events and process the connection event if the designated thread is a connect handler thread. Thus an extra thread is reserved per thread group to handle connection login/authentication events . Hence we have at least two threads per thread group:

• A thread that waits on the IO event wait system call to process a query event.
• Another thread which sleeps on a condition variable awaiting new connection event.