WL#12138: Add Admin Port

Affects: Server-8.0   —   Status: Complete

Add capability to specify a separate IP address and TCP port that are ready
to accept TCP connection and speak MySQL protocol. Provide a dedicated thread
to handle interaction between a server and a system administrator communicating
via this socket. Introduce a separate privilege to restrict network access
to a socket specified by this pair of IP address/TCP port.

User's use cases:
This worklog was initiated by the request from Facebook. See BUG#90395
CONTRIBUTION BY FACEBOOK: ADD ADMIN TCP/IP PORT IN MYSQL.

The following explanation was received on request for comments about the new 
feature:

Admin port is a separate port and thread handling it’s connections so that even
if the main server thread is overloaded accepting too many connections critical
infrastructure automation and humans can still connect to the server to maintain
and remediate it. Given the high CPU and RTT cost of TLS, even if you have
max_connections set to 3000 for example, that’s all way up the stack in the
application layer, and nothing is there to prevent 200,000 clients from trying
to establish a TCP / TLS connection with the server. In scenarios like this
max_connections doesn’t help because the effort that is overloading the server 
is all in layer 4 / layer 5 before it even gets to the application (mysqld).
FR.1 Provide command options --admin-address and --admin-port to specify
an address and TCP port for socket binding and listening incoming connections.

FR.2 The option --admin-address must specify either IPv4 address, or IPv6
address or a host name.

FR.3 The option --admin-address mustn't allow to specify any of the following
wildcard values *, ::, 0.0.0.0.

FR.3.1 The new error code ER_INVALID_VALUE_OF_ADMIN_ADDRESS must be introduced
to report about error when wildcard values specified for command line option
--admin-address.

FR.3.2 Attempt to specify any of the special treated values '*', '::' and
'0.0.0.0' for command line option --admin-address must lead to output
of the error ER_INVALID_VALUE_OF_ADMIN_ADDRESS to an error stream and stop
server running.

FR.4 The option --admin-port must accept valid value of TCP port for incoming
connections.

FR.5 In case a value for the command option --admin-port is not specified,
default value must be provided.

FR.5.1 Default value for the option --admin-port must be 33062.

FR.6 The new privilege SERVICE_CONNECTION_ADMIN must be provided to restrict
access to a new socket specified by the options --admin-address/--admin-port.

FR.6.1 In case a user without the privilege SERVICE_CONNECTION_ADMIN
tries to establish mysql session to a socket specified by the command options
--admin-address/--admin-port the error ER_SERVER_NEW_ABORTING_CONNECTION must be
returned to the user and TCP connections must be closed.

FR.7 Limitation on a number of concurrent connections to a server mustn't
affect connections to a socket specified by the command options
--admin-address/--admin-port.

FR.8 Provide possibility to specify whether handling of incoming TCP connection
requests be done by a dedicated thread or in regular way as for other incoming
TCP connection requests.

FR.9 In case a server can't bind to a socket specified by the options
--admin-address/--admin-port it must produce an error and does not start.

FR.9.1 The error ER_CONN_TCP_BIND_FAIL must be logged in case a server can't
bind on a socket specified by the options --admin-address/--admin-port.

FR.10 Server must open admin interface for listening and accept connection
requests on it only in case the option --admin-address specified on
server start up. If the option --admin-address missed on server start up
a socket corresponding to admin interface mustn't be opened.
New command options to support admin interface.
==============================================
The new command-line options introduced to support admin interface.
These options are --admin-address and --admin-port and they are read-only global
variables. The options --admin-address is to specify an interface address to 
listen for connection requests. The option --admin-port is to specify port
number on that a transport connection can be established. Together these two 
command-line options specify an admin interface.

New privilege to get access to admin interface.
===============================================
To restrict access to admin interface only privileged users, the new privilege 
SERVICE_CONNECTION_ADMIN is introduced. In case a user without this privilege
tries to establish a connection to the admin interface the error message with 
error code ER_SPECIFIC_ACCESS_DENIED_ERROR returned.

Two approaches to process connection request on admin interface.
================================================================
There are two ways to handle connection requests on admin interface (presented
by a pair of command line options --admin-address/--admin-port). The first way
is to listen incoming connection requests on admin interface from the same
thread as it is done for sockets presented by command line options
--port/--bind-address. This case assumes only one minor exception in handling
connections requests on admin interface in comparing with ordinary sockets
(presented by command line options --port/--bind-address), namely in case a
socket corresponding to an admin interface has ready incoming connection request
handle it first before handling connection requests on other sockets configured
by command line options --port/--bind-address..

The second one is to listen incoming connection requests on admin interface in
a separate thread created exclusively for this task.

The first approach doesn't increase burden on OS scheduler since no new thread
is created to handle connection requests on admin interface while admin socket
still given higher priority in getting connected socket for a new established
connection on admin interface. The second approach with a dedicated thread for
handling incoming connection requests is a use case suggested by Facebook in its
contributions patch. At the moment of implementing this WL it is not obvious
what approach better and more efficient. Therefore, both ways to handle
incoming connection requests on admin interface were implemented and further
decision what approach is the best should be done after comprehensive 
performance analysis by QA.

To choose which approach to use for listening incoming connection requests on
the admin interface the new command line option --create-admin-listener-thread
is provided. By default this option has a false value. In case this option
has a true value when MySQL server being started a new thread for handling 
connection requests on admin interface spawned.

Properties of the new thread for handling connection requests on admin interface
================================================================================
This thread is spawned with attributes THREAD_CREATE_JOINABLE and
PTHREAD_SCOPE_SYSTEM. 

The main thread spawns the thread for handling connection requests on admin
interface and waiting for the new thread be started up.

On server shutdown, method close_listener() of the class Mysqld_socket_listener
is called to terminate the thread for listening incoming connection requests on
admin interface and wait for the thread be finished. After the tread finished
the socket corresponding admin interface will be closed.

Checking for restrictions on max allowed connections.
=====================================================
Current implementation contains two limits on a maximum number of connections.
The first limit is set by the option max_connections and specifies the maximum
permitted number of simultaneous client connections. The second limit is set by
the option max_user_connections and specifies the maximum number of simultaneous
connections permitted to any given MySQL user account.

In case a connection being established on admin interface the checking for a
limit specified by the option max_connections is done before user 
authentication. In other word, any attempt to establish a transport connection
to admin interface will be successful and a connected sock for the new
connection be accepted by the server even if a user, on behalf this connection
request being made, doesn't have the the privilege SERVICE_CONNECTION_ADMIN.
Of course, after connection be established, authentication procedure will be
started for the user and error be returned in case a user doesn't have the
privilege SERVICE_CONNECTION_ADMIN. To summarize, checking for maximum number of
simultaneous transport connections is done before user authentication procedure
started. Such separation of checking for limit specified by the option
max_connections and checking for privilege to establish a connection to admin
interface looks strange at first glance but have strong reasoning. 
The matter is that control on maximum number of simultaneous connections is done
when no information about user who tries to establish connection is known.
At this moment only transport connection taken place and no exchange of messages
over MySQL session protocol has been initiated. So, at this stage all that
known is that transport connection has been established to admin interface.

On the other hand, the limit specified by the option max_user_connections
is checked for after a user was authenticated and check for the privilege 
SERVICE_CONNECTION_ADMIN was already done. That is, first user is authenticated
and checking for permission SERVICE_CONNECTION_ADMIN and after that restriction
specified by the option max_user_connections is checked.

New information messages added by the worklog 
=============================================
When server is configured to listen connection requests on admin interface
(that is, when the option --admin-address is set) the information message 
"Admin interface ready for connections, address: '%s'  port: %d" is output to
console. This information message has the code ER_SERVER_STARTUP_ADMIN_INTERFACE

Changes in access to counters on number of network specific errors
==================================================================
Before support for admin interface be implemented, the  following counters: 
  the counters for a number of connection errors when selecting on the listening
  port,
  the counter for a number of connection errors when accepting sockets on the
  listening port,
  the counter for a number of connection errors from TCP wrappers
were just integer variables.
Since support for admin interface assumes that these counters can be modified
from two threads simultaneously, these counters are converted to std::atomic.