WL#12138: Add Admin Port
Affects: Server-8.0 — Status: Complete — Priority: Medium
Add capability to specify a separate IP address and TCP port that are ready to accept TCP connection and speak MySQL protocol. Provide a dedicated thread to handle interaction between a server and a system administrator communicating via this socket. Introduce a separate privilege to restrict network access to a socket specified by this pair of IP address/TCP port. User's use cases: This worklog was initiated by the request from Facebook. See BUG#90395 CONTRIBUTION BY FACEBOOK: ADD ADMIN TCP/IP PORT IN MYSQL. The following explanation was received on request for comments about the new feature: Admin port is a separate port and thread handling it’s connections so that even if the main server thread is overloaded accepting too many connections critical infrastructure automation and humans can still connect to the server to maintain and remediate it. Given the high CPU and RTT cost of TLS, even if you have max_connections set to 3000 for example, that’s all way up the stack in the application layer, and nothing is there to prevent 200,000 clients from trying to establish a TCP / TLS connection with the server. In scenarios like this max_connections doesn’t help because the effort that is overloading the server is all in layer 4 / layer 5 before it even gets to the application (mysqld).
FR.1 Provide command options --admin-address and --admin-port to specify an address and TCP port for socket binding and listening incoming connections. FR.2 The option --admin-address must specify either IPv4 address, or IPv6 address or a host name. FR.3 The option --admin-address mustn't allow to specify any of the following wildcard values *, ::, 0.0.0.0. FR.3.1 The new error code ER_INVALID_VALUE_OF_ADMIN_ADDRESS must be introduced to report about error when wildcard values specified for command line option --admin-address. FR.3.2 Attempt to specify any of the special treated values '*', '::' and '0.0.0.0' for command line option --admin-address must lead to output of the error ER_INVALID_VALUE_OF_ADMIN_ADDRESS to an error stream and stop server running. FR.4 The option --admin-port must accept valid value of TCP port for incoming connections. FR.5 In case a value for the command option --admin-port is not specified, default value must be provided. FR.5.1 Default value for the option --admin-port must be 33062. FR.6 The new privilege SERVICE_CONNECTION_ADMIN must be provided to restrict access to a new socket specified by the options --admin-address/--admin-port. FR.6.1 In case a user without the privilege SERVICE_CONNECTION_ADMIN tries to establish mysql session to a socket specified by the command options --admin-address/--admin-port the error ER_SERVER_NEW_ABORTING_CONNECTION must be returned to the user and TCP connections must be closed. FR.7 Limitation on a number of concurrent connections to a server mustn't affect connections to a socket specified by the command options --admin-address/--admin-port. FR.8 Provide possibility to specify whether handling of incoming TCP connection requests be done by a dedicated thread or in regular way as for other incoming TCP connection requests. FR.9 In case a server can't bind to a socket specified by the options --admin-address/--admin-port it must produce an error and does not start. FR.9.1 The error ER_CONN_TCP_BIND_FAIL must be logged in case a server can't bind on a socket specified by the options --admin-address/--admin-port. FR.10 Server must open admin interface for listening and accept connection requests on it only in case the option --admin-address specified on server start up. If the option --admin-address missed on server start up a socket corresponding to admin interface mustn't be opened.
New command options to support admin interface. ============================================== The new command-line options introduced to support admin interface. These options are --admin-address and --admin-port and they are read-only global variables. The options --admin-address is to specify an interface address to listen for connection requests. The option --admin-port is to specify port number on that a transport connection can be established. Together these two command-line options specify an admin interface. New privilege to get access to admin interface. =============================================== To restrict access to admin interface only privileged users, the new privilege SERVICE_CONNECTION_ADMIN is introduced. In case a user without this privilege tries to establish a connection to the admin interface the error message with error code ER_SPECIFIC_ACCESS_DENIED_ERROR returned. Two approaches to process connection request on admin interface. ================================================================ There are two ways to handle connection requests on admin interface (presented by a pair of command line options --admin-address/--admin-port). The first way is to listen incoming connection requests on admin interface from the same thread as it is done for sockets presented by command line options --port/--bind-address. This case assumes only one minor exception in handling connections requests on admin interface in comparing with ordinary sockets (presented by command line options --port/--bind-address), namely in case a socket corresponding to an admin interface has ready incoming connection request handle it first before handling connection requests on other sockets configured by command line options --port/--bind-address.. The second one is to listen incoming connection requests on admin interface in a separate thread created exclusively for this task. The first approach doesn't increase burden on OS scheduler since no new thread is created to handle connection requests on admin interface while admin socket still given higher priority in getting connected socket for a new established connection on admin interface. The second approach with a dedicated thread for handling incoming connection requests is a use case suggested by Facebook in its contributions patch. At the moment of implementing this WL it is not obvious what approach better and more efficient. Therefore, both ways to handle incoming connection requests on admin interface were implemented and further decision what approach is the best should be done after comprehensive performance analysis by QA. To choose which approach to use for listening incoming connection requests on the admin interface the new command line option --create-admin-listener-thread is provided. By default this option has a false value. In case this option has a true value when MySQL server being started a new thread for handling connection requests on admin interface spawned. Properties of the new thread for handling connection requests on admin interface ================================================================================ This thread is spawned with attributes THREAD_CREATE_JOINABLE and PTHREAD_SCOPE_SYSTEM. The main thread spawns the thread for handling connection requests on admin interface and waiting for the new thread be started up. On server shutdown, method close_listener() of the class Mysqld_socket_listener is called to terminate the thread for listening incoming connection requests on admin interface and wait for the thread be finished. After the tread finished the socket corresponding admin interface will be closed. Checking for restrictions on max allowed connections. ===================================================== Current implementation contains two limits on a maximum number of connections. The first limit is set by the option max_connections and specifies the maximum permitted number of simultaneous client connections. The second limit is set by the option max_user_connections and specifies the maximum number of simultaneous connections permitted to any given MySQL user account. In case a connection being established on admin interface the checking for a limit specified by the option max_connections is done before user authentication. In other word, any attempt to establish a transport connection to admin interface will be successful and a connected sock for the new connection be accepted by the server even if a user, on behalf this connection request being made, doesn't have the the privilege SERVICE_CONNECTION_ADMIN. Of course, after connection be established, authentication procedure will be started for the user and error be returned in case a user doesn't have the privilege SERVICE_CONNECTION_ADMIN. To summarize, checking for maximum number of simultaneous transport connections is done before user authentication procedure started. Such separation of checking for limit specified by the option max_connections and checking for privilege to establish a connection to admin interface looks strange at first glance but have strong reasoning. The matter is that control on maximum number of simultaneous connections is done when no information about user who tries to establish connection is known. At this moment only transport connection taken place and no exchange of messages over MySQL session protocol has been initiated. So, at this stage all that known is that transport connection has been established to admin interface. On the other hand, the limit specified by the option max_user_connections is checked for after a user was authenticated and check for the privilege SERVICE_CONNECTION_ADMIN was already done. That is, first user is authenticated and checking for permission SERVICE_CONNECTION_ADMIN and after that restriction specified by the option max_user_connections is checked. New information messages added by the worklog ============================================= When server is configured to listen connection requests on admin interface (that is, when the option --admin-address is set) the information message "Admin interface ready for connections, address: '%s' port: %d" is output to console. This information message has the code ER_SERVER_STARTUP_ADMIN_INTERFACE Changes in access to counters on number of network specific errors ================================================================== Before support for admin interface be implemented, the following counters: the counters for a number of connection errors when selecting on the listening port, the counter for a number of connection errors when accepting sockets on the listening port, the counter for a number of connection errors from TCP wrappers were just integer variables. Since support for admin interface assumes that these counters can be modified from two threads simultaneously, these counters are converted to std::atomic.
Copyright (c) 2000, 2019, Oracle Corporation and/or its affiliates. All rights reserved.