WL#9267: Mysqlx connection timeout

Affects: Server-8.0   —   Status: Complete   —   Priority: Medium

Description
===========

X Plugin monitors not authenticated connections for their time of living. User
is dropped after the time reaches the maximum allowed time. Authenticated
connections are not monitored. When peer application is going to hang after
authentication (doesn't generate queries) then it is going to take server
resources forever. If fault application leaves a connection then after some
time the number of maximum allowed connections could be reached easily
('mysqlx_max_connections' or 'max_connections'). It could have serious
consequences for other users, like blocking access for new connections.

To secure the MySQL Servers and X Plugins resources the plugin needs to monitor
time of all I/O operations and drop idle connections.


Notes
=====

Same timeouts for MySQL Server:

* [net_read_timeout](https://dev.mysql.com/doc/refman/5.7/en/server-system-variables.html#sysvar_net_read_timeout)
* [net_write_timeout](https://dev.mysql.com/doc/refman/5.7/en/server-system-variables.html#sysvar_net_write_timeout)
* [interactive_timeout](https://dev.mysql.com/doc/refman/5.7/en/server-system-variables.html#sysvar_interactive_timeout)
* [wait_timeout](https://dev.mysql.com/doc/refman/5.7/en/server-system-variables.html#sysvar_wait_timeout)
* [Aborted_clients](https://dev.mysql.com/doc/refman/5.7/en/server-status-variables.html#statvar_Aborted_clients)
Functional requirements
=======================

1. User should be able to set all Mysqlx io-timeouts system variables from
   MySQLs Server configuration file
2. User should be able to set all Mysqlx io-timeouts system variables from
   command line arguments passed to MySQL Server
3. Any user should be able to set all Mysqlx io-timeouts session system
   variables
4. User without super privilege or SYSTEM_VARIABLES_ADMIN privilege, shouldn't be 
   able to set any of Mysqlx io-timeouts global system variables
5. User should be able to get values of all Mysqlx io-timeouts global system
   variables
6. Client should be able to inform the server that the session is
   interactive one
7. Connection must be dropped after "mysqlx_wait_timeout" time on
   noninteractive client when client doesn't send the message header
8. Connection must be dropped after "mysqlx_interactive_timeout" time on
   interactive client when client doesn't send the message header
9. Connection must be dropped after "mysqlx_read_timeout" time when client
    doesn't send the message payload
10. Connection must be dropped after "mysqlx_write_timeout" time when
    the send data weren't stored in TCP stack
11. Connection must be dropped after 'min(mysqlx_wait_timeout,
    mysqlx_connection_timeout)' time on non-authenticated connection when client
    doesn't send the message header
12. Connection must be droppped after 'min(mysqlx_read_timeout,
    mysqlx_connection_timeout)' time on non-authenticated connection when client
    doesn't send the message payload

Non-functional requirements
===========================

1. X Plugin performance should be at the same level
2. X Plugin percent coverage shouldn't be less then before the feature
Protocol
========

Each message in X Protocol begins with a header followed by protobuf encoded
message. Currently the plugin assumes that the user is active after reception
of the header. This is visible in performance schema in table
"socket_instances", column "STATE". Current worklog should use same definition
of active connection and differentiate timeout variables base on this.

I/O timeouts
------------

There are three different scenarios which must be "secured":

1. X Plugin tries to send data to client. If the write operation isn't going to
   end in time defined by "mysqlx_write_timeout" variable (look at
   configuration section) then the connection must be aborted.
2. X Plugin tries to read message header from client and it isn't going to end
   in time defined by "mysqlx_wait_timeout" then the connection must be aborted.
3. X Plugin tries to read message payload from client and it isn't going to end
   in time defined by "mysqlx_read_timeout" then the connection must be aborted.

When aborting after read failure, the plugin needs to send an error message
to the client. The "Mysqlx.Error" message can't be used because it is
synchronous, it goes as an response in a concrete flow. X Protocol defines
global notices which are asynchronous and they may be send at any moment.
Before aborting the plugin must transmit a global notice with "frame" that
contains a "Mysqlx.Notice.Warning" message with "level" field set to "ERROR",
"code" field set to "ER_IO_READ_ERROR", "message" field set to information
about the timeout. In case of write failure the plugin can't transmit anything
other to client, notice isn't send in this case.

When X Plugin is going to send a notice, the information isn't guarantied to
reach the client. It because of an issue with windows VIO implementation which
doesn't takes into account the behavior of TCP stack.
Windows TCP stack requires that the TCP buffers are empty when calling "close"
on the socket. In case when TCP buffers contain data, "close" is going to send
"RST" packet, breaking the connection (notice isn't delivered).

Time of living of unauthenticated client is already monitored and can't
exceed "mysqlx_connection_timeout". X Plugin while doing I/O should get the
timeout value from one of cases described above and compare it with
"mysqlx_connection_timeout" to get lowest value. The chosen value must be
considered as the timeout for the I/O operation. The time that client has for
authentication is still the same.

Capabilities
------------

"session.mysqlx_wait_timeout" must be used for every connection that is active.
In case of non-interactive client the default value of
"session.mysqlx_wait_timeout" is copied from "global.mysqlx_wait_timeout"
variable, in other case it's copied from "global.mysqlx_interactive_timeout".
To distinguish interactive and non-interactive client a new capability must be
introduced. The new capabilities name should be "client.interactive" and it must
accept "boolean", "integer", "float" values which must be interpreted as
"boolean" value.

    client -> server: Mysqlx.Connection.CapabilitySet(["client.interactive"=true])
    client <- server: Mysqlx.Ok | Mysqlx.Error

Setting this capability on old MySQL Servers (X Plugins) is going to return
an error because it doesn't exist there. If this capability is optional for the client then by ignoring the error, it accepts that X Plugin uses unlimited timeout values.


Configuration
=============

All timeout values must be configurable using plugin system variables. X Plugin must support following:

* mysqlx_wait_timeout

|Command-Line Format    |--mysqlx_wait_timeout=#          |
|-----------------------|---------------------------------|
|System Variable Name   |mysqlx_wait_timeout              |
|Variable Scope         |Session                          |
|Dynamic Variable       |Yes                              |
|Permitted Values Type  |integer                          |
|Default                |28800                            |
|Min Value              |1                                |
|Max Value              |2147483                          |

Number or seconds that X Plugin must wait for activity on a connection. After this time the read-operation/connection is going to be aborted.
If the client is non-interactive, the initial value of "session" variable is copied from "global.mysqlx_wait_timeout". For interactive client the initial value is copied from "global.mysqlx_interactive_timeout".

* mysqlx_interactive_timeout

|Command-Line Format    |--mysqlx_interactive_timeout=#   |
|-----------------------|---------------------------------|
|System Variable Name   |mysqlx_interactive_timeout       |
|Variable Scope         |Global                           |
|Dynamic Variable       |Yes                              |
|Permitted Values Type  |integer                          |
|Default                |28800                            |
|Min Value              |1                                |
|Max Value              |2147483                          |

Default value for "session.mysqlx_wait_timeout" for interactive client.

* mysqlx_read_timeout

|Command-Line Format    |--mysqlx_read_timeout=#          |
|-----------------------|---------------------------------|
|System Variable Name   |mysqlx_read_timeout              |
|Variable Scope         |Session                          |
|Dynamic Variable       |Yes                              |
|Permitted Values Type  |integer                          |
|Default                |30                               |
|Min Value              |1                                |
|Max Value              |2147483                          |

Number or seconds that X Plugin must wait for blocking read operation to
complete. After this time the operation/connection is going to be aborted.

* mysqlx_write_timeout

|Command-Line Format    |--mysqlx_write_timeout=#         |
|-----------------------|---------------------------------|
|System Variable Name   |mysqlx_write_timeout             |
|Variable Scope         |Session                          |
|Dynamic Variable       |Yes                              |
|Permitted Values Type  |integer                          |
|Default                |60                               |
|Min Value              |1                                |
|Max Value              |2147483                          |

Number or seconds that X Plugin must wait for blocking write operation to
complete. After this time the operation/connection is going to be aborted.


Error and Warnings
==================

All errors generated by timeouts must be transmitted in global notice,
in following flow:

    client <- server: Mysqlx.Notice.Frame

Aborted read operations are going to send following error:

|Property    |Value                                         |
|------------|----------------------------------------------|
|Name        |ER_IO_READ_ERROR                              |
|Error code  |1810                                          |
|Error text  |IO Read error: (%lu, %s) %s                   |


Instrumentation
===============

Transmission of local notice containing a "Mysqlx.Notice.Warning" increments
"Mysqlx_notice_warning_sent" status variable. The same status variable must be
increment in case of global notice that contain "Mysqlx.Notice.Warning"
message.

* Mysqlx_notice_warning_sent

|Property       |Value                     |
|---------------|--------------------------|
|Variable Name  |Mysqlx_notice_warning_sent|
|Type           |integer                   |
|Scope          |GLOBAL, SESSION           |
|Default        |0                         |

Number of local and global warning notices sent back to client.


* Mysqlx_aborted_clients

|Property       |Value                     |
|---------------|--------------------------|
|Variable Name  |Mysqlx_aborted_clients    |
|Type           |integer                   |
|Scope          |GLOBAL                    |
|Default        |0                         |

Number of clients that were disconnected because of IO timeout.
VIO library already supports timeouts. After setting the "timeout" all I/O
operations are switched from blocking "read/write" to nonblocking with
additional call to "select".

The implementation can be summarized in several steps:

* Add plugin system variables
* Add storage for those variables in "xpl::Plugin_system_variables"
* Add capability "client.interactive"
* Add storage for the capability in "(ngs::|xpl::)Client"
* Choose a timeout variable and pass it to "vio_timeout"
* when "vio_read" fails, send a notice and disconnect
* when "vio_write" fails, disconnect