WL#9252: X Protocol connection compression
Motivation
Latency of requests on networks depends on:
- networks RTT
- utilization of the network
While the networks RTT can't be changed, the utilization of the network can be changed by compressing the packets to reduce the traffic.
In context of the xprotocol most of the data that's transferred are:
- statement text (strings)
- resultset data (strings, numbers, ...)
which both should be nicely compressible.
CPU latency vs. network latency trade-off
Compression uses CPU time to encode the data and to decode the data. Sending less data over the network allows to reduce network utilization and therefore the extra latency added due to overloaded network.
Compression algorithms
While zlib/deflate have been the de-facto compression algorithm in network protocols, in the recent years new algorithms where developed that compress faster, compress better and/or are more CPU efficient. To support add new algorithms over time, the server should announce which algorithms it can understand.
Clients may support zero or more of the servers compression algorithms and should select one that fits the users requirements best.
- DR1
- make compression algorithm negotiable
Impact on middleboxes
(middleboxes like MySQL Router, ...)
Middleboxes like Routers/Load-Balancers need to track the state of the protocol flow to make decisions to which backend should be sent and when the connection enters a bad state. That's done by tracking the message headers and error messages and notices.
If a middlebox must decompress all messages to be able to track state they may get CPU bound and overloaded and slow down the communication between client and server.
- DR2
- middleboxes MUST be able to read message headers without noticeable overhead if the data is compressed.
Security Considerations
- DR3
- To prevent DoS each side MUST apply limits on the uncompressed payload before decompression.
Functional requirements
- FR1
- a client MUST be able to query the compression algorithms a server supports
an admin MUST be able to limit servers supported compression algorithms
a client MUST be able to choose one compression algorithm of the supported algorithms in capability negotiation phase (excluding reseted sessions)
server MUST enable compression after successful authentication in case when compression algorithm was set successful
a server MUST return an error when client tries to set unsupported compression algorithm
a server MUST allow multiple changes to compression algorithm in capability negotiation phase (last change is used)
- FR2
- a client MUST be able to control how and if uncompressed messages may be combined into a compressed message.
- FR3
- a client MUST be able to enable compression on connection after successful authentication in case when compression algorithm was set successful
client and server MUST manage a client-incoming compression context
client and server MUST manage a client-outgoing compression context
a server MUST report an error and disconnect after decompression error
a client MUST disconnect after it detect decompression error
a server MUST decompress each compressed message to keep compression context in sync on both sides even if expectation block in X Protocol is skipped
a server MAY consider to compress following X Protocol messages ColumnMetaData, Row, FetchDone, FetchSuspended, FetchMoreResultssets, FetchDoneMoreOutParams, Notice
- FR4
- a client MUST be able to query compression statistics
- FR5
- a client MUST be able to try to enable compression and accept a connection when server doesn't support compression.
Protocol
Negotiation
Compression needs to be negotiated between client and server to agree:
- compression algorithm
- combining of messages
- max uncompressed messages per compressed message
Compression must be configured before session setup, at capability negotiation phase and is enabled after successful authentication.
User may check which compression algorithms are available by sending:
client->server: Mysqlx.Connection.CapabilitiesGet()
client<-server: Mysqlx.Connection.Capabilities(....)
To enable compression a client must set the compression configuration:
client->server: Mysqlx.Connection.CapabilitiesSet(...new configuration...)
client<-server: Mysqlx.Ok()
compression
- type
- object
- fields
-
- algorithm
- server_combine_mixed_messages
- server_max_combine_messages
algorithm (GetCapability)
- type
- array-of-strings
- description
- names of supported compression algorithms
algorithm (SetCapability)
- type
- string
- description
- compression algorithm to use after session is estabilished.
server_combine_mixed_messages (SetCapability)
- type
- bool
- description
- if true, server is allowed to combine different message types into a compressed message
- default
- true
server_max_combine_messages (SetCapability)
- type
- integer
- description
- if set, the server MUST not store more than N uncompressed messages into a compressed message
- default
- no limit
Example Message Flow
client->server: Mysqlx.Connection.CapabilitiesGet()
client<-server: Mysqlx.Connection.Capabilities(
"compression":{"algorithm": [ "deflate_stream", "lz4_message"]}, ...)
Clients can ask the server to enable compression by using "CapabilitiesSet" with key "compression.algorithm" and a string value.
- Connection.CapabilitiesSet({ "compression":{"algorithm": "deflate_stream",...}, ...)
-
returns
Okand use "Compression Algorithm" forCompressedmessage after authentication orError
If a client does not set a "compression":{"algorithm":...}, then compression neither client nor server will use Compressed messages.
Compression options can be changed multiple times, where last data from the last
Mysqlx.Connection.CapabilitiesSet will be used:
client->server: Mysqlx.Connection.CapabilitiesSet("compression":{"algorithm": "deflate_stream",..})
client<-server: Mysqlx.Ok
client->server: Mysqlx.Connection.CapabilitiesSet("compression":{"algorithm": "lz4_message",..})
client<-server: Mysqlx.Ok
Compressed Message
Once compression has been negotiated client and server can send compressed messages.
Compressed messages are announced in a xproto frame as:
// mysqlx.proto
message ClientMessages {
enum Type {
// ...
COMPRESSED = 46;
}
}
message ServerMessages {
enum Type {
// ...
COMPRESSED = 19;
}
}
Both client and server wrap a compressed payload in a Compressed message:
// mysqlx_connection.proto
message Compressed {
optional uint64 uncompressed_size = 1;
optional Mysqlx.ServerMessages.Type server_messages = 2;
optional Mysqlx.ClientMessages.Type client_messages = 3;
required bytes payload = 4;
}
- uncompressed_size
- size of the uncompressed payload that is passed to the compressor
- server_messages
-
set by server to message-type of the uncompressed message if only a single uncompressed message type is contained in
payload. - client_messages
-
set by client to message-type of the uncompressed message if only a single uncompressed message type is contained in
payload. - payload
- output of the compression algorithm.
Optimized Encoding
Protobuf doesn't enforce a order of fields in serialized message.
But, a sender of compressed message SHOULD send uncompressed size before payload to allow the receiver to implement a streaming decode where
uncompressed size and the initial part of the payload are passed to the decompressor before the whole payload is received.
Example Implementation using libprotobuf with C++
void send_compressed_payload(const Mysqlx::ServerMessages::Type msg_type,
const int uncompressed_size, const std::string &compressed_payload) {
std::string output;
google::protobuf::StringOutputStream string_zero_stream(&output)
google::protobuf::CodedOutputStream ostream(&string_zero_stream);
Mysqlx::Compression compression_first_fields;
Mysqlx::Compression compression_payload;
compression_first_fields.server_messages = msg_type;
compression_first_fields.uncompressed_size = uncompressed_size;
compression_payload.set_payload(compressed_payload);
// use SerializePartial to encode the "first_fields" before the "payload"
compression_first_fields.SerializePartialToCodedStream(&ostream);
compression_payload.SerializePartialToCodedStream(&ostream);
return output;
}
Compression
The compressor receives up to server_max_combine_messages
xproto frames:
length = 4 byte;
msg_type = 1 byte;
payload = *<length>;
msg = length + msg_type + payload;
msgs = msg msg*;
payload = compressor(msgs);
Compression Algorithm: lz4_message
LZ4 is a fast compression algorithm.
The payload of the Compressed message consists of:
- a
lz4_frameas described in https://github.com/lz4/lz4/blob/master/doc/lz4_Frame_format.md - Compression context is reset after each compressed message.
Compression Algorithm: deflate_stream
deflate is the compression algorithm from zlib, PNG, ... and is widely available.
The payload of the Compressed message is built in following way:
The deflate's compression context is kept alive over all messages to improve compression over time.
Note: each X Protocol compressed frame must be complete. Data from the zlib
layer must be flushed before serializing them to X Protocol frame.
Maximum message size
The size of messages in xprotocol is limited to value of 'Mysqlx_max_allowed_packet' plugin variable.
The value applies to both:
Compressedmessages- messages after uncompression
When a single message extracted from compressed message is
larger than 'Mysqlx_max_allowed_packet' then decompression must be stopped and
X Plugin must return ER_X_DECOMPRESSION_FAILED error and close the connection.
Compression and message type
The server will not compress control flow messages like:
Mysqlx.ErrorMysqlx.Sql.StmtExecuteOkMysqlx.Ok.- global
Mysqlx.Notice.Frame
to make it easier for middleware to track the protocol without having to uncompress the whole stream.
On the other side:
- Mysqlx.Resultset.ColumnMetaData
- Mysqlx.Resultset.Row
- Mysqlx.Resultset.FetchDone
- Mysqlx.Resultset.FetchSuspended
- Mysqlx.Resultset.FetchDoneMoreResultsets
- Mysqlx.Resultset.FetchDoneMoreOutParams
- Mysqlx.Notice.Frame (only local notices)
MAY be compressed.
Instrumentation
Status variables
Monitoring of compressed data by the server
mysqlx_bytes_sent_compressed_payload- data after compression which were send to the wire, one thing needs to be taken into concern. mysqlx_bytes_sent consist of mysqlx_bytes_sent_compressed_payload and X Protocol data that were not compressed like X Protocol headers, X Protocol uncompressed messages.Property Value Name Mysqlx_bytes_sent_compressed_payload Type LONGLONG Scope SESSIONandGLOBALDefault 0 mysqlx_bytes_sent_uncompressed_frame- data before compression which were compressed tomysqlx_bytes_sent_compressed_payloadbytes.Efficiency of compression algorithm can be checked by looking at the compression ratio:
mysqlx_bytes_sent_uncompressed_frame / mysqlx_bytes_sent_compressed_payloadEfficiency of compression on X Protocol can be checked by looking at the compression following:
(mysqlx_bytes_sent - mysqlx_bytes_sent_compressed_payload + mysqlx_bytes_sent_uncompressed_frame) / mysqlx_bytes_sentProperty Value Name mysqlx_bytes_sent_uncompressed_frame Type LONGLONG Scope SESSIONandGLOBALDefault 0
Monitoring of uncompressed data by the server
mysqlx_bytes_received_compressed_payload- compressed data received on the wire (is a component of mysqlx_bytes_received)Property Value Name mysqlx_bytes_received_compressed_payload Type LONGLONG Scope SESSIONandGLOBALDefault 0 mysqlx_bytes_received_uncompressed_frame - compressed data (mysqlx_bytes_received_compressed_payload) were uncompressed into mysqlx_bytes_received_uncompressed_frame bytes
Property Value Name mysqlx_bytes_received_uncompressed_frame Type LONGLONG Scope SESSIONandGLOBALDefault 0
Errors and Warnings
When client sends a compressed frame, but no compression is negotiated with the xplugin, it returns:
client->server: Compressed client<-server: Error(ER_X_FRAME_COMPRESSION_DISABLED)Property Value Name ER_X_FRAME_COMPRESSION_DISABLED Error code 5170 Error text Client didn't enable%s compression. When client sends compressed a frame and decompression fails in the xplugin, it returns:
client->server:Compressed client<-server:Error(ER_X_DECOMPRESSION_FAILED)Property Value Name ER_X_DECOMPRESSION_FAILED Error code 5171 Error text Payload decompression failed When client sends a compressed frame and X Plugin can't read a proper compression header or there are more or less data in the frame, in comparison to sizes presented in headers, it returns:
client->server: Compressed client<-server: Error(ER_X_BAD_COMPRESSED_FRAME)Property Value Name ER_X_BAD_COMPRESSED_FRAME Error code 5174 Error text Payload decompression failed When client tries to negotiate compression and uses an unsupported or disabled compression algorithm:
client->server: CapabilitySet("compression":{"algorithm":"unsupported"...}) client<-server: Error(ER_X_CAPABILITY_COMPRESSION_INVALID_ALGORITHM)Property Value Name ER_X_CAPABILITY_COMPRESSION_INVALID_ALGORITHM Error code 5175 Error text Invalid or unsupported value for 'compression.algorithm' When client tries to negotiate compression and uses a unsupported option:
client->server: CapabilitySet("compression":{"unsupported_field":"..."...}) client<-server: Error(ER_X_CAPABILITY_COMPRESSION_INVALID_OPTION)Property Value Name ER_X_CAPABILITY_COMPRESSION_INVALID_OPTION Error code 5178 Error text Invalid or unsupported option for 'compression' When client tries to negotiate compression and uses compression without required fields. Required is "compression.algorithm":
client->server: CapabilitySet("compression":{}) client<-server: Error(ER_X_CAPABILITY_COMPRESSION_MISSING_REQUIRED_FIELDS)Property Value Name ER_X_CAPABILITY_COMPRESSION_MISSING_REQUIRED_FIELDS Error code 5179 Error text The algorithm is required for 'compression'
Configuration
mysqlx_compression_algorithms- using this variable, admin may limit supported compression algorithmsclient->server: CapabilitiesGet client->server: Capabilities(compression:{algorithm:["deflate_message", "lz4_frame"]}) ... admin executes SET @@mysqlx_compression_algorithms="deflate_message"... client->server: CapabilitiesGet client->server: Capabilities(compression:{algorithm=["deflate_message"]})Property Value Name mysqlx_compression_algorithms Type SET("deflate_message", "lz4_frame") Scope Global Default 'deflate_message,lz4_frame'