Motivation

Latency of requests on networks depends on:

• networks RTT
• utilization of the network

While the networks RTT can't be changed, the utilization of the network can be changed by compressing the packets to reduce the traffic.

In context of the xprotocol most of the data that's transferred are:

• statement text (strings)
• resultset data (strings, numbers, ...)

which both should be nicely compressible.

CPU latency vs. network latency trade-off

Compression uses CPU time to encode the data and to decode the data. Sending less data over the network allows to reduce network utilization and therefore the extra latency added due to overloaded network.

Compression algorithms

While zlib/deflate have been the de-facto compression algorithm in network protocols, in the recent years new algorithms where developed that compress faster, compress better and/or are more CPU efficient. To support add new algorithms over time, the server should announce which algorithms it can understand.

Clients may support zero or more of the servers compression algorithms and should select one that fits the users requirements best.

DR1

make compression algorithm negotiable

Impact on middleboxes

(middleboxes like MySQL Router, ...)

Middleboxes like Routers/Load-Balancers need to track the state of the protocol flow to make decisions to which backend should be sent and when the connection enters a bad state. That's done by tracking the message headers and error messages and notices.

If a middlebox must decompress all messages to be able to track state they may get CPU bound and overloaded and slow down the communication between client and server.

DR2

middleboxes MUST be able to read message headers without noticeable overhead if the data is compressed.

Security Considerations

DR3

To prevent DoS each side MUST apply limits on the uncompressed payload before decompression.

Functional requirements

FR1

a client MUST be able to query the compression algorithms a server supports

1. an admin MUST be able to limit servers supported compression algorithms

2. a client MUST be able to choose one compression algorithm of the supported algorithms in capability negotiation phase (excluding reseted sessions)

3. server MUST enable compression after successful authentication in case when compression algorithm was set successful

4. a server MUST return an error when client tries to set unsupported compression algorithm

5. a server MUST allow multiple changes to compression algorithm in capability negotiation phase (last change is used)

FR2

a client MUST be able to control how and if uncompressed messages may be combined into a compressed message.

FR3

a client MUST be able to enable compression on connection after successful authentication in case when compression algorithm was set successful

1. client and server MUST manage a client-incoming compression context

2. client and server MUST manage a client-outgoing compression context

3. a server MUST report an error and disconnect after decompression error

4. a client MUST disconnect after it detect decompression error

5. a server MUST decompress each compressed message to keep compression context in sync on both sides even if expectation block in X Protocol is skipped

6. a server MAY consider to compress following X Protocol messages ColumnMetaData, Row, FetchDone, FetchSuspended, FetchMoreResultssets, FetchDoneMoreOutParams, Notice

FR4

a client MUST be able to query compression statistics

FR5

a client MUST be able to try to enable compression and accept a connection when server doesn't support compression.

Protocol

Negotiation

Compression needs to be negotiated between client and server to agree:

• compression algorithm
• combining of messages
• max uncompressed messages per compressed message

Compression must be configured before session setup, at capability negotiation phase and is enabled after successful authentication.

User may check which compression algorithms are available by sending:

client->server: Mysqlx.Connection.CapabilitiesGet()
client<-server: Mysqlx.Connection.Capabilities(....)

To enable compression a client must set the compression configuration:

client->server: Mysqlx.Connection.CapabilitiesSet(...new configuration...)
client<-server: Mysqlx.Ok()

compression

type

object

fields

• algorithm
• server_combine_mixed_messages
• server_max_combine_messages

algorithm (GetCapability)

type

array-of-strings

description

names of supported compression algorithms

algorithm (SetCapability)

type

string

description

compression algorithm to use after session is estabilished.

server_combine_mixed_messages (SetCapability)

type

bool

description

if true, server is allowed to combine different message types into a compressed message

default

true

server_max_combine_messages (SetCapability)

type

integer

description

if set, the server MUST not store more than N uncompressed messages into a compressed message

default

no limit

Example Message Flow

client->server: Mysqlx.Connection.CapabilitiesGet()
client<-server: Mysqlx.Connection.Capabilities(
    "compression":{"algorithm": [ "deflate_stream", "lz4_message"]}, ...)

Clients can ask the server to enable compression by using "CapabilitiesSet" with key "compression.algorithm" and a string value.

Connection.CapabilitiesSet({ "compression":{"algorithm": "deflate_stream",...}, ...)

returns Ok and use "Compression Algorithm" for Compressed message after authentication or Error

If a client does not set a "compression":{"algorithm":...}, then compression neither client nor server will use Compressed messages.

Compression options can be changed multiple times, where last data from the last Mysqlx.Connection.CapabilitiesSet will be used:

client->server: Mysqlx.Connection.CapabilitiesSet("compression":{"algorithm": "deflate_stream",..})
client<-server: Mysqlx.Ok
client->server: Mysqlx.Connection.CapabilitiesSet("compression":{"algorithm": "lz4_message",..})
client<-server: Mysqlx.Ok

Compressed Message

Once compression has been negotiated client and server can send compressed messages.

Compressed messages are announced in a xproto frame as:

// mysqlx.proto
message ClientMessages {
  enum Type {
    // ...
    COMPRESSED = 46;
  }
}

message ServerMessages {
  enum Type {
    // ...

    COMPRESSED = 19;
  }
}

Both client and server wrap a compressed payload in a Compressed message:

// mysqlx_connection.proto
message Compressed {
  optional uint64 uncompressed_size = 1;
  optional Mysqlx.ServerMessages.Type server_messages = 2;
  optional Mysqlx.ClientMessages.Type client_messages = 3;
  required bytes payload = 4;
}

uncompressed_size

size of the uncompressed payload that is passed to the compressor

server_messages

set by server to message-type of the uncompressed message if only a single uncompressed message type is contained in payload.

client_messages

set by client to message-type of the uncompressed message if only a single uncompressed message type is contained in payload.

payload

output of the compression algorithm.

Optimized Encoding

Protobuf doesn't enforce a order of fields in serialized message.

But, a sender of compressed message SHOULD send uncompressed size before payload to allow the receiver to implement a streaming decode where uncompressed size and the initial part of the payload are passed to the decompressor before the whole payload is received.

Example Implementation using libprotobuf with C++

void send_compressed_payload(const Mysqlx::ServerMessages::Type msg_type,     
    const int uncompressed_size, const std::string &compressed_payload) {
  std::string output;
  google::protobuf::StringOutputStream string_zero_stream(&output)
  google::protobuf::CodedOutputStream ostream(&string_zero_stream);
  Mysqlx::Compression compression_first_fields;
  Mysqlx::Compression compression_payload;

  compression_first_fields.server_messages = msg_type;
  compression_first_fields.uncompressed_size = uncompressed_size;

  compression_payload.set_payload(compressed_payload);

  // use SerializePartial to encode the "first_fields" before the "payload"  
  compression_first_fields.SerializePartialToCodedStream(&ostream);
  compression_payload.SerializePartialToCodedStream(&ostream);

  return output;
}

Compression

The compressor receives up to server_max_combine_messages xproto frames:

length   = 4 byte;
msg_type = 1 byte;
payload  = *<length>;

msg      = length + msg_type + payload;
msgs     = msg msg*;
payload  = compressor(msgs);

Compression Algorithm: lz4_message

LZ4 is a fast compression algorithm.

The payload of the Compressed message consists of:

• a lz4_frame as described in https://github.com/lz4/lz4/blob/master/doc/lz4_Frame_format.md
• Compression context is reset after each compressed message.

Compression Algorithm: deflate_stream

deflate is the compression algorithm from zlib, PNG, ... and is widely available.

The payload of the Compressed message is built in following way:

The deflate's compression context is kept alive over all messages to improve compression over time.

Note: each X Protocol compressed frame must be complete. Data from the zlib layer must be flushed before serializing them to X Protocol frame.

Maximum message size

The size of messages in xprotocol is limited to value of 'Mysqlx_max_allowed_packet' plugin variable.

The value applies to both:

• Compressed messages
• messages after uncompression

When a single message extracted from compressed message is larger than 'Mysqlx_max_allowed_packet' then decompression must be stopped and X Plugin must return ER_X_DECOMPRESSION_FAILED error and close the connection.

Compression and message type

The server will not compress control flow messages like:

• Mysqlx.Error
• Mysqlx.Sql.StmtExecuteOk
• Mysqlx.Ok.
• global Mysqlx.Notice.Frame

to make it easier for middleware to track the protocol without having to uncompress the whole stream.

On the other side:

• Mysqlx.Resultset.ColumnMetaData
• Mysqlx.Resultset.Row
• Mysqlx.Resultset.FetchDone
• Mysqlx.Resultset.FetchSuspended
• Mysqlx.Resultset.FetchDoneMoreResultsets
• Mysqlx.Resultset.FetchDoneMoreOutParams
• Mysqlx.Notice.Frame (only local notices)

MAY be compressed.

Instrumentation

Status variables

Monitoring of compressed data by the server

• mysqlx_bytes_sent_compressed_payload - data after compression which were send to the wire, one thing needs to be taken into concern. mysqlx_bytes_sent consist of mysqlx_bytes_sent_compressed_payload and X Protocol data that were not compressed like X Protocol headers, X Protocol uncompressed messages.

  Property	Value
  Name	Mysqlx_bytes_sent_compressed_payload
  Type	LONGLONG
  Scope	SESSION and GLOBAL
  Default	0

• mysqlx_bytes_sent_uncompressed_frame - data before compression which were compressed to mysqlx_bytes_sent_compressed_payload bytes.

  Efficiency of compression algorithm can be checked by looking at the compression ratio:

  mysqlx_bytes_sent_uncompressed_frame / mysqlx_bytes_sent_compressed_payload
  

  Efficiency of compression on X Protocol can be checked by looking at the compression following:

  (mysqlx_bytes_sent - mysqlx_bytes_sent_compressed_payload + mysqlx_bytes_sent_uncompressed_frame) / mysqlx_bytes_sent
  

  Property	Value
  Name	mysqlx_bytes_sent_uncompressed_frame
  Type	LONGLONG
  Scope	SESSION and GLOBAL
  Default	0

Monitoring of uncompressed data by the server

• mysqlx_bytes_received_compressed_payload - compressed data received on the wire (is a component of mysqlx_bytes_received)

  Property	Value
  Name	mysqlx_bytes_received_compressed_payload
  Type	LONGLONG
  Scope	SESSION and GLOBAL
  Default	0

• mysqlx_bytes_received_uncompressed_frame - compressed data (mysqlx_bytes_received_compressed_payload) were uncompressed into mysqlx_bytes_received_uncompressed_frame bytes

  Property	Value
  Name	mysqlx_bytes_received_uncompressed_frame
  Type	LONGLONG
  Scope	SESSION and GLOBAL
  Default	0

Errors and Warnings

• When client sends a compressed frame, but no compression is negotiated with the xplugin, it returns:

  client->server: Compressed
  client<-server: Error(ER_X_FRAME_COMPRESSION_DISABLED)
  

  Property	Value
  Name	ER_X_FRAME_COMPRESSION_DISABLED
  Error code	5170
  Error text	Client didn't enable%s compression.

• When client sends compressed a frame and decompression fails in the xplugin, it returns:

  client->server:Compressed
  client<-server:Error(ER_X_DECOMPRESSION_FAILED)
  

  Property	Value
  Name	ER_X_DECOMPRESSION_FAILED
  Error code	5171
  Error text	Payload decompression failed

• When client sends a compressed frame and X Plugin can't read a proper compression header or there are more or less data in the frame, in comparison to sizes presented in headers, it returns:

  client->server: Compressed
  client<-server: Error(ER_X_BAD_COMPRESSED_FRAME)
  

  Property	Value
  Name	ER_X_BAD_COMPRESSED_FRAME
  Error code	5174
  Error text	Payload decompression failed

• When client tries to negotiate compression and uses an unsupported or disabled compression algorithm:

  client->server: CapabilitySet("compression":{"algorithm":"unsupported"...})
  client<-server: Error(ER_X_CAPABILITY_COMPRESSION_INVALID_ALGORITHM)
  

  Property	Value
  Name	ER_X_CAPABILITY_COMPRESSION_INVALID_ALGORITHM
  Error code	5175
  Error text	Invalid or unsupported value for 'compression.algorithm'

• When client tries to negotiate compression and uses a unsupported option:

  client->server: CapabilitySet("compression":{"unsupported_field":"..."...})
  client<-server: Error(ER_X_CAPABILITY_COMPRESSION_INVALID_OPTION)
  

  Property	Value
  Name	ER_X_CAPABILITY_COMPRESSION_INVALID_OPTION
  Error code	5178
  Error text	Invalid or unsupported option for 'compression'

• When client tries to negotiate compression and uses compression without required fields. Required is "compression.algorithm":

  client->server: CapabilitySet("compression":{})
  client<-server: Error(ER_X_CAPABILITY_COMPRESSION_MISSING_REQUIRED_FIELDS)
  

  Property	Value
  Name	ER_X_CAPABILITY_COMPRESSION_MISSING_REQUIRED_FIELDS
  Error code	5179
  Error text	The algorithm is required for 'compression'

Configuration

• mysqlx_compression_algorithms - using this variable, admin may limit supported compression algorithms

  client->server: CapabilitiesGet
  client->server: Capabilities(compression:{algorithm:["deflate_message", "lz4_frame"]})
  ... admin executes SET @@mysqlx_compression_algorithms="deflate_message"...
  client->server: CapabilitiesGet
  client->server: Capabilities(compression:{algorithm=["deflate_message"]})
  

  Property	Value
  Name	mysqlx_compression_algorithms
  Type	SET("deflate_message", "lz4_frame")
  Scope	Global
  Default	'deflate_message,lz4_frame'