WL#13442: X Protocol - zstd compression

Affects: Server-8.0   —   Status: Complete


WL#9252 added deflate and lz4 compression support to xprotocol.

WL#12039 added zstd support to the Classic protocol, which currently supports deflate and zstd compression algorithms.

  • deflate has a good compression ration, but is low throughput.
  • lz4 has high throughput, but low compression ratio
  • zstd how good throughput, and good compression ratio.

Adding zstd compression to the xprotocol gives the user a good choice over saturated networks with lower CPU costs then deflate, but higher compression ratios than lz4.

Results comparison

  • performance:

    Average - Degradation Algorithm
    Test Style DEFLATE LZ4 ZSTD
    OLTP_RO GROUP -24,68% -2,06% -10% (TBD)
    POINT_SELECT GROUP -7,12% -4,03% -12% (TBD)
    SELECT_RANGES GROUP -42,66% +3,23% -10% (TBD)
  • Protocol compression ratio:

    Average - Ratio Algorithm
    Test Style DEFLATE LZ4 ZSTD
    OLTP_RO GROUP 19.6 8.7 27.9
    POINT_SELECT GROUP 5.4 1.7 4.8
    SELECT_RANGES GROUP 23.3 11 37.9


Give user a possibility to use "modern" compression algorithm, which performance is better than deflate and compression ratio is much better than lz4.

Functional requirements

a client MUST be able to query the ZSTD compression algorithms if server supports it
  1. an admin MUST be able to disable/enable ZSTD compression algorithms on the server

  2. a client MUST be able to choose ZSTD compression algorithm
    in capability negotiation phase (excluding reseted sessions)

  3. server MUST enable ZSTD compression after successful authentication in case when ZSTD compression algorithm was set successful

  4. a server MUST allow multiple changes to compression algorithm in capability negotiation phase (last change is used)

  5. server MUST return an error when client tries to set unsupported compression algorithm

a client MUST be able to control how and if uncompressed messages may be combined into a compressed message.
a client MUST be able to enable compression on connection after successful authentication in case when compression algorithm was set successful
  1. client and server MUST manage a client-incoming compression context

  2. client and server MUST manage a client-outgoing compression context

  3. a server MUST report an error and disconnect after decompression error

  4. a client MUST disconnect after it detect decompression error

  5. a server MUST decompress each compressed message to keep compression context in sync on both sides even if expectation block in X Protocol is skipped

  6. a server MAY consider to compress following X Protocol messages ColumnMetaData, Row, FetchDone, FetchSuspended, FetchMoreResultssets, FetchDoneMoreOutParams, Notice

a client MUST be able to query compression statistics
a client MUST be able to try to enable compression and accept a connection when server doesn't support compression (including ZSTD algorithm)


In WL#9252 X Protocol added support for compression, which can be negotiated by client using connection capabilities.

To add zstd support to xprotocol this WL needs to:

  • implementing the new algorithm: zstd_stream
  • allow zstd_stream for compression.algorithm capability
  • allow zstd_stream for mysqlx_compression_algorithms


Server presents to the client known compression algorithm after receiving CapabilitiesGet protocol message:

client->server: Mysqlx.Connection.CapabilitiesGet()
client<-server: Mysqlx.Connection.Capabilities("compression":{"algorithm": ["deflate_stream", "lz4_message"]}, ...)

With WL#9252 it returns "deflate_stream" and "lz4_message". With zstd_stream added, it should return:

client->server: Mysqlx.Connection.CapabilitiesGet()
client<-server: Mysqlx.Connection.Capabilities("compression":{"algorithm": ["deflate_stream", "lz4_message", "zstd_stream"]}, ...)

Compression algorithm is chosen by "CapabilitiesSet" message, which should allow "zstd_stream" value under "compression.algorithm" field:

client->server: Mysqlx.Connection.CapabilitiesSet("compression",{"algorithm"="zstd_stream",..})
client<-server: Mysqlx.Ok

The "zstd" compression algorithms must be enabled after authentication like other algorithms and must be applied in all compression configurations:

  • "compression.server_combine_mixed_messages"
  • "compression.server_max_combine_messages"

Compression Algorithm: zstd_stream

The payload of the Compressed message contains:

msgs    = ...; // see WL#9252
payload = ZSTD_compressStream2(ctx, msgs.data(), msgs.size(), ...)

The zstd's compression context is kept alive over all messages to improve compression over time.

Note: each X Protocol compressed frame must be complete. Data from the zstd layer must be flushed before serializing them to X Protocol frame.

Maximum message size

The size of messages in xprotocol is limited to value of 'Mysqlx_max_allowed_packet' plugin variable.

The value applies to both:

* Compressed messages
* messages after uncompression

When a single message extracted from compressed message is larger than 'Mysqlx_max_allowed_packet' then decompression must be stopped and X Plugin must return ER_X_DECOMPRESSION_FAILED error and close the connection.

Compressed messages

As WL#9252 defines that everything except messages before authentication and Mysqlx.Ok, Mysqlx.Error, Mysqlx.Sql.StmtExecuteOk might be compressed. Same rules must be applied to "zstd" algorithm.


Status variables

Monitoring of compressed data by the server

Monitoring of compression was implemented in WL#9252, and this worklog should increment same counters for the new compression algorithm. Those status variables are:

  • mysqlx_bytes_sent_compressed_payload
  • mysqlx_bytes_sent_uncompressed_frame
  • mysqlx_bytes_received_compressed_payload
  • mysqlx_bytes_received_uncompressed_frame


  • mysqlx_compression_algorithms - using this variable, admin may limit supported compression algorithms. This variable already exists, it must be extended with new value "zstd" which represents the algorithm.

    client->server: CapabilitiesGet
    client->server: Capabilities(compression:{algorithm:["deflate_stream", "lz4_message",

    "zstd_stream"]}) ... admin executes SET @@mysqlx_compression_algorithms="zstd_stream"... client->server: CapabilitiesGet client->server: Capabilities(compression:{algorithm=["zstd_stream"]})

    Property Value
    Name mysqlx_compression_algorithms
    Type SET("deflate_stream", "lz4_message", "zstd_stream")
    Scope Global
    Default 'deflate_stream,lz4_message,zstd_stream'
    Comment Already existing variable, new "zstd" value added