X Protocol traffic compression is available on MySQL Server since version 8.0.19. A connector that also supports compression on its end can leverage this feature and reduce the byte streams that are exchanged with the Server.

By default, connections to a MySQL server are uncompressed, thus permitting exchanging data with a client or connector that doesn’t support compression. However, given a client or connector that also supports compression, it is recommended that client and server negotiate the connection compression by default. If this negotiation concludes successfully, both ends can then compress the data they send.

Compression at this level allows reducing the amount of bytes exchanged over the network, but at the cost of additional CPU resources required to run data inflate and deflate operations. The benefits of compression, therefore, occur primarily on low network bandwidth. One can assess the gain or loss due to the compression only after measuring properly the average traffic sizes for a period of time for both the compressed and uncompressed connections.

Connector/J version 8.0.20 came out with basic support for traffic compression over X Protocol connections. For this purpose, a new connection option: xdevapi.compression, was introduced. As of Connector/J 8.0.22, this feature was leveled up with the introduction of two additional connection options: xdevapi.compression-extensions and xdevapi.compression-algorithms.

Compression Negotiation

Enabling compression starts with a negotiation phase between the client and server when establishing new connections. The server initiates the negotiation by telling the connector what compression algorithms it supports. The connector then chooses one it also supports and replies its choice to the server.

As long as the negotiation succeeds, both ends can, optionally, start compressing X Protocol messages. Some control-flow messages never get compressed—because they are too small or because of they cannot be compressed due to the need for backward compatibility with clients and servers not supporting compression; also, data messages smaller than a predefined threshold (250 bytes in the case of Connector/J) also don’t get compressed.

Depending on how the connection compression is set up, a failed compression algorithm negotiation may end with an exception being thrown to the client application or fallback to a compression-disabled connection.

Compression Algorithms

Currently, MySQL, through the X Plugin, supports the compression algorithms Deflate, LZ4 and zstd. These algorithms are referred to by an identifier composed of two parts—the algorithm name and an operation mode:

• deflate_stream
• lz4_message
• zstd_stream

The operation mode, stream or message, determines how the deflater and inflater procedures operate. In the stream mode, all X Protocol messages from a single connection are compressed into a continuous flow (stream) and their decompression must be done the same manner—following the order they were compressed and without skipping any message. In the message mode, each message is compressed individually and independently, so that the order by which the messages are decompressed needs not be the same as the order they were compressed. Also, the message mode does not require all compressed messages to be decompressed.

By default, Connector/J negotiates a compression algorithm following the priority recommended by X DevAPI: first zstd_stream, then lz4_message, and finally deflate_stream.

In all Connector/J options where a compression algorithm identifier can be specified , the aliases “zstd”, “lz4”, and “deflate” can be used instead of “zstd_stream”, “lz4_message”, and “deflate_stream”, respectively.

Default Compression

Note that Connector/J provides out-of-the-box support for Deflate only. This is so because none of the other compression algorithms are natively supported by the existing JREs. The other algorithms can still be used though, as explained below.

Given that the server has X Protocol compression enabled, Connector/J by default tries to negotiate connection compression in all X DevAPI sessions. In the negotiation, Connector/J and the server attempt to mutually agree on the compression algorithm deflate_stream; and if the negotiation fails, they fall back on non-compressed connections. This behavior corresponds to the connection option xdevapi.compression=preferred available since Connector/J 8.0.20.

Besides preferred, the connection option xdevapi.compression takes two other values: disabled and required. These options, respectively, disable compression altogether (by not even starting any negotiation efforts), or enforce compression and throw an exception in case of an unsuccessful negotiation.

Setting up Additional Compression Algorithms

In order for Connector/J to support the compression algorithms LZ4 and zstd, the client application must provide some implementation for the corresponding deflate and inflate operations in the form of an OutputStream and an InputStream object, respectively. The easiest way to accomplish this is by using a third-party library such as the Apache Commons Compress™ library, which supports both algorithms.

The connection option xdevapi.compression-extensions, introduced in Connector/J 8.0.22, allows configuring Connector/J to use any compression algorithm that is supported by MySQL Server, as long as there is a Java implementation for that algorithm. The option takes a list of triplets separated by commas. Each triplet in turn contains the following elements, delimited by colon:

1. The compression algorithm name, indicated by the identifier used by the server.
2. A fully-qualified class name of a class implementing the interface java.io.InputStream that will be used to inflate data compressed with the named algorithm.
3. A fully-qualified class name of a class implementing the interface java.io.OutputStream that will be used to deflate data using the named algorithm.

Here’s an example that sets up the support for the algorithms lz4_message and zstd_stream using the Apache Commons Compress™ library:

String connStr = "jdbc:mysql://johndoe:secret@localhost:33060/mydb?"
  + "xdevapi.compression-algorithm="
  + "lz4_message:" // LZ4 triplet
  + FramedLZ4CompressorInputStream.class.getName() + ":"
  + FramedLZ4CompressorOutputStream.class.getName() + ","
  + "zstd_stream:" // zstd triplet
  + ZstdCompressorInputStream.class.getName() + ":"
  + ZstdCompressorOutputStream.class.getName();

SessionFactory sessFact = new SessionFactory();
Session sess = sessFact.getSession(connStr);
Collection col = sess.getDefaultSchema().getCollection("myCollection");

// (...)

sess.close();

With this connection string, Connector/J negotiates the compression following the recommended order: first zstd, next LZ4, and finally Deflate. The final choice depends on what algorithms are enabled on the server. As compression is not required by default, if the negotiation fails, the connection won’t be compressed, but the client will still be able to communicate with the server.

Influencing Compression Negotiation

Assuming Connector/J has been configured with extensions that enable additional compression algorithms through setting the connection option described in the previous section, the algorithms and the order by which they are attempted in the negotiation phase can be customized by setting the third compression related connection option—xdevapi.compression-algorithms.

The connection option xdevapi.compression-algorithms takes a list of compression algorithms identifiers and is used to override the default order by which the compression algorithm is negotiated. Only the enumerated algorithms are attempted during the negotiation, thus, it also allows to specify which algorithms to enable from the client side. Note how this works similarly to the server global variable mysqlx_compression_algorithms (also available as a command line option), except that, if configured from the client side, different X DevAPI sessions can be set with different compression algorithms while, if configured from the server side, all sessions are forced to negotiate for the same compression algorithms.

Unknown algorithms specified in the option xdevapi.compression-algorithms are simply discarded. However, if such algorithms are also configured in the connection option xdevapi.compression-extensions, then they become options during the compression negotiation, as long as the server also supports such algorithms, then they can be effectively used, thus making this a fully pluggable feature in Connector/J.

Disabling compression can be done from the client side, via the connection option xdevapi.compression=disabled, or from the server side, by setting the server variable mysqlx_compression_algorithms to an empty string. It is, however, not possible to completely disallow uncompressed connections from the server side.

Profiling Compression Efficiency

Connector/J does not provide any metrics regarding ratio of compressed vs. uncompressed payloads. However, some metrics can be obtained from the server through its status variables Mysqlx_bytes_received, Mysqlx_bytes_received_compressed_payload, Mysqlx_bytes_received_uncompressed_frame, Mysqlx_bytes_sent, Mysqlx_bytes_sent_compressed_payload, and Mysqlx_bytes_sent_uncompressed_frame.

Typically the compression ratio obtained from the client side can be calculated using the following formula:

Mysqlx_bytes_received_uncompressed_frame / Mysqlx_bytes_received_compressed_payload

And the efficiency of compression for X Protocol messages sent by the connector can be calculated using the following formula:

(Mysqlx_bytes_received - Mysqlx_bytes_received_uncompressed_frame + Mysqlx_bytes_received_uncompressed_frame) / Mysqlx_bytes_received

Similar calculations can be made for data compressed on the server side, as documented in the MySQL Reference Manual.

Additional Information and Resources

• Connection Compression Using X DevAPI
• Connection Compression with X Plugin
• mysqlx_compression_algorithms
• Apache Commons Compress™
• Zstd-jni

I invite you to join the MySQL Community on Slack and hang out in the #connectors channel: