Motivation

WL#9252 and WL#13442 introduce compression to X Protocol.

User Stories

1. As a DBA, I want to control how much time/memory is used to compress connection data to optimize the transactions per second over a network connection.
2. As a low bandwidth client, I want the server to spend more time in compressing data.

Compression Levels

Compression algorithms allow to control the effort to compress the data which impacts the compression time and compression ratio in no-linear ways by setting the compression level:

• doubling the compression-time, only slightly improves the compression ratio,
• more time spent to compress increases the CPU utilization (and memory usage) itself and impacts the QPS of the server/client.

Design Requirement

The server MUST have a way to set a default compression level.

Client-side compression-level

Another factor which has an impact on QPS is network latency. The time necessary to transfer data can be decreased by increasing the compression ratio of transferred data. In this way, the compression level can be used to tradeoff cpu/network latency and make it selectable by the user.

Design Requirement

The client MUST have a way announce a preferred compression level.

Security Considerations

If the client can select any compression level, it may trigger:

• DoS due to high CPU usage
• DoS due to high Memory usage

Design Requirement

The server MUST have a way to limit the client-selectable compression level to a acceptable range.

Goal

• Allow client and server operators to specify default compression-levels
• Allow client and server negotiate fast and safe compression-levels

Functional requirements

FR1

a compression level MUST take part in compression algorithm capability negotiation phase

FR2

a server MUST set compression level when compression algorithm was set successfully

FR3

a server MUST be able to use a default value of compression level in a case when its value is not explicitly specified

FR4

a server MUST be able to limit maximal value of compression level to prevent too high resource usage

FR5

a server MUST use the nearest supported value in case the specified compression level is out of range

FR6

a server MUST return an error when a client tries to set compression level as unsupported type value; integer is supported only

FR7

a client MUST be able to set a preferred compression level for data sent from server to client

Protocol

Negotiation

Compression capabilities are negotiated between the client and server and set to agree on:

• compression algorithm
• combining of messages
• max uncompressed messages per compressed message
• compression level

Compression level must be configured before session setup, at a capability negotiation phase and is enabled after successful authentication. This capability negotiation phase is an existing solution but should be extended by compression level capability.

compression

type

object

fields

• algorithm (existing)
• server_combine_mixed_messages (existing)
• server_max_combine_messages (existing)
• level (new feature)

level (SetCapability)

type

integer

description

if set, the server MUST compress messages using a specified algorithm and specified compression level; if the value of level is not supported by the chosen algorithm the value is adjusted to the nearest supported value; if not set, the server MUST use a default value for a specific algorithm

Acceptable Range of level

The acceptable range for the level depends on the compression-method and the library version.

The level is a range of always increasing values: 1 is faster (or equal) then 2.

Note: If the level would map to the libraries "default value" MUST be mapped nearest supported value instead.

The min and max for each level are determined at runtime.

Ranges of compression level

deflate_stream

1 - 9

lz4_message

0 - 16

zstd_stream

-(a lot) - -1 and 1 - 22 (0 is adjusted to 1),
or 3 if zstd library ver. < 1.4.0, due to inefficient use of memory for stream compression

Background

acceptable of zlib 1.2.11 (used for deflate_stream)

• -1-9
• -1 means use library default (6)
• 0 means no compression
• 1-9

acceptable range by liblz4 1.7.1 (used for lz4_stream)

• 0-16

acceptable range by zstd 1.4.3 (used for zstd_stream)

• -(a lot)-22
• 0 means use library default (3)
• negative values are faster than 1

Default value of level

deflate_stream

3

lz4_message

2

zstd_stream

3

The default values where determined through performance testing and represent a good trade-off between compression-time and network-time due to less bytes being sent.

Example Message Flow

Clients can ask the server to enable compression by using "CapabilitiesSet" with "compression.algorithm" and "compression.level".

Connection.CapabilitiesSet({ "compression":{"algorithm": "deflate_stream", "level": 3, ...}, ...)

returns Ok and use "Compression Algorithm" for Compressed message after authentication or Error

If a client does not set a "compression":{"level":...}, then compression level will set to a default value and will use Compressed messages.

Compression options can be changed multiple times, where last data from the last Mysqlx.Connection.CapabilitiesSet will be used:

client->server: Mysqlx.Connection.CapabilitiesSet("compression":{"algorithm": "deflate_stream", "level": 3, ...})
client<-server: Mysqlx.Ok
client->server: Mysqlx.Connection.CapabilitiesSet("compression":{"algorithm": "lz4_message", "level": 5, ...})
client<-server: Mysqlx.Ok

Instrumentation

• Mysqlx_compression_algorithm - the name of the compression algorithm in use for the current connection to the server. Possible values come from global system variable Mysqlx_compression_algorithms.

  Property	Value
  Name	Mysqlx_compression_algorithm
  Type	STRING
  Scope	SESSION
  Default	(empty)

• Mysqlx_compression_level - the compression level in use for the current connection to the server

  Property	Value
  Name	Mysqlx_compression_level
  Type	STRING
  Scope	SESSION
  Default	(empty)

  Errors and Warnings

• When a client tries to negotiate compression and uses compression level encoded in unsupported data type a generic (existing) error has been raised:

  client->server: CapabilitySet("compression":{"level": "5", ....})
  client<-server: Error(ER_X_CAPABILITIES_PREPARE_FAILED)
  

  Property	Value
  Name	ER_X_CAPABILITIES_PREPARE_FAILED
  Error code	5001
  Error text	Capability prepare failed for 'compression'

Configuration

• Mysqlx_deflate_max_client_compression_level - using this variable, an admin may limit the upper bound of accepted compression levels for the deflate algorithm.

  Property	Value
  Name	Mysqlx_deflate_max_client_compression_level
  Type	INT
  Scope	GLOBAL
  Default	5
  Value range	1 - 9

• Mysqlx_lz4_max_client_compression_level - using this variable, an admin may limit the upper bound of accepted compression levels for the lz4 algorithm.

  Property	Value
  Name	Mysqlx_lz4_max_client_compression_level
  Type	INT
  Scope	GLOBAL
  Default	8
  Value range	0 - 16

• Mysqlx_zstd_max_client_compression_level - using this variable, an admin may limit the upper bound of accepted compression levels for the zstd algorithm.

  Property	Value
  Name	Mysqlx_zstd_max_client_compression_level
  Type	INT
  Scope	GLOBAL
  Default	11
  Value range	-131072 - -1 and 1 - 22 (for zstd lib ver. >= 1.4.0)
  Value range	3 (for zstd lib ver. < 1.4.0)

• Mysqlx_deflate_default_compression_level - using this variable, an admin may set the default compression level for the deflate algorithm.

  Property	Value
  Name	Mysqlx_deflate_default_compression_level
  Type	INT
  Scope	GLOBAL
  Default	3
  Value range	1 - 9

• Mysqlx_lz4_default_compression_level - using this variable, an admin may set the default compression level for the lz4 algorithm.

  Property	Value
  Name	Mysqlx_lz4_default_compression_level
  Type	INT
  Scope	GLOBAL
  Default	2
  Value range	0 - 16

• Mysqlx_zstd_default_compression_level - using this variable, an admin may set the default compression level for the zstd algorithm.

  Property	Value
  Name	Mysqlx_zstd_default_compression_level
  Type	INT
  Scope	GLOBAL
  Default	3
  Value range	-131072 - -1 and 1 - 22 (for zstd lib ver. >= 1.4.0)
  Value range	3 (for zstd lib ver. < 1.4.0)