This section describes how to configure audit logging characteristics, such as the file to which the audit log plugin writes events, the format of written events, whether to enable log file compression and encryption, and space management.
Encryption capabilities described here apply as of MySQL 8.0.17, with the exception of the section that compares current encryption capabilities to the previous more-limited capabilities; see Audit Log File Encryption Prior to MySQL 8.0.17.
For additional information about the functions and system variables that affect audit logging, see Audit Log Functions, and Audit Log Options and Variables.
The audit log plugin can also control which audited events are written to the audit log file, based on event content or the account from which events originate. See Section 8.4.5.7, “Audit Log Filtering”.
To configure the audit log file name, set the
audit_log_file
system
variable at server startup. The default name is
audit.log
in the server data directory.
For best security, write the audit log to a directory
accessible only to the MySQL server and to users with a
legitimate reason to view the log.
The plugin interprets the
audit_log_file
value as
composed of an optional leading directory name, a base name,
and an optional suffix. If compression or encryption are
enabled, the effective file name (the name actually used to
create the log file) differs from the configured file name
because it has additional suffixes:
If compression is enabled, the plugin adds a suffix of
.gz
.If encryption is enabled, the plugin adds a suffix of
.
, wherepwd_id
.encpwd_id
indicates which encryption password to use for log file operations. The audit log plugin stores encryption passwords in the keyring; see Encrypting Audit Log Files.
The effective audit log file name is the name resulting from
the addition of applicable compression and encryption suffixes
to the configured file name. For example, if the configured
audit_log_file
value is
audit.log
, the effective file name is one
of the values shown in the following table.
Enabled Features | Effective File Name |
---|---|
No compression or encryption | audit.log |
Compression | audit.log.gz |
Encryption | audit.log. |
Compression, encryption | audit.log.gz. |
pwd_id
indicates the ID of the
password used to encrypt or decrypt a file.
pwd_id
format is
pwd_timestamp-seq
, where:
pwd_timestamp
is a UTC value in
format indicating when the password was created.YYYYMMDD
Thhmmss
seq
is a sequence number. Sequence numbers start at 1 and increase for passwords that have the samepwd_timestamp
value.
Here are some example pwd_id
password ID values:
20190403T142359-1
20190403T142400-1
20190403T142400-2
To construct the corresponding keyring IDs for storing
passwords in the keyring, the audit log plugin adds a prefix
of audit_log-
to the
pwd_id
values. For the example
password IDs just shown, the corresponding keyring IDs are:
audit_log-20190403T142359-1
audit_log-20190403T142400-1
audit_log-20190403T142400-2
The ID of the password currently used for encryption by the
audit log plugin is the one having the largest
pwd_timestamp
value. If multiple
passwords have that pwd_timestamp
value, the current password ID is the one with the largest
sequence number. For example, in the preceding set of password
IDs, two of them have the largest timestamp,
20190403T142400
, so the current password ID
is the one with the largest sequence number
(2
).
The audit log plugin performs certain actions during initialization and termination based on the effective audit log file name:
During initialization, the plugin checks whether a file with the audit log file name already exists and renames it if so. (In this case, the plugin assumes that the previous server invocation exited unexpectedly with the audit log plugin running.) The plugin then writes to a new empty audit log file.
During termination, the plugin renames the audit log file.
File renaming (whether during plugin initialization or termination) occurs according to the usual rules for automatic size-based log file rotation; see Manual Audit Log File Rotation (Before MySQL 8.0.31).
To configure the audit log file format, set the
audit_log_format
system
variable at server startup. These formats are available:
NEW
: New-style XML format. This is the default.OLD
: Old-style XML format.JSON
: JSON format. Writes the audit log as a JSON array. Only this format supports the optional query time and size statistics, which are available from MySQL 8.0.30.
For details about each format, see Section 8.4.5.4, “Audit Log File Formats”.
Starting in MySQL 8.0.34, MySQL Enterprise Audit provides the capability of
setting a refresh interval to dispose of the in-memory cache
automatically. A flush task configured using the
audit_log_flush_interval_seconds
system variable has a value of zero by default, which means
the task is not scheduled to run.
When the task is configured to run (the value is non-zero), MySQL Enterprise Audit attempts to call the scheduler component at its initialization and configure a regular, recurring flush of its memory cache:
If the audit log cannot find an implementation of the scheduler registration service, it does not schedule the flush and continue loading.
Audit log implements the
dynamic_loader_services_loaded_notification
service and listens for new registrations ofmysql_scheduler
so that audit log can register its scheduled task into the newly loaded scheduler.Audit log only registers itself into the first scheduler implementation loaded.
Similarly, MySQL Enterprise Audit calls the scheduler
component at its deinitialization and unconfigures the
recurring flush that it has scheduled. It keeps an active
reference to the scheduler registration service until the
scheduled task is unregistered, ensuring that the
scheduler
component cannot be unloaded
while there are active scheduled jobs. All of the results from
executing the scheduler and its tasks are written to the
server error log.
To schedule an audit log flush task:
Confirm that the
scheduler
component is loaded and enabled. The component is enabled (ON
) by default (seecomponent_scheduler.enabled
).SELECT * FROM mysql.components; +--------------+--------------------+----------------------------+ | component_id | component_group_id | component_urn | +--------------+--------------------+----------------------------+ | 1 | 1 | file://component_scheduler | +--------------+--------------------+----------------------------+
Install the
audit_log
plugin, if it is not installed already (see Section 8.4.5.2, “Installing or Uninstalling MySQL Enterprise Audit”).Start the server using
audit_log_flush_interval_seconds
and set the value to a number greater than 59. The upper limit of the value varies by platform. For example, to configure the flush task to recur every two minutes:$> mysqld --audit_log_flush_interval_seconds=120
For more information, see the
audit_log_flush_interval_seconds
system variable.
In MySQL 8.0.30 and later, you can extend log files in JSON
format with optional data fields to show the query time, the
number of bytes sent and received, the number of rows returned
to the client, and the number of rows examined. This data is
available in the slow query log for qualifying queries, and in
the context of the audit log it similarly helps to detect
outliers for activity analysis. The extended data fields can
be added only when the audit log is in JSON format
(audit_log_format=JSON
),
which is not the default setting.
The query statistics are delivered to the audit log through
component services that you set up as an audit log filtering
function. The services are named
mysql_audit_print_service_longlong_data_source
and
mysql_audit_print_service_double_data_source
.
You can choose either data type for each output item. For the
query time, longlong
outputs the value in
microseconds, and double
outputs the value
in seconds.
You add the query statistics using the
audit_log_filter_set_filter()
audit log function, as the service
element
of the JSON filtering syntax, as follows:
SELECT audit_log_filter_set_filter('QueryStatistics',
'{ "filter": { "class": { "name": "general", "event": { "name": "status", "print" : '
'{ "service": { "implementation": "mysql_server", "tag": "query_statistics", "element": [ '
'{ "name": "query_time", "type": "double" }, '
'{ "name": "bytes_sent", "type": "longlong" }, '
'{ "name": "bytes_received", "type": "longlong" }, '
'{ "name": "rows_sent", "type": "longlong" }, '
'{ "name": "rows_examined", "type": "longlong" } ] } } } } } }');
For the bytes_sent
and
bytes_received
fields to be populated, the
system variable
log_slow_extra
must be set to
ON
. If the system variable is value is
OFF
, a null value is written to the log
file for these fields.
If you want to stop collecting the query statistics, use the
audit_log_filter_set_filter()
audit log function to remove the filter, for example:
SELECT audit_log_filter_remove_filter('QueryStatistics');
Audit log file compression can be enabled for any logging format.
To configure audit log file compression, set the
audit_log_compression
system
variable at server startup. Permitted values are
NONE
(no compression; the default) and
GZIP
(GNU Zip compression).
If both compression and encryption are enabled, compression occurs before encryption. To recover the original file manually, first decrypt it, then uncompress it. See Manually Uncompressing and Decrypting Audit Log Files.
Audit log file encryption can be enabled for any logging format. Encryption is based on user-defined passwords (with the exception of the initial password that the audit log plugin generates). To use this feature, the MySQL keyring must be enabled because audit logging uses it for password storage. Any keyring component or plugin can be used; for instructions, see Section 8.4.4, “The MySQL Keyring”.
To configure audit log file encryption, set the
audit_log_encryption
system
variable at server startup. Permitted values are
NONE
(no encryption; the default) and
AES
(AES-256-CBC cipher encryption).
To set or get an encryption password at runtime, use these audit log functions:
To set the current encryption password, invoke
audit_log_encryption_password_set()
. This function stores the new password in the keyring. If encryption is enabled, it also performs a log file rotation operation that renames the current log file, and begins a new log file encrypted with the password. File renaming occurs according to the usual rules for automatic size-based log file rotation; see Manual Audit Log File Rotation (Before MySQL 8.0.31).If the
audit_log_password_history_keep_days
system variable is nonzero, invokingaudit_log_encryption_password_set()
also causes expiration of old archived audit log encryption passwords. For information about audit log password history, including password archiving and expiration, see the description of that variable.To get the current encryption password, invoke
audit_log_encryption_password_get()
with no argument. To get a password by ID, pass an argument that specifies the keyring ID of the current password or an archived password.To determine which audit log keyring IDs exist, query the Performance Schema
keyring_keys
table:mysql> SELECT KEY_ID FROM performance_schema.keyring_keys WHERE KEY_ID LIKE 'audit_log%' ORDER BY KEY_ID; +-----------------------------+ | KEY_ID | +-----------------------------+ | audit_log-20190415T152248-1 | | audit_log-20190415T153507-1 | | audit_log-20190416T125122-1 | | audit_log-20190416T141608-1 | +-----------------------------+
For additional information about audit log encryption functions, see Audit Log Functions.
When the audit log plugin initializes, if it finds that log
file encryption is enabled, it checks whether the keyring
contains an audit log encryption password. If not, the plugin
automatically generates a random initial encryption password
and stores it in the keyring. To discover this password,
invoke
audit_log_encryption_password_get()
.
If both compression and encryption are enabled, compression occurs before encryption. To recover the original file manually, first decrypt it, then uncompress it. See Manually Uncompressing and Decrypting Audit Log Files.
Audit log files can be uncompressed and decrypted using standard tools. This should be done only for log files that have been closed (archived) and are no longer in use, not for the log file that the audit log plugin is currently writing. You can recognize archived log files because they have been renamed by the audit log plugin to include a timestamp in the file name just after the base name.
For this discussion, assume that
audit_log_file
is set to
audit.log
. In that case, an archived
audit log file has one of the names shown in the following
table.
Enabled Features | Archived File Name |
---|---|
No compression or encryption | audit. |
Compression | audit. |
Encryption | audit. |
Compression, encryption | audit. |
As discussed in Naming Conventions for Audit Log Files,
pwd_id
format is
pwd_timestamp-seq
. Thus, the names
of archived encrypted log files actually contain two
timestamps. The first indicates file rotation time, and the
second indicates when the encryption password was created.
Consider the following set of archived encrypted log file names:
audit.20190410T205827.log.20190403T185337-1.enc
audit.20190410T210243.log.20190403T185337-1.enc
audit.20190415T145309.log.20190414T223342-1.enc
audit.20190415T151322.log.20190414T223342-2.enc
Each file name has a unique rotation-time timestamp. By contrast, the password timestamps are not unique:
The first two files have the same password ID and sequence number (
20190403T185337-1
). They have the same encryption password.The second two files have the same password ID (
20190414T223342
) but different sequence numbers (1
,2
). These files have different encryption passwords.
To uncompress a compressed log file manually, use gunzip, gzip -d, or equivalent command. For example:
gunzip -c audit.timestamp.log.gz > audit.timestamp.log
To decrypt an encrypted log file manually, use the openssl command. For example:
openssl enc -d -aes-256-cbc -pass pass:password -md sha256
-in audit.timestamp.log.pwd_id.enc
-out audit.timestamp.log
To execute that command, you must obtain
password
, the encryption password.
To do this, use
audit_log_encryption_password_get()
.
For example, if the audit log file name is
audit.20190415T151322.log.20190414T223342-2.enc
,
the password ID is 20190414T223342-2
and
the keyring ID is
audit-log-20190414T223342-2
. Retrieve the
keyring password like this:
SELECT audit_log_encryption_password_get('audit-log-20190414T223342-2');
If both compression and encryption are enabled for audit
logging, compression occurs before encryption. In this case,
the file name has .gz
and
.
suffixes added, corresponding to the order in which those
operations occur. To recover the original file manually,
perform the operations in reverse. That is, first decrypt the
file, then uncompress it:
pwd_id
.enc
openssl enc -d -aes-256-cbc -pass pass:password -md sha256
-in audit.timestamp.log.gz.pwd_id.enc
-out audit.timestamp.log.gz
gunzip -c audit.timestamp.log.gz > audit.timestamp.log
This section covers the differences in audit log file encryption capabilities prior to and as of MySQL 8.0.17, which is when password history was implemented (which includes password archiving and expiration). It also indicates how the audit log plugin handles upgrades to MySQL 8.0.17 or higher from versions lower than 8.0.17.
Feature | Prior to MySQL 8.0.17 | As of MySQL 8.0.17 |
---|---|---|
Number of passwords | Single password only | Multiple passwords permitted |
Encrypted log file names | .enc suffix |
. suffix |
Password keyring ID | audit_log |
audit_log- |
Password history | No | Yes |
Prior to MySQL 8.0.17, there is no password history, so setting a new password makes the old password inaccessible, rendering MySQL Enterprise Audit unable to read log files encrypted with the old password. Should you anticipate a need to decrypt those files manually, you must maintain a record of previous passwords.
If audit log file encryption is enabled when you upgrade to MySQL 8.0.17 or higher from a lower version, the audit log plugin performs these upgrade actions:
During plugin initialization, the plugin checks for an encryption password with a keyring ID of
audit_log
. If it finds one, the plugin duplicates the password using a keyring ID inaudit_log-
format and uses it as the current encryption password. (For details aboutpwd_id
pwd_id
syntax, see Naming Conventions for Audit Log Files.)Existing encrypted log files have a suffix of
.enc
. The plugin does not rename these to have a suffix of.
, but can read them as long as the key with the ID ofpwd_id
.encaudit_log
remains in the keyring.When password cleanup occurs, if the plugin expires any password with a keyring ID in
audit_log-
format, it also expires the password with a keyring ID ofpwd_id
audit_log
, if it exists. (At this point, encrypted log files that have a suffix of.enc
rather than.
become unreadable by the plugin, so it is assumed that you no longer need them.)pwd_id
.enc
The audit log file has the potential to grow quite large and consume a great deal of disk space. If you are collecting the optional query time and size statistics, which are available from MySQL 8.0.30, this increases the space requirements. The query statistics are only supported with JSON format.
To manage the space used, employ these methods:
Log file rotation. This involves rotating the current log file by renaming it, then opening a new current log file using the original name. Rotation can be performed manually, or configured to occur automatically.
Pruning of rotated JSON-format log files, if automatic rotation is enabled. Pruning can be performed based on log file age (as of MySQL 8.0.24), or combined log file size (as of MySQL 8.0.26).
To configure audit log file space management, use the following system variables:
If
audit_log_rotate_on_size
is 0 (the default), automatic log file rotation is disabled.No rotation occurs unless performed manually.
To rotate the current file, use one of the following methods:
Before MySQL 8.0.31, manually rename the file, then enable
audit_log_flush
to close it and open a new current log file using the original name. This file rotation method and theaudit_log_flush
variable are deprecated in MySQL 8.0.31.With this file rotation method, pruning of rotated JSON-format log files does not occur;
audit_log_max_size
andaudit_log_prune_seconds
have no effect.From MySQL 8.0.31, run
SELECT audit_log_rotate();
to rename the file and open a new audit log file using the original name.With this file rotation method, pruning of rotated JSON-format log files occurs if
audit_log_max_size
oraudit_log_prune_seconds
has a value greater than 0.
If
audit_log_rotate_on_size
is greater than 0, automatic audit log file rotation is enabled:Automatic rotation occurs when a write to the current log file causes its size to exceed the
audit_log_rotate_on_size
value, as well as under certain other conditions; see Automatic Audit Log File Rotation. When automatic rotation occurs, the audit log plugin renames the current log file and opens a new current log file using the original name.Pruning of rotated JSON-format log files occurs if
audit_log_max_size
oraudit_log_prune_seconds
has a value greater than 0.audit_log_flush
has no effect.
For JSON-format log files, rotation also occurs when the
value of the
audit_log_format_unix_timestamp
system variable is changed at runtime. However, this does
not occur for space-management purposes, but rather so that,
for a given JSON-format log file, all records in the file
either do or do not include the time
field.
Rotated (renamed) log files are not removed automatically. For example, with size-based log file rotation, renamed log files have unique names and accumulate indefinitely. They do not rotate off the end of the name sequence. To avoid excessive use of space:
As of MySQL 8.0.24 (for JSON-format log files): Enable log file pruning as described in Audit Log File Pruning.
Otherwise (for non-JSON files, or prior to MySQL 8.0.24 for all log formats): Remove old files periodically, backing them up first as necessary. If backed-up log files are encrypted, also back up the corresponding encryption passwords to a safe place, should you need to decrypt the files later.
The following sections describe log file rotation and pruning in greater detail.
Manual Audit Log File Rotation (Before MySQL 8.0.31)
From MySQL 8.0.31, the
audit_log_flush
variable
and this method of audit log file rotation are deprecated;
expect support to be removed in a future version of MySQL.
If audit_log_rotate_on_size
is 0 (the default), no log rotation occurs unless performed
manually. In this case, the audit log plugin closes and
reopens the log file when the
audit_log_flush
value changes
from disabled to enabled. Log file renaming must be done
externally to the server. Suppose that the log file name is
audit.log
and you want to maintain the
three most recent log files, cycling through the names
audit.log.1
through
audit.log.3
. On Unix, perform rotation
manually like this:
From the command line, rename the current log files:
mv audit.log.2 audit.log.3 mv audit.log.1 audit.log.2 mv audit.log audit.log.1
This strategy overwrites the current
audit.log.3
contents, placing a bound on the number of archived log files and the space they use.At this point, the plugin is still writing to the current log file, which has been renamed to
audit.log.1
. Connect to the server and flush the log file so the plugin closes it and reopens a newaudit.log
file:SET GLOBAL audit_log_flush = ON;
audit_log_flush
is special in that its value remainsOFF
so that you need not disable it explicitly before enabling it again to perform another flush.
If compression or encryption are enabled, log file names include suffixes that signify the enabled features, as well as a password ID if encryption is enabled. If file names include a password ID, be sure to retain the ID in the name of any files you rename manually so that the password to use for decryption operations can be determined.
For JSON-format logging, renaming audit log files manually
makes them unavailable to the log-reading functions because
the audit log plugin can no longer determine that they are
part of the log file sequence (see
Section 8.4.5.6, “Reading Audit Log Files”). Consider setting
audit_log_rotate_on_size
greater than 0 to use size-based rotation instead.
Manual Audit Log File Rotation (From MySQL 8.0.31)
If audit_log_rotate_on_size
is 0 (the default), no log rotation occurs unless performed
manually.
To rotate the audit log file manually, run SELECT
audit_log_rotate();
to rename the current audit log
file and open a new audit log file. Files are renamed
according to the conventions described in
Naming Conventions for Audit Log Files.
The AUDIT_ADMIN
privilege is
required to use the
audit_log_rotate()
function.
Managing the number of archived log files (the files that have been renamed) and the space they use is a manual task that involves removing archived audit log files that are no longer needed from your file system.
The content of audit log files that are renamed using the
audit_log_rotate()
function can be read by
audit_log_read()
function.
Automatic Audit Log File Rotation
If audit_log_rotate_on_size
is greater than 0, setting
audit_log_flush
has no
effect. Instead, whenever a write to the current log file
causes its size to exceed the
audit_log_rotate_on_size
value, the audit log plugin automatically renames the current
log file and opens a new current log file using the original
name.
Automatic size-based rotation also occurs under these conditions:
During plugin initialization, if a file with the audit log file name already exists (see Naming Conventions for Audit Log Files).
During plugin termination.
When the
audit_log_encryption_password_set()
function is called to set the encryption password, if encryption is enabled. (Rotation does not occur if encryption is disabled.)
The plugin renames the original file by inserting a timestamp
just after its base name. For example, if the file name is
audit.log
, the plugin renames it to a
value such as audit.20210115T140633.log
.
The timestamp is a UTC value in
format. For XML logging, the timestamp indicates rotation
time. For JSON logging, the timestamp is that of the last
event written to the file.
YYYYMMDD
Thhmmss
If log files are encrypted, the original file name already
contains a timestamp indicating the encryption password
creation time (see Naming Conventions for Audit Log Files). In
this case, the file name after rotation contains two
timestamps. For example, an encrypted log file named
audit.log.20210110T130749-1.enc
is
renamed to a value such as
audit.20210115T140633.log.20210110T130749-1.enc
.
Audit Log File Pruning
The audit log plugin supports pruning of rotated JSON-format audit log files, if automatic log file rotation is enabled. To use this capability:
Set
audit_log_format
toJSON
. (In addition, consider also changingaudit_log_file
; see Selecting Audit Log File Format.)Set
audit_log_rotate_on_size
greater than 0 to specify the size in bytes at which automatic log file rotation occurs.By default, no pruning of automatically rotated JSON-format log files occurs. To enable pruning, set one of these system variables to a value greater than 0:
Set
audit_log_max_size
greater than 0 to specify the limit in bytes on the combined size of rotated log files above which the files become subject to pruning.audit_log_max_size
is available as of MySQL 8.0.26.Set
audit_log_prune_seconds
greater than 0 to specify the number of seconds after which rotated log files become subject to pruning.audit_log_prune_seconds
is available as of MySQL 8.0.24.
Nonzero values of
audit_log_max_size
take precedence over nonzero values ofaudit_log_prune_seconds
. If both are set greater than 0 at plugin initialization, a warning is written to the server error log. If a client sets both greater than 0 at runtime, a warning is returned to the client.NoteWarnings to the error log are written as Notes, which are information messages. To ensure that such messages appear in the error log and are not discarded, make sure that error-logging verbosity is sufficient to include information messages. For example, if you are using priority-based log filtering, as described in Section 7.4.2.5, “Priority-Based Error Log Filtering (log_filter_internal)”, set the
log_error_verbosity
system variable to a value of 3.
Pruning of JSON-format log files, if enabled, occurs as follows:
When automatic rotation takes place; for the conditions under which this happens, see Automatic Audit Log File Rotation.
When the global
audit_log_max_size
oraudit_log_prune_seconds
system variable is set at runtime.
For pruning based on combined rotated log file size, if the
combined size is greater than the limit specified by
audit_log_max_size
, the audit
log plugin removes the oldest files until their combined size
does not exceed the limit.
For pruning based on rotated log file age, the pruning point
is the current time minus the value of
audit_log_prune_seconds
. In
rotated JSON-format log files, the timestamp part of each file
name indicates the timestamp of the last event written to the
file. The audit log plugin uses file name timestamps to
determine which files contain only events older than the
pruning point, and removes them.
The audit log plugin can use any of several strategies for log writes. Regardless of strategy, logging occurs on a best-effort basis, with no guarantee of consistency.
To specify a write strategy, set the
audit_log_strategy
system
variable at server startup. By default, the strategy value is
ASYNCHRONOUS
and the plugin logs
asynchronously to a buffer, waiting if the buffer is full.
It's possible to tell the plugin not to wait
(PERFORMANCE
) or to log synchronously,
either using file system caching
(SEMISYNCHRONOUS
) or forcing output with a
sync()
call after each write request
(SYNCHRONOUS
).
For asynchronous write strategy, the
audit_log_buffer_size
system
variable is the buffer size in bytes. Set this variable at
server startup to change the buffer size. The plugin uses a
single buffer, which it allocates when it initializes and
removes when it terminates. The plugin does not allocate this
buffer for nonasynchronous write strategies.
Asynchronous logging strategy has these characteristics:
Minimal impact on server performance and scalability.
Blocking of threads that generate audit events for the shortest possible time; that is, time to allocate the buffer plus time to copy the event to the buffer.
Output goes to the buffer. A separate thread handles writes from the buffer to the log file.
With asynchronous logging, the integrity of the log file may
be compromised if a problem occurs during a write to the file
or if the plugin does not shut down cleanly (for example, in
the event that the server host exits unexpectedly). To reduce
this risk, set
audit_log_strategy
to use
synchronous logging.
A disadvantage of PERFORMANCE
strategy is
that it drops events when the buffer is full. For a heavily
loaded server, the audit log may have events missing.