WL#9323: Logging services: Improved error logging in 8.0

Affects: Server-8.0   —   Status: Complete

STRUCTURED LOGGING, BIRD'S EYE VIEW


1  The Status Quo

The log interface accepts a single string

  sql_print_information("hi");

or rather, a printf-style format string, and the arguments needed for its % 
substitutions; the result of filling in those substitutions is the 
aforementioned single flat string:

  sql_print_information("Hi, I am MySQL, and I am %d years old!", 21);

Depending on the destination (log file, syslog, Windows EventLog, ...), some 
other information (syslog facility, connection ID, process ID, ...) may appear 
the data data.


2  The Problem

Whenever a new log destination or format is needed (e.g. JSON or XML log format, 
systemd journal log destination, etc.), the server needs to be modified. 
Customers cannot have customized logging; they need to accept our formats, 
and convert them as needed.

To add insult to injury, the majority of the relevant information is encoded in 
an error-message that is easy for humans to parse, but much harder for machines 
to reliably and efficiently process.


3  The Future

3.1  Pluggable logging

Logging shall be recreated as a framework for pluggable components/services (as 
defined in the framework specified in WL#4102). This will allow for customized 
components, implementing new logging formats, or even new kinds of log filtering 
within the process boundary of mysqld. Components could even add custom log 
items (for instance, on Linux a log-source might pull relevant information from 
/proc).

Advantages:

- Only what is needed gets loaded

- Clients can create and maintain their own proprietary modules

- Option to realize certain functionality as Enterprise-only components


3.2  Structured information

Logging shall now use information of higher granularity. The primary concern is 
the adding of error numbers (for easier processing by machines than what just 
the flat error string allowed). Other info-bits shall be supported (e.g. 
component-name). The design attempts to be future-proof/extensible by allowing 
third-party log-sources to add "generic" key/value pairs.

The structured information expressed by the new log-writers might look like so:

XML log-entry:

  2
  parser
  XX999
  sql_parse
  ER_PARSER_TRACE
  Parser saw: INSTALL COMPONENT 
"file://component_log_sink_xml"
  Note



JSON log-entry:
{ "log_priority": 2, "subsystem": "parser", "SQL_state": "XX999", "source_file": 
"sql_parse", "MySQL_err_symbol": "ER_PARSER_TRACE", "log_message": "Parser saw: 
INSTALL COMPONENT \"file://component_log_sink_json\"", "log_label": "Note" }


3.3  Operation

The new API accepts the submitted data, filters it (verbosity, rate-limiting, 
re-priotizing, etc.), and finally forwards the modified set to one or several 
log-writer services.

The log-writers can eventually take many shapes (XML, JSON, ...). In the short 
term, the present ones (syslog, EventLog, flatfile) shall be re-implemented on 
top of the new model and service infastructure. JSON will additionally 
be available as the first sink that can handle structured entries (so that we 
will have a means of testing the new infrastructure).


4  Relationship with other work:

error messages

Error messages (sql_print_*()) should be switched over to the "richer" calls 
(with error codes etc.), and similar messages should be collapsed into a single 
"well-known" one.

error messages 

Currently, if we wish to have a high priority message, we need to tag it "error" 
to prevent its being verbosity-filtered, even if an actual error has not 
occurred.
The new design separates message priority and label ("Warning", "Error", etc.).
Both coincide by default (with the label being derived from the priority), but 
they may be set them separately were necessary (and where freeform labeling is 
supported by the backend; syslog and EventLog do not).

log info sources  *** DEFERRED ***

Implement an example log source that adds additional information about a client 
(where known) to error-log messages, such as _client_role and program_name.
This component is not critical, and is not intended to be implemented and rolled 
out with the first tech demonstrator.

log filtering (filtering engine)

Re-implement current error logging options (verbosity, throttle, etc.) on top of 
the new infrastructure. For error-logging, the centralized filtering is to
replace the current, decentral subclassing of the throttle class as well as
other throttling solutions in the server (e.g. in rpl).

log filtering (configuration engine)  *** DEFERRED ***

More of the new filtering service should be exposed to "power-DBAs" through a 
versatile configuration language.
This component is not critical, and is not intended to be implemented and rolled 
out with the first tech demonstrator.

log writing

Writing error log messages: new service plugins to handle writing to log-
file/syslog/etc.  Re-implements current logging on top of the new service 
infrastructure and logging subsystem.  Adds a new JSON log writer to demonstrate 
richer messages.

service configuration

When the configuration system for new services becomes available later, plugins 
should heed its specifications for their configuration (updating to that is 
currently beyond the scope of the above items, however).