MySQL  8.0.17
Source Code Documentation
log_builtins_filter.cc File Reference
#include <mysqld_error.h>
#include "log_builtins_filter_imp.h"
#include "log_builtins_imp.h"
#include "my_atomic.h"
#include "my_systime.h"
#include "mysys_err.h"
#include "sql/derror.h"
#include "sql/log.h"
#include "sql/mysqld.h"

Macros

#define THROTTLE_DEFAULT_WINDOW_SIZE_IN_SECONDS   60
 services: log filter: basic filtering More...
 
#define THROTTLE_MICROSECOND_MULTIPLIER   1000000
 

Functions

static bool log_filter_ruleset_full (log_filter_ruleset *rs)
 Predicate: can we add any more rules? More...
 
static log_filter_rulelog_builtins_filter_rule_init (log_filter_ruleset *ruleset)
 Initialize a new rule. More...
 
static int log_builtins_filter_rule_free (log_filter_rule *ri)
 Release all resources associated with a filter rule. More...
 
static void log_builtins_filter_rule_remove (log_filter_ruleset *ruleset, int elem)
 Release filter rule (key/value pair) with the index "elem" in "ruleset". More...
 
static log_filter_rulesetlog_builtins_filter_ruleset_new (log_filter_tag *tag, size_t count)
 Create a new set of filter rules. More...
 
static int log_builtins_filter_ruleset_lock (log_filter_ruleset *ruleset, log_builtins_filter_lock lt)
 Lock and get the filter rules. More...
 
static void log_builtins_filter_ruleset_drop (log_filter_ruleset *ruleset)
 Drop an entire filter rule-set. More...
 
static void log_builtins_filter_ruleset_unlock (log_filter_ruleset *ruleset)
 Unlock filter ruleset. More...
 
static void log_builtins_filter_ruleset_free (log_filter_ruleset **ruleset)
 Free filter ruleset. More...
 
static void log_builtins_filter_set_defaults (log_filter_ruleset *ruleset)
 Defaults for when the configuration engine isn't loaded; aim for 5.7 compatibilty. More...
 
int log_builtins_filter_exit ()
 Deinitialize filtering engine. More...
 
int log_builtins_filter_init ()
 Initialize filtering engine. More...
 
static log_filter_apply log_filter_try_apply (log_line *ll, int ln, log_filter_rule *r)
 Apply the action of an individual rule to an individual log line (or a part thereof, i.e. More...
 
static log_filter_match log_filter_try_match (log_item *li, log_filter_rule *ri)
 Try to match an individual log line-field against an individual rule's condition. More...
 
int log_builtins_filter_run (log_filter_ruleset *ruleset, log_line *ll)
 Apply all matching rules from a filter rule set to a given log line. More...
 
int log_builtins_filter_update_verbosity (int verbosity)
 This is part of the 5.7 emulation: If –log_error_verbosity is changed, we generate an artificial filter rule from it here. More...
 
int log_builtins_filter_parse_suppression_list (char *list, bool update)
 @global.log_error_suppression_list accepts a comma-separated list of error-codes that should not be included in the error-log. More...
 

Variables

static bool filter_inited = false
 
static ulong filter_rule_uuid = 0
 
log_filter_rulesetlog_filter_builtin_rules = nullptr
 
log_filter_tag rule_tag_builtin = {"log_filter_builtin", nullptr}
 

Macro Definition Documentation

◆ THROTTLE_DEFAULT_WINDOW_SIZE_IN_SECONDS

#define THROTTLE_DEFAULT_WINDOW_SIZE_IN_SECONDS   60

services: log filter: basic filtering

This implementation, "dragnet" is currently the default filter and therefore built-in. Basic configuration is built into the server proper (via log_error_verbosity etc.); for advanced configuration, load the service log_filter_dragnet which implements a configuration language for this engine. See there for details about the Configuration Stage. Some of the code-paths are only available via the configuration language or an equivalent service (but not without any such service loaded).

At present, the design is such that multiple threads can call the filter concurrently; the ruleset is global and shared between all users.

FILTERING STAGE

At run time, the filter iterates over its rule-set. For each rule, if the condition contains a well-known item, it looks for an item of that type in the event. If the condition contains an ad hoc-item, it looks for an item of any ad hoc-type with the given key within the event.

If there is a match, the filter will verify whether the storage class of the value in the event and that in the condition are either both strings, or both not. If the classes do not match, it flags an error. Otherwise, it now compares both values using the requested comparator, and reports the result.

If a log event matches a rule, an action ("suppress log line", "delete field", etc.) will be applied to that event.

LOCKING

During the filtering stage, a shared lock on the ruleset is held. An exclusive lock on the ruleset is only taken as response to the user's changing of the filter configuration, which should be rare.

For debugging puroposes, rules feature a counter of how often events matched them; this counter is updated atomically.

Rate-limiting ("throttle") needs some bookkeeping data (when does the current window expire? how many matches have we had so far within the current window? etc.). A write-lock is taken on the individual rule (not the entire ruleset) to update this information; any throttling-related actions taken on the event happen after this lock has been released.

The event itself is not locked.

◆ THROTTLE_MICROSECOND_MULTIPLIER

#define THROTTLE_MICROSECOND_MULTIPLIER   1000000

Function Documentation

◆ log_builtins_filter_exit()

int log_builtins_filter_exit ( )

Deinitialize filtering engine.

Return values
0Success!
-1De-initialize? Filter wasn't even initialized!

◆ log_builtins_filter_init()

int log_builtins_filter_init ( )

Initialize filtering engine.

We need to do this early, before the component system is up.

Return values
0Success!
-1Couldn't initialize ruleset
-2Filter was already initialized?

◆ log_builtins_filter_parse_suppression_list()

int log_builtins_filter_parse_suppression_list ( char *  list,
bool  update 
)

@global.log_error_suppression_list accepts a comma-separated list of error-codes that should not be included in the error-log.

Events with a severity of System or Error can not be filtered in this way and will always be forwarded to the log-sinks.

This provides simple filtering for cases where the flexibility of the loadable filter-language is not needed. (The same engine is used however, just with a more limited interface.)

For this filtering to be active, @global.log_error_services has to feature "log_filter_internal", as it does by default. When that is the case, one or both of log_error_verbosity and this variable may be used. Only one of "log_filter_internal" and "log_filter_dragnet" should be used at a time.

The semantics follow that of our system variables; that is to say, when called with update==false, the function acts as a check-function that validates the entire list given to it; when called with update==true, it creates filter-rules from the list items. This way, we either create all rules, or no rules, rather than ending up with an incomplete rule-set when we encounter a problem in the input.

The return value encodes the location in the argument where the failure occurred, like so:

  • if 0 is returned, no issues were detected
  • if a value less than zero is returned, -(retval + 1) is the byte position (counting from 0) in the list argument at which point the failure was detected
Parameters
listlist of error-codes that should not appear in the error-log
updatefalse: verify list only true: create filtering rules from suppression list
Return values
0success
!0failure (see above)

◆ log_builtins_filter_rule_free()

static int log_builtins_filter_rule_free ( log_filter_rule ri)
static

Release all resources associated with a filter rule.

Leaves a "gap" (an uninitialized rule) for immediate re-filling; if this is undesired, use log_builtins_filter_rule_remove() (see there). Must hold rule-set lock.

Parameters
rithe rule to release
Return values
thereturn value from mysql_rwlock_destroy()

◆ log_builtins_filter_rule_init()

static log_filter_rule* log_builtins_filter_rule_init ( log_filter_ruleset ruleset)
static

Initialize a new rule.

This clears the first unused rule. It does not update the rules count; this is for the caller to do if it succeeds in setting up the rule to its satisfaction. If the caller fails, it should log_builtins_filter_rule_free() the incomplete rule.

Return values
nullptrcould not initialize rule. Do not call rule_free.
!nullptrthe address of the rule. fill in. on success, caller must increase rule count. on failure, it must call rule_free.

◆ log_builtins_filter_rule_remove()

static void log_builtins_filter_rule_remove ( log_filter_ruleset ruleset,
int  elem 
)
static

Release filter rule (key/value pair) with the index "elem" in "ruleset".

This frees whichever of key and value were dynamically allocated. This moves any trailing items to fill the "gap" and decreases the counter of elements in the rule-set. If the intention is to leave a "gap" in the bag that may immediately be overwritten with an updated element, use log_builtins_filter_rule_free() instead! Must hold rule-set lock.

Parameters
rulesetfilter rule-set
elemindex of the filter rule to release

◆ log_builtins_filter_ruleset_drop()

static void log_builtins_filter_ruleset_drop ( log_filter_ruleset ruleset)
static

Drop an entire filter rule-set.

Must hold lock.

◆ log_builtins_filter_ruleset_free()

static void log_builtins_filter_ruleset_free ( log_filter_ruleset **  ruleset)
static

Free filter ruleset.

◆ log_builtins_filter_ruleset_lock()

static int log_builtins_filter_ruleset_lock ( log_filter_ruleset ruleset,
log_builtins_filter_lock  lt 
)
static

Lock and get the filter rules.

Parameters
rulesetthe ruleset to lock
ltLOG_BUILTINS_LOCK_SHARED lock for reading LOG_BUILTINS_LOCK_EXCLUSIVE lock for writing
Return values
0lock acquired
!0failed to acquire lock

◆ log_builtins_filter_ruleset_new()

static log_filter_ruleset* log_builtins_filter_ruleset_new ( log_filter_tag tag,
size_t  count 
)
static

Create a new set of filter rules.

Parameters
tagtag for this ruleset
countnumber of rules to allocate, 0 for default
Return values
apointer to a ruleset structure, or nullptr on failure

◆ log_builtins_filter_ruleset_unlock()

static void log_builtins_filter_ruleset_unlock ( log_filter_ruleset ruleset)
static

Unlock filter ruleset.

◆ log_builtins_filter_run()

int log_builtins_filter_run ( log_filter_ruleset ruleset,
log_line ll 
)

Apply all matching rules from a filter rule set to a given log line.

Parameters
rulesetrule-set to apply
llthe current log line
Return values
intnumber of matched rules

◆ log_builtins_filter_set_defaults()

static void log_builtins_filter_set_defaults ( log_filter_ruleset ruleset)
static

Defaults for when the configuration engine isn't loaded; aim for 5.7 compatibilty.

◆ log_builtins_filter_update_verbosity()

int log_builtins_filter_update_verbosity ( int  verbosity)

This is part of the 5.7 emulation: If –log_error_verbosity is changed, we generate an artificial filter rule from it here.

For this filtering to be active, @global.log_error_services has to feature "log_filter_internal", as it does by default. When that is the case, one or both of log_error_verbosity and log_error_suppression_list (see below) may be used. Only one of "log_filter_internal" and "log_filter_dragnet" should be used at a time.

Parameters
verbositylog_error_verbosity style, range(1,3) 1:errors, 2:+=warnings, 3:+=notes
Return values
0success
!0failure

◆ log_filter_ruleset_full()

static bool log_filter_ruleset_full ( log_filter_ruleset rs)
static

Predicate: can we add any more rules?

Parameters
rsthe ruleset to check
Return values
truefull, no more rules can be added
falsenot full, further rules can be added

◆ log_filter_try_apply()

static log_filter_apply log_filter_try_apply ( log_line ll,
int  ln,
log_filter_rule r 
)
static

Apply the action of an individual rule to an individual log line (or a part thereof, i.e.

a "field"). At this point, we already know that the current log line matches the condition.

Parameters
[in,out]llthe current log line
lnindex of the matching field, -1 for none (when a test for absence matched)
rthe rule to apply. internal state may be changed (i.e. the number of seen matches for a throttle rule)
Return values
log_filter_apply0 on success, an error-code otherwise

◆ log_filter_try_match()

static log_filter_match log_filter_try_match ( log_item li,
log_filter_rule ri 
)
static

Try to match an individual log line-field against an individual rule's condition.

Parameters
lithe log item we try to match
rithe rule containing the condition
Return values
log_filter_match0 (LOG_FILTER_MATCH_SUCCESS) on match, 1 (LOG_FILTER_MATCH_UNSATISFIED) or an error-code otherwise

Variable Documentation

◆ filter_inited

bool filter_inited = false
static

◆ filter_rule_uuid

ulong filter_rule_uuid = 0
static

◆ log_filter_builtin_rules

log_filter_ruleset* log_filter_builtin_rules = nullptr

◆ rule_tag_builtin

log_filter_tag rule_tag_builtin = {"log_filter_builtin", nullptr}