WL#17327: Health Monitor: Configuration Policy

Affects: Server-26.x   —   Status: Complete

High Level Description

This worklog defines the Health Monitor configuration for the Community Edition.

The first Community Edition release keeps disk fencing available, but fencing is disabled by default. Disk, memory, system and forecast monitoring remain enabled.

Scope for Milestone 1

This worklog includes:

  • Community Edition defaults for disk_monitored and disk_fallback_enable.
  • Automatic selection of the filesystem that contains the MySQL data directory.
  • Low disk space fencing disabled by default.
  • Safe handling of invalid or unavailable monitored paths.
  • Correct fencing ownership and recovery cancellation when ownership ends.
  • Focused unit and MTR coverage for the changed configuration.

These changes are deferred to Milestone 2:

  • Separate status message intervals for the disk, memory and system collectors and the disk forecaster.
  • Changes to the Disk Forecaster status cadence or its use of the shared status_interval variable.
  • General review and rewriting of retained system variable descriptions and metadata.
  • New validation for sample interval multiples, forecast test filenames and report definitions.

  • Atomic validation of related system variables set during a SET statement, including the proposed warning level ordering

    level_1 >= level_2 >= level_3

  • Changes to completed recovery cycle count.

  • Broad persistence testing for retained system variables.
  • Cleanup of Disk Forecaster run mode names or enum values.

Functional and Non-functional Requirements

Functional Requirements

FR-1 Community Edition Defaults

The public component provides defaults that work on a fresh Community or Enterprise installation without an MHS filesystem layout or MHS configuration.

FR-2 Monitored Filesystem

When health_monitor.disk_monitored is empty or reset with SET ... = DEFAULT, the Health Monitor monitors the filesystem that contains the MySQL data directory.

A nonempty configured value identifies a path on the filesystem to monitor. The value does not have to be a mount point.

FR-3 Fencing Default

health_monitor.disk_fallback_enable defaults to OFF.

When it is OFF, the Health Monitor continues disk collection, warnings, Performance Schema updates and forecasting. It does not set OFFLINE_MODE or SUPER_READ_ONLY, terminate sessions or start a new fencing operation.

FR-4 Retained Recovery Default

health_monitor.disk_recovery_enable defaults to ON.

This is retained behavior. Recovery is inactive until the Health Monitor has fenced the server.

FR-5 Collector Status Intervals (Deferred)

Collector-specific status interval variables are deferred to a future worklog.

WL#17327 does not change the existing health_monitor.status_interval behavior. The Disk Forecaster retains its existing internal status cadence.

FR-6 Required Status Messages (Deferred)

Disabling recurring status messages for each collector is deferred with FR-5. Startup, shutdown, configuration change, resource state transition, fencing, recovery and error messages remain unchanged.

FR-7 Changed System Variable Metadata

The descriptions for disk_monitored and disk_fallback_enable state their new Community Edition behavior. General metadata cleanup for retained system variables is deferred.

FR-8 Monitored Path Validation

disk_monitored rejects nonempty values that are not absolute paths, values longer than FN_REFLEN - 1 bytes and values that contain embedded null bytes. The empty value remains valid and selects the data directory filesystem.

The following validation changes are deferred:

  • Exact multiple validation for disk and forecast sample intervals.
  • New validation for the forecast test filename and report definitions.
  • Validate each warning level against the other levels.
  • Atomic validation of variables set in a multi-variable SET statement.

The deferred work will define and enforce the proposed warning level ordering level_1 >= level_2 >= level_3. WL#17327 only keeps the existing per-value range checks.

Also, warning levels can be set with one statement:

SET GLOBAL health_monitor.disk_usage_warning_level_1 = 8,
    GLOBAL health_monitor.disk_usage_warning_level_2 = 7,
    GLOBAL health_monitor.disk_usage_warning_level_3 = 6;

FR-9 Dynamic Updates

The changed variables remain global and dynamic. The Supervisor applies collector settings without acquiring a collector command lock or reading collector settings from a system variable callback.

FR-10 Persistence

Health Monitor system variables retain PLUGIN_VAR_PERSIST_AS_READ_ONLY registration.

Focused tests cover persistence and component reload for disk_monitored and disk_fallback_enable. Broad persistence testing for retained variables is deferred.

FR-11 Unavailable Monitored Path

An invalid or unavailable monitored path disables disk sample collection and disk forecasting. It does not stop memory or system collection and does not make component initialization fail.

The Health Monitor reports the unavailable path once. It does not repeat the same warning on every collection cycle. While input is unavailable in NORMAL mode, the Disk Forecaster does not consume queued metrics, enable metric writes, advance forecast, report or status timers, generate forecasts or write cached forecast reports. It resumes after the next valid disk sample. TEST mode is unchanged. The Health Monitor reports the valid path once when the path becomes available again.

FR-12 Fencing Ownership

Setting health_monitor.disk_fallback_enable=OFF cancels a pending fencing operation.

If the Health Monitor already set OFFLINE_MODE or SUPER_READ_ONLY, it allows recovery to normal operating mode when storage recovers.

If the Health Monitor determines that either OFFLINE_MODE or SUPER_READ_ONLY were changed externally, then the Health Monitor cancels any active recovery and resets its recovery timers. It preserves completed recovery cycle history.

FR-13 Status Variables

The super_read_only_disk_full and offline_mode_disk_full status variables (aka 'disk full status') indicates whether the Health Monitor fenced the server. If the server is fenced and another user or component clears SUPER_READ_ONLY or OFFLINE_MODE, the Health Monitor sets the disk full status variables to OFF and regards the server as unfenced.

If another user or component enables SUPER_READ_ONLY or OFFLINE_MODE, the Health Monitor does not claim the server is fenced, and the corresponding disk full statuses remains OFF.

FR-14 MHS Compatibility

MHS must now explicitly configure disk_monitored and disk_fallback_enable to retain cloud behavior.

IMPORTANT: The MHS configuration change is a prerequisite for changing the component defaults.

FR-15 Focused Tests

The worklog adds focused unit and MTR tests for the two changed defaults, monitored path selection and validation, unavailable input, fencing opt in, ownership reconciliation, recovery cancellation and the required MHS overrides.

Non-functional Requirements

NFR-1 Safe Defaults

Installing the Community or Enterprise Edition with default settings must not fence the server or terminate sessions.

NFR-2 Single Implementation

The Community and Enterprise editions and MHS use the same source and system variable definitions. The implementation must not use preprocessor guards to select defaults for cloud or on-premise.

NFR-3 Logging

An unavailable resource or disabled feature must not produce repeated noisy warnings. Logs must distinguish disabled configuration from collection failure.

NFR-4 Predictable Changed Configuration

The changed defaults and effective monitored path must be visible and easy to document.

NFR-5 Compatibility

Existing MHS behavior must not change until the required MHS configuration is present.

Note: The Control Plane changes will be handled separately before the server release.

NFR-6 Security

The monitored path is length checked and rejects embedded null bytes. It is treated as data and is not passed to a shell or interpreted as a command.

NFR-7 Scope Control

This worklog does not change default disk and memory thresholds, state transitions, Performance Schema tables or disk forecasting calculations.

High Level Specification

Current Configuration

The current implementation registers the Health Monitor variables as health_monitor.<name> component system variables.

The current behavior is:

  • health_monitor.disk_monitored defaults to /db on non-Windows platforms.
  • health_monitor.disk_fallback_enable defaults to ON.
  • health_monitor.status_interval defaults to 10 sample intervals. WL#17327 does not change its application behavior.
  • All variables are global and can be changed with SET GLOBAL.
  • All variables use PLUGIN_VAR_PERSIST_AS_READ_ONLY so persisted values can be loaded when the component registers.
  • Most collector settings are applied by restarting one collector through the Health Monitor supervisor.
  • Disk forecast run mode and test filename are applied live.

Community Edition Defaults

This table lists the two changed defaults and the explicitly retained shared status interval. All other defaults remain as listed in Retained System Variables.

Variable Current value New value Rationale
health_monitor.disk_monitored /db Empty string Empty means the filesystem that contains @@GLOBAL.datadir.
health_monitor.disk_fallback_enable ON OFF Disk fencing is opt in.
health_monitor.status_interval 10 sample intervals 10 sample intervals Unchanged. Per-collector replacement is deferred.

The shared status_interval retains its current range and behavior. The Disk Forecaster retains its existing internal status cadence.

Monitored Filesystem

health_monitor.disk_monitored identifies a path, not a block device or mount point.

The path is selected as follows:

  1. If health_monitor.disk_monitored contains a nonempty value, use that value.
  2. Otherwise, read @@GLOBAL.datadir.
  3. Pass the selected path to the platform library API (see WL#17325).
  4. Use the filesystem identity, mount point and usage returned for that path.

Note:

  • The Health Monitor does not modify the system variable.
  • SELECT @@GLOBAL.health_monitor.disk_monitored returns the configured value.
  • Collector status messages will show the fully qualified path.
  • The Health Monitor disk tables will show the fully qualified path and mount information.

A valid path must:

  • Be an absolute path on the active platform.
  • Contain no more than FN_REFLEN - 1 bytes, excluding the terminating null byte.
  • Contain no embedded null byte.

Note:

  • The path does not have to exist when the variable is set.
  • Availability of the path is determined during runtime.
  • A filesystem can be mounted after component startup.
  • A previously available filesystem can become unavailable.

Disk sample collection is unavailable when:

  • The configured path or data directory does not exist.
  • The path cannot be read because of permissions or another operating system error.
  • Filesystem identity or usage cannot be collected.
  • The sample fails the filesystem validation defined by the platform library (WL#17325).

While disk input is unavailable:

  • The disk collector remains installed but does not publish a new disk sample.
  • The last valid rows retain their normal table retention behavior.
  • In NORMAL mode, the disk forecaster does not consume queued metrics, enable metric writes, advance its forecast, report or status timers, generate forecasts or write cached forecast reports.
  • TEST mode is unchanged.
  • No fencing or recovery transition is started from an unavailable sample.
  • The collector retries at its configured collection interval.
  • One warning records the unavailable transition and its reason.

When input becomes available, one recovery message is written and normal sampling resumes.

Fencing and Recovery

The Health Monitor classifies disk space as normal, warning, low or critical when fencing is disabled. Users can observe the state through Performance Schema and error log messages.

With health_monitor.disk_fallback_enable=OFF:

  • The Health Monitor does not set OFFLINE_MODE.
  • The Health Monitor does not set SUPER_READ_ONLY.
  • The Health Monitor does not terminate foreground sessions.
  • health_monitor.disk_fallback_force has no effect.
  • The low space timer does not start a new fencing operation.
  • Disk warnings, metrics and forecasts continue.

health_monitor.disk_fallback_force remains OFF by default.

health_monitor.disk_recovery_enable remains ON by default. Recovery can only happen if the Health Monitor fenced the server. It must not clear OFFLINE_MODE or SUPER_READ_ONLY when those modes were set outside the Health Monitor.

These behaviors remain unchanged:

  • Disabling fencing during a low space countdown cancels that countdown.
  • Disabling fencing after the Health Monitor fences the server does not restore normal operating mode.
  • Recovery continues according to the recovery settings when disk space reaches the recovery level.
  • If role or server mode reconciliation removes the final Health Monitor-owned mode, any active recovery is canceled and its timers are reset. Completed recovery cycle history is retained.

Status Message Intervals

The Health Monitor continues to use the shared health_monitor.status_interval variable with its existing behavior. The Disk Forecaster retains its existing internal status cadence. Changes to its status timer, including zero-value behavior, are deferred with the per-collector status interval work.

These proposed variables are deferred to a future worklog:

  • health_monitor.disk_status_interval
  • health_monitor.memory_status_interval
  • health_monitor.system_status_interval
  • health_monitor.disk_forecast_status_interval

System Variable Behavior

The changed system variables remain:

  • Global.
  • Dynamic through SET GLOBAL.
  • Persistable through SET PERSIST or SET PERSIST_ONLY.
  • Registered with PLUGIN_VAR_PERSIST_AS_READ_ONLY.

SET GLOBAL updates the active component setting. A collector restart may be required, but the MySQL server does not restart.

SET PERSIST writes the value to persisted configuration and updates the active value. SET PERSIST_ONLY writes the value without updating the active value. Because the variables are persist-as-read-only, the stored value is available early enough to be applied when the server or component next loads.

The Health Monitor Supervisor applies changed settings outside the system variable callback. The callback only marks work pending and signals the Supervisor when immediate work is required. This avoids an inverse lock order between LOCK_global_system_variables and collector command locks.

Existing settings retain their current application behavior:

Setting group Application behavior
Collector enable, sample, retention, path and threshold settings Restart only the affected collector.
Disk forecast report interval Restart only the Disk Forecaster.
Disk forecast run mode and test input selection Apply live through the existing Disk Forecaster command path.
Reporter settings Apply through the reporter without restarting unrelated collectors.

System Variable Validation

disk_monitored contains at most FN_REFLEN - 1 bytes and rejects embedded null bytes. A nonempty value must be an absolute path on the active platform.

Existing validation for retained system variables does not change. New sample interval, forecast filename, report definition and related-value validation is deferred to Milestone 2.

Retained System Variables

The system variables in this section keep their current names and Community Edition defaults unless listed in Community Edition Defaults.

The descriptions in these tables explain the retained variables. They do not require WL#17327 to rewrite the registered descriptions or other metadata.

All integer ranges are inclusive.

General

Variable Default Valid values Description
health_monitor.status_interval 10 0 to 1000 sample intervals Shared status message interval in multiples of each collector's sample interval.

Disk Collector

Variable Default Valid values Description
health_monitor.disk_running ON ON, OFF Enable disk collection.
health_monitor.disk_sample_rate 60 1 to 86400 seconds Disk sample interval in seconds.
health_monitor.disk_retention 86400 1 to 864000 seconds Disk sample retention in seconds.
health_monitor.disk_monitored Empty Empty or valid absolute path of at most FN_REFLEN - 1 bytes Path on the filesystem to monitor. Empty selects the data directory filesystem.
health_monitor.disk_fallback_enable OFF ON, OFF Enable low disk space fencing.
health_monitor.disk_fallback_force OFF ON, OFF Permit termination of blocking foreground sessions during fencing.
health_monitor.disk_low_space_duration 300 1 to 86400 seconds Seconds below the low space threshold before fencing.
health_monitor.disk_low_space_level 5000 0 to 18446744073709551614 MiB Fixed low space threshold in MiB used with the percentage threshold.
health_monitor.disk_low_space_percent 4 0 to 100 percent Low space threshold as a percentage of filesystem capacity, used with the fixed threshold.
health_monitor.disk_low_space_critical_level 2000 0 to 18446744073709551614 MiB Critical low space threshold in MiB for immediate fencing when fencing is enabled.
health_monitor.disk_usage_warning_level_1 20 0 to 100 percent First warning level as a percentage above reserved free space.
health_monitor.disk_usage_warning_level_2 10 0 to 100 percent Second warning level as a percentage above reserved free space.
health_monitor.disk_usage_warning_level_3 5 0 to 100 percent Third warning level as a percentage above reserved free space.
health_monitor.disk_recovery_enable ON ON, OFF Enable automatic recovery from Health Monitor fencing.
health_monitor.disk_recovery_level 5 0 to 100 percent Free space percentage above the reserve required to start recovery.
health_monitor.disk_recovery_time_1 300 1 to 86400 seconds Seconds above the recovery level before clearing Health Monitor-owned OFFLINE_MODE.
health_monitor.disk_recovery_time_2 300 1 to 86400 seconds Additional seconds above the recovery level before clearing Health Monitor-owned SUPER_READ_ONLY.
health_monitor.disk_recovery_max_cycles 3 0 to 2147483647 cycles Maximum number of automatic recovery cycles within the recovery window.
health_monitor.disk_recovery_increment 300 0 to 86400 seconds Delay in seconds added after each recovery cycle.
health_monitor.disk_recovery_window 86400 0 to 18446744073709551614 seconds Window in seconds used to reset the recovery cycle count.

The effective low space reserve remains the lesser of disk_low_space_level and disk_low_space_percent of total filesystem capacity.

Disk Forecaster

Variable Default Valid values Description
health_monitor.disk_forecast_running ON ON, OFF Enable disk metric collection and forecasting.
health_monitor.disk_forecast_sample_interval 60 10 to 86400 seconds Disk forecast metric interval in seconds.
health_monitor.disk_forecast_report_interval 240 1 to 10080 minutes Diagnostic disk forecast report interval in minutes.
health_monitor.disk_forecast_run_mode NORMAL NONE, NORMAL, TEST, STANDBY Disk Forecaster operating mode.
health_monitor.disk_forecast_test_filename Empty String Input file selected for TEST mode.

Memory Collector and Reporter

Variable Default Valid values Description
health_monitor.memory_running ON ON, OFF Enable host and MySQL process memory collection.
health_monitor.memory_sample_rate 60 1 to 86400 seconds Memory sample interval in seconds.
health_monitor.memory_retention 86400 1 to 864000 seconds Memory sample retention in seconds.
health_monitor.memory_usage_warning_level_1 1024 0 to 18446744073709551614 MiB First memory warning level in MiB.
health_monitor.memory_usage_warning_level_2 500 0 to 18446744073709551614 MiB Second memory warning level in MiB.
health_monitor.memory_usage_warning_level_3 100 0 to 18446744073709551614 MiB Third memory warning level in MiB.
health_monitor.memory_reporting ON ON, OFF Enable triggered memory reports.
health_monitor.memory_report_definition Built-in definition Valid report definition of at most 65,535 bytes Select sections for triggered memory reports.

Periodic Reporter

Variable Default Valid values Description
health_monitor.periodic_reporting ON ON, OFF Enable periodic Health Monitor reports.
health_monitor.periodic_report_definition Built-in definition Valid report definition of at most 65,535 bytes Select sections and intervals for periodic reports.

Reporter variables control report generation. A report destination such as the diagnostic log must also be available. An unavailable destination does not prevent collector operation.

System Collector

Variable Default Valid values Description
health_monitor.system_running ON ON, OFF Enable CPU, disk, memory and MySQL process statistics collection.
health_monitor.system_sample_rate 60 1 to 86400 seconds System statistics sample interval in seconds.
health_monitor.system_retention 86400 1 to 864000 seconds System statistics retention in seconds.
health_monitor.system_cpu_stats_history ON ON, OFF Enable CPU history rows.
health_monitor.system_disk_stats_history ON ON, OFF Enable disk history rows when disk statistics are available.
health_monitor.system_memory_stats_history ON ON, OFF Enable system memory history rows.
health_monitor.system_process_stats_history ON ON, OFF Enable MySQL process history rows.

An unavailable metric remains NULL and not zero (see WL#17325).

Status Variables

This worklog keeps the existing global status variable names and types.

Status variable Type Meaning
offline_mode_disk_full Boolean ON if the Health Monitor set OFFLINE_MODE=ON for low disk space.
super_read_only_disk_full Boolean ON if the Health Monitor set SUPER_READ_ONLY=ON for low disk space.
recovery_cycles Integer Number of fence and recovery cycles in the active recovery window.
disk_forecaster_run_mode String Current Disk Forecaster mode.
disk_forecaster_test_state String Current state of disk forecasting test or simulation.
option_tracker_usage:Health Monitor Integer Option Tracker usage count when the Option Tracker service is present. It is not used for Community Edition.

The two disk full status variables do not mirror the server variables directly. If OFFLINE_MODE or SUPER_READ_ONLY are set externally, the Health Monitor status remains OFF unless the Health Monitor owns that change.

MHS Compatibility

IMPORTANT

The component defaults will now be public defaults rather than default for MHS. This means that MHS will now have to explicitly set cloud-specific defaults. #TODO: Jira ticket

Before the new defaults are deployed in MHS, the MHS Control Plane must set:

Variable MHS value
health_monitor.disk_monitored Set to /db.
health_monitor.disk_fallback_enable Set ON.

Failure Behavior

Here's how configuration failures are handled:

Failure Behavior
Invalid value in SET GLOBAL Reject the statement and keep the active value.
Invalid value in persisted configuration Report component variable registration or load failure with the variable name. Do not start collectors with a partially applied configuration.
Data directory lookup failure with automatic disk selection Leave disk collection unavailable. Continue memory and system collection.
Configured disk path unavailable Leave disk collection unavailable and retry. Do not fence from missing input.
Optional platform metric unavailable Publish NULL where defined by WL#17325. Do not treat it as a configuration error.
Optional report destination unavailable Skip that report.

NOTE: An invalid setting is different from an unavailable runtime resource. Invalid settings are rejected. Runtime resources can disappear and recover without having to reload the component.

Testing

Unit Tests

Pure configuration unit tests cover:

  • The two changed Community Edition defaults.
  • Monitored path validation boundaries, empty value behavior and embedded null rejection.
  • Automatic monitored path cache and refresh behavior.

Collector and state tests cover:

  • Fencing disabled during low and critical disk states created with artificial thresholds.
  • disk_fallback_enable must be set explicitly rather than by default.
  • Forced fencing is ignored if disk_fallback_enable=OFF.
  • Fencing countdown is canceled when disk_fallback_enable is set to OFF.
  • Restoration of normal operating mode after the server was fenced.
  • Detect that OFFLINE_MODE and SUPER_READ_ONLY were set externally.
  • Cancel active recovery when reconciliation removes the final owned server mode.
  • Only valid data enters the disk forecaster time series.

MTR Tests

MTR coverage verifies:

  • The two changed defaults after component installation.
  • The default empty disk_monitored value and effective data directory filesystem.
  • SET GLOBAL update behavior for the changed variables.
  • SET PERSIST and SET PERSIST_ONLY behavior for disk_monitored and disk_fallback_enable across component reload.
  • Disk Forecaster pause and resume when disk input becomes unavailable.
  • No fencing with the default configuration during low and critical disk states.
  • disk_fallback_enable=ON enabled fencing and recovery if enabled.
  • Recovery is canceled if reconciliation removes the final owned server mode.
  • No repeated error messages if disk_monitored becomes unavailable.

Milestone 1 tests can use artificial thresholds to create deterministic low and critical disk states.

References

  • WL#17325 Health Monitor: Platform Support.
  • WL#17327 Health Monitor: Configuration.


Copyright (c) 2000, 2026, Oracle Corporation and/or its affiliates. All rights reserved.