WL#342: Replication Heartbeat
Affects: Server-5.5 — Status: Complete
RATIONALE --------- 1. To solve BUG#20435 - extra relay log rotation; 2. to detect failures more easily and precisely; 3. to have better second_behind_master value on slave (BUG#29309). DESIGN ------ 1. New extension to CHANGE MASTER: CHANGE MASTER SET master_heartbeat_period= val; to be stored in master info file and mi object to provide the value sent to master. The value 'val' needs to be within some reasonable interval. As the cost of handling creation, sending and treating the event on slave side is supposed to be low, the val can be as small as 1 seconds even less. 2. Slave io thread prepares and sends the query 'SET SESSION @master_heartbeat_period= val' to master. From the query the master's dump thread finds out slave's preference about heartbeat sending period. 3. The heartbeat event is created by master's DUMP thread and sent each time @master_heartbeat_period elapses which designate nothing has added to binlog for the period time. 4. On the slave's side the heartbeat event is handled exclusively by IO thread avoiding its recording to relay log as well as engaging slave sql thread. Upon receiving the last sent event's coordinates are compared against the ones slave io thread maintains and updates per each received event, except some "phantoms" including the new Heartbeat. The heartbeat event does not update that local information. 5. The Heartbeat period and the number of received event should be monitorable via SHOW STATUS like 'slave_heartbeat period' and SHOW STATUS like 'slave_received_heartbeats' respectively. FEATURES -------- A newer heartbeat-aware slave will not have any error response from an unintelligent old master about the slave connecting time query set @master_heartbeat_period=val; and naturally the old master will not send heartbeats. In that case, slave will show its chosen heartbeat value in the status, but there will be no real actions. Using a the user variable @master_heartbeat_period instead of the system one avoids displaying the name within a list of available variables for a plain user session. Existence of the user variable master_heartbeat_period can be noticed only via the general query log. User observable behaviour -------------------------- 1. Requesting from the slave to send heartbeats from master with a period: CHANGE MASTER master_heartbeat_period= val where val is the period being of the decimal type with the value in the range [0.001, 4294967] seconds. Notice, that heartbeats are sent by the master only if there is no more unsent events in the actual binlog file for a period longer that master_heartbeat_period. Whenever the master's binlog is updated with an event, the waiting for heartbeat sending condition gets reset. If `val' is zero no hearbeats will be sending. Notice, heartbeat is active by default with the period slave_net_timeout/2. 2. SHOW STATUS like 'slave_heartbeat_period' Slave's side status variable which gets the value from either CHANGE MASTER, master.info or implicitly as `slave_net_timeout/2' (the default). The denominator 2 provides a reasonable default period to guarantee no reconnection will happen to an idling master upon elapsing slave_net_timeout. 3. SHOW STATUS like 'slave_received_heartbeats'; The counter that initializes at slave init time, increments by every received heartbeat and resets to zero with CHANGE MASTER; The memory size for the counter is the size for ulonglong i.e normally 8 bytes. Overflowing it even with the fastest heartbeat is possible only on a cosmic time scale. 4. RESET SLAVE; resets the current heartbeat's period to the default (see 2.). The check of the valid range remains after computing slave_net_timeout/2 with dropping the period's value to the max allowable if the ratio would be greater. 5. SET @@global.slave_net_timeout=`value less than the current hb period` is warned as that'd be an irrational intention. OLD INFO: --------- Suggestion from Jeremy Zawodny at OSCON, July 2001, as a discussion idea: Need active heartbeat detection mechanism for replication monitoring. Writing an error to log file is insufficient notice of failure. Instead need a way to trigger an alert message to DBA or a method for a watcher program to immediately detect when a slave has failed.
Detailed design --------------- A) Data structs and the new object 1. Heartbeat event class with fields: - master_log_file (the last binlog file name on master) master_log_pos (last written event on master) and master_current_time at the moment of creating the heartbeat event are recorded into existing members of the parent Log_event class. Other reasons to inherit heartbeat class from Log_event are as the following: - the cost of a single heartbeat event processing will be low enough in this case even if heartbeat would be sent several times in a second. - Extendability. If we later want to add fields to the event, the log_event has dynamic headers... 2. master_info struct on slave is augmented with - heartbeat_period type of float to allow selecting a real number with precision up to 1 millisecond. - received_heartbeat type of ulonglong counter. Given 1 msec precision and 4 bytes of the heartbeat value's storage the maximum value of the heartbeat is bound to be within 0 and ULONG_MAX/1000 interval. I.e the effective interval for the period is [0.001, 4294967]; zero is excluded to mean not to send the heartbeat. The value of master_info.heartbeat_period is initialized via three options: - syntacticly extended CHANGE MASTER with MASTER_HEARTBEAT_PERIOD=val. - reading from master.info (float number format) - default (slave_net_timeout/2) Notice, there is no way to do that from server startup options which is a consequence of a deprecation BUG#21490. 3. To change master dumping thread to wait with a timeout at MYSQL_LOG::wait_for_update() as pthread_cond_timedwait(&update_cond, &LOCK_log, master_heartbeat_period); If at return from waiting there is ETIMEDOUT or ETIME error condition then heartbeat event is to be sent. 4. Slave io receives a heartbeat and handles it without recording it in relay log. Slave's side waiting status for master's activity is reset upon receiving anything from the socket which produces the desired effect - no reconnection although no real events are received but a hearbeat only. Slave io thread still instantiates the event for checking its validity status as is supposed to be done for most of the replication events. The event's members log_file_name and log_pos are compared against the slave's local knowledge to stop the io thread if log_pos does not match the value from the last event except heartbeats the slave has received. The file names and the log positions must be equal except the case when slave starts with empty master.info and thus does not know the last received event from the master; the slave will update its local mi->log_file_name upon receiving Rotate event (normally it should happen in some fraction of second after connecting). 5. Slave does not need to do anything special if heartbeat does not come. The current logics for reconnecting upon elapsing slave_net_timeout makes its job. 6. Monitoring facilities are added via "standard" procedure - new status var and its display through a function. Notice, rli struct remains intact and changes are done to master_info only as relay_log_info is supposed to deal with relay-loggable feature that the heartbeat does not belong to.
Copyright (c) 2000, 2021, Oracle Corporation and/or its affiliates. All rights reserved.