25.6.2.1 NDB Cluster: Messages in the Cluster Log

The following table lists the most common NDB cluster log messages. For information about the cluster log, log events, and event types, see Section 25.6.3, “Event Reports Generated in NDB Cluster”. These log messages also correspond to log event types in the MGM API; see The Ndb_logevent_type Type, for related information of interest to Cluster API developers.

Table 25.53 Common NDB cluster log messages

Log Message	Description	Event Name	Event Type	Priority	Severity
Node mgm_node_id: Node data_node_id Connected	The data node having node ID node_id has connected to the management server (node mgm_node_id).	Connected	Connection	8	INFO
Node mgm_node_id: Node data_node_id Disconnected	The data node having node ID data_node_id has disconnected from the management server (node mgm_node_id).	Disconnected	Connection	8	ALERT
Node data_node_id: Communication to Node api_node_id closed	The API node or SQL node having node ID api_node_id is no longer communicating with data node data_node_id.	CommunicationClosed	Connection	8	INFO
Node data_node_id: Communication to Node api_node_id opened	The API node or SQL node having node ID api_node_id is now communicating with data node data_node_id.	CommunicationOpened	Connection	8	INFO
Node mgm_node_id: Node api_node_id: API version	The API node having node ID api_node_id has connected to management node mgm_node_id using NDB API version version (generally the same as the MySQL version number).	ConnectedApiVersion	Connection	8	INFO
Node node_id: Global checkpoint gci started	A global checkpoint with the ID gci has been started; node node_id is the master responsible for this global checkpoint.	GlobalCheckpointStarted	Checkpoint	9	INFO
Node node_id: Global checkpoint gci completed	The global checkpoint having the ID gci has been completed; node node_id was the master responsible for this global checkpoint.	GlobalCheckpointCompleted	Checkpoint	10	INFO
Node node_id: Local checkpoint lcp started. Keep GCI = current_gci oldest restorable GCI = old_gci	The local checkpoint having sequence ID lcp has been started on node node_id. The most recent GCI that can be used has the index current_gci, and the oldest GCI from which the cluster can be restored has the index old_gci.	LocalCheckpointStarted	Checkpoint	7	INFO
Node node_id: Local checkpoint lcp completed	The local checkpoint having sequence ID lcp on node node_id has been completed.	LocalCheckpointCompleted	Checkpoint	8	INFO
Node node_id: Local Checkpoint stopped in CALCULATED_KEEP_GCI	The node was unable to determine the most recent usable GCI.	LCPStoppedInCalcKeepGci	Checkpoint	0	ALERT
Node node_id: Table ID = table_id, fragment ID = fragment_id has completed LCP on Node node_id maxGciStarted: started_gci maxGciCompleted: completed_gci	A table fragment has been checkpointed to disk on node node_id. The GCI in progress has the index started_gci, and the most recent GCI to have been completed has the index completed_gci.	LCPFragmentCompleted	Checkpoint	11	INFO
Node node_id: ACC Blocked num_1 and TUP Blocked num_2 times last second	Undo logging is blocked because the log buffer is close to overflowing.	UndoLogBlocked	Checkpoint	7	INFO
Node node_id: Start initiated version	Data node node_id, running NDB version version, is beginning its startup process.	NDBStartStarted	StartUp	1	INFO
Node node_id: Started version	Data node node_id, running NDB version version, has started successfully.	NDBStartCompleted	StartUp	1	INFO
Node node_id: STTORRY received after restart finished	The node has received a signal indicating that a cluster restart has completed.	STTORRYRecieved	StartUp	15	INFO
Node node_id: Start phase phase completed (type)	The node has completed start phase phase of a type start. For a listing of start phases, see Section 25.6.4, “Summary of NDB Cluster Start Phases”. (type is one of initial, system, node, initial node, or <Unknown>.)	StartPhaseCompleted	StartUp	4	INFO
Node node_id: CM_REGCONF president = president_id, own Node = own_id, our dynamic id = dynamic_id	Node president_id has been selected as “president”. own_id and dynamic_id should always be the same as the ID (node_id) of the reporting node.	CM_REGCONF	StartUp	3	INFO
Node node_id: CM_REGREF from Node president_id to our Node node_id. Cause = cause	The reporting node (ID node_id) was unable to accept node president_id as president. The cause of the problem is given as one of Busy, Election with wait = false, Not president, Election without selecting new candidate, or No such cause.	CM_REGREF	StartUp	8	INFO
Node node_id: We are Node own_id with dynamic ID dynamic_id, our left neighbor is Node id_1, our right is Node id_2	The node has discovered its neighboring nodes in the cluster (node id_1 and node id_2). node_id, own_id, and dynamic_id should always be the same; if they are not, this indicates a serious misconfiguration of the cluster nodes.	FIND_NEIGHBOURS	StartUp	8	INFO
Node node_id: type shutdown initiated	The node has received a shutdown signal. The type of shutdown is either Cluster or Node.	NDBStopStarted	StartUp	1	INFO
Node node_id: Node shutdown completed [, action] [Initiated by signal signal.]	The node has been shut down. This report may include an action, which if present is one of restarting, no start, or initial. The report may also include a reference to an NDB Protocol signal; for possible signals, refer to Operations and Signals.	NDBStopCompleted	StartUp	1	INFO
Node node_id: Forced node shutdown completed [, action]. [Occurred during startphase start_phase.] [ Initiated by signal.] [Caused by error error_code: 'error_message(error_classification). error_status'. [(extra info extra_code)]]	The node has been forcibly shut down. The action (one of restarting, no start, or initial) subsequently being taken, if any, is also reported. If the shutdown occurred while the node was starting, the report includes the start_phase during which the node failed. If this was a result of a signal sent to the node, this information is also provided (see Operations and Signals, for more information). If the error causing the failure is known, this is also included; for more information about NDB error messages and classifications, see NDB Cluster API Errors.	NDBStopForced	StartUp	1	ALERT
Node node_id: Node shutdown aborted	The node shutdown process was aborted by the user.	NDBStopAborted	StartUp	1	INFO
Node node_id: StartLog: [GCI Keep: keep_pos LastCompleted: last_pos NewestRestorable: restore_pos]	This reports global checkpoints referenced during a node start. The redo log prior to keep_pos is dropped. last_pos is the last global checkpoint in which data node the participated; restore_pos is the global checkpoint which is actually used to restore all data nodes.	StartREDOLog	StartUp	4	INFO
startup_message [Listed separately; see below.]	There are a number of possible startup messages that can be logged under different circumstances. These are listed separately; see Section 25.6.2.2, “NDB Cluster Log Startup Messages”.	StartReport	StartUp	4	INFO
Node node_id: Node restart completed copy of dictionary information	Copying of data dictionary information to the restarted node has been completed.	NR_CopyDict	NodeRestart	8	INFO
Node node_id: Node restart completed copy of distribution information	Copying of data distribution information to the restarted node has been completed.	NR_CopyDistr	NodeRestart	8	INFO
Node node_id: Node restart starting to copy the fragments to Node node_id	Copy of fragments to starting data node node_id has begun	NR_CopyFragsStarted	NodeRestart	8	INFO
Node node_id: Table ID = table_id, fragment ID = fragment_id have been copied to Node node_id	Fragment fragment_id from table table_id has been copied to data node node_id	NR_CopyFragDone	NodeRestart	10	INFO
Node node_id: Node restart completed copying the fragments to Node node_id	Copying of all table fragments to restarting data node node_id has been completed	NR_CopyFragsCompleted	NodeRestart	8	INFO
Node node_id: Node node1_id completed failure of Node node2_id	Data node node1_id has detected the failure of data node node2_id	NodeFailCompleted	NodeRestart	8	ALERT
All nodes completed failure of Node node_id	All (remaining) data nodes have detected the failure of data node node_id	NodeFailCompleted	NodeRestart	8	ALERT
Node failure of node_idblock completed	The failure of data node node_id has been detected in the blockNDB kernel block, where block is 1 of DBTC, DBDICT, DBDIH, or DBLQH; for more information, see NDB Kernel Blocks	NodeFailCompleted	NodeRestart	8	ALERT
Node mgm_node_id: Node data_node_id has failed. The Node state at failure was state_code	A data node has failed. Its state at the time of failure is described by an arbitration state code state_code: possible state code values can be found in the file include/kernel/signaldata/ArbitSignalData.hpp.	NODE_FAILREP	NodeRestart	8	ALERT
President restarts arbitration thread [state=state_code] or Prepare arbitrator node node_id [ticket=ticket_id] or Receive arbitrator node node_id [ticket=ticket_id] or Started arbitrator node node_id [ticket=ticket_id] or Lost arbitrator node node_id - process failure [state=state_code] or Lost arbitrator node node_id - process exit [state=state_code] or Lost arbitrator node node_id - error_message [state=state_code]	This is a report on the current state and progress of arbitration in the cluster. node_id is the node ID of the management node or SQL node selected as the arbitrator. state_code is an arbitration state code, as found in include/kernel/signaldata/ArbitSignalData.hpp. When an error has occurred, an error_message, also defined in ArbitSignalData.hpp, is provided. ticket_id is a unique identifier handed out by the arbitrator when it is selected to all the nodes that participated in its selection; this is used to ensure that each node requesting arbitration was one of the nodes that took part in the selection process.	ArbitState	NodeRestart	6	INFO
Arbitration check lost - less than 1/2 nodes left or Arbitration check won - all node groups and more than 1/2 nodes left or Arbitration check won - node group majority or Arbitration check lost - missing node group or Network partitioning - arbitration required or Arbitration won - positive reply from node node_id or Arbitration lost - negative reply from node node_id or Network partitioning - no arbitrator available or Network partitioning - no arbitrator configured or Arbitration failure - error_message [state=state_code]	This message reports on the result of arbitration. In the event of arbitration failure, an error_message and an arbitration state_code are provided; definitions for both of these are found in include/kernel/signaldata/ArbitSignalData.hpp.	ArbitResult	NodeRestart	2	ALERT
Node node_id: GCP Take over started	This node is attempting to assume responsibility for the next global checkpoint (that is, it is becoming the master node)	GCP_TakeoverStarted	NodeRestart	7	INFO
Node node_id: GCP Take over completed	This node has become the master, and has assumed responsibility for the next global checkpoint	GCP_TakeoverCompleted	NodeRestart	7	INFO
Node node_id: LCP Take over started	This node is attempting to assume responsibility for the next set of local checkpoints (that is, it is becoming the master node)	LCP_TakeoverStarted	NodeRestart	7	INFO
Node node_id: LCP Take over completed	This node has become the master, and has assumed responsibility for the next set of local checkpoints	LCP_TakeoverCompleted	NodeRestart	7	INFO
Node node_id: Trans. Count = transactions, Commit Count = commits, Read Count = reads, Simple Read Count = simple_reads, Write Count = writes, AttrInfo Count = AttrInfo_objects, Concurrent Operations = concurrent_operations, Abort Count = aborts, Scans = scans, Range scans = range_scans	This report of transaction activity is given approximately once every 10 seconds	TransReportCounters	Statistic	8	INFO
Node node_id: Operations=operations	Number of operations performed by this node, provided approximately once every 10 seconds	OperationReportCounters	Statistic	8	INFO
Node node_id: Table with ID = table_id created	A table having the table ID shown has been created	TableCreated	Statistic	7	INFO
Node node_id: Mean loop Counter in doJob last 8192 times = count		JobStatistic	Statistic	9	INFO
Mean send size to Node = node_id last 4096 sends = bytes bytes	This node is sending an average of bytes bytes per send to node node_id	SendBytesStatistic	Statistic	9	INFO
Mean receive size to Node = node_id last 4096 sends = bytes bytes	This node is receiving an average of bytes of data each time it receives data from node node_id	ReceiveBytesStatistic	Statistic	9	INFO
Node node_id: Data usage is data_memory_percentage% (data_pages_used 32K pages of total data_pages_total) / Node node_id: Index usage is index_memory_percentage% (index_pages_used 8K pages of total index_pages_total)	This report is generated when a DUMP 1000 command is issued in the cluster management client	MemoryUsage	Statistic	5	INFO
Node node1_id: Transporter to node node2_id reported error error_code: error_message	A transporter error occurred while communicating with node node2_id; for a listing of transporter error codes and messages, see NDB Transporter Errors, in MySQL NDB Cluster Internals Manual	TransporterError	Error	2	ERROR
Node node1_id: Transporter to node node2_id reported error error_code: error_message	A warning of a potential transporter problem while communicating with node node2_id; for a listing of transporter error codes and messages, see NDB Transporter Errors, for more information	TransporterWarning	Error	8	WARNING
Node node1_id: Node node2_id missed heartbeat heartbeat_id	This node missed a heartbeat from node node2_id	MissedHeartbeat	Error	8	WARNING
Node node1_id: Node node2_id declared dead due to missed heartbeat	This node has missed at least 3 heartbeats from node node2_id, and so has declared that node “dead”	DeadDueToHeartbeat	Error	8	ALERT
Node node1_id: Node Sent Heartbeat to node = node2_id	This node has sent a heartbeat to node node2_id	SentHeartbeat	Info	12	INFO
Node node_id: Event buffer status (object_id): used=bytes_used (percent_used% of alloc) alloc=bytes_allocated max=bytes_available latest_consumed_epoch=latest_consumed_epoch latest_buffered_epoch=latest_buffered_epoch report_reason=report_reason	This report is seen during heavy event buffer usage, for example, when many updates are being applied in a relatively short period of time; the report shows the number of bytes and the percentage of event buffer memory used, the bytes allocated and percentage still available, and the latest buffered and consumed epochs; for more information, see Section 25.6.2.3, “Event Buffer Reporting in the Cluster Log”	EventBufferStatus2	Info	7	INFO
Node node_id: Entering single user mode, Node node_id: Entered single user mode Node API_node_id has exclusive access, Node node_id: Entering single user mode	These reports are written to the cluster log when entering and exiting single user mode; API_node_id is the node ID of the API or SQL having exclusive access to the cluster (for more information, see Section 25.6.6, “NDB Cluster Single User Mode”); the message Unknown single user report API_node_id indicates an error has taken place and should never be seen in normal operation	SingleUser	Info	7	INFO
Node node_id: Backup backup_id started from node mgm_node_id	A backup has been started using the management node having mgm_node_id; this message is also displayed in the cluster management client when the START BACKUP command is issued; for more information, see Section 25.6.8.2, “Using The NDB Cluster Management Client to Create a Backup”	BackupStarted	Backup	7	INFO
Node node_id: Backup backup_id started from node mgm_node_id completed. StartGCP: start_gcp StopGCP: stop_gcp #Records: records #LogRecords: log_records Data: data_bytes bytes Log: log_bytes bytes	The backup having the ID backup_id has been completed; for more information, see Section 25.6.8.2, “Using The NDB Cluster Management Client to Create a Backup”	BackupCompleted	Backup	7	INFO
Node node_id: Backup request from mgm_node_id failed to start. Error: error_code	The backup failed to start; for error codes, see MGM API Errors	BackupFailedToStart	Backup	7	ALERT
Node node_id: Backup backup_id started from mgm_node_id has been aborted. Error: error_code	The backup was terminated after starting, possibly due to user intervention	BackupAborted	Backup	7	ALERT