This section contains unified change history highlights for all
MySQL Cluster releases based on version 7.2 of the
NDB storage engine through MySQL
Cluster NDB 7.2.18. Included are all changelog
entries in the categories MySQL Cluster,
Disk Data, and Cluster
For an overview of features that were added in MySQL Cluster NDB 7.2, see MySQL Cluster Development History.
Version 5.5.39-ndb-7.2.18 has no changelog entries.
Functionality Added or Changed
Added as an aid to debugging the ability to specify a
human-readable name for a given
Ndb object and later to
retrieve it. These operations are implemented, respectively, as
To make tracing of event handling between a user application and
NDB easier, you can use the reference (from
getReference() followed by
the name (if provided) in printouts; the reference ties together
Ndb object, the event buffer,
NDB storage engine's
Processing a NODE_FAILREP signal that contained an invalid node ID could cause a data node to fail. (Bug #18993037, Bug #73015)
References: This bug is a regression of Bug #16007980.
When building out of source, some files were written to the
source directory instead of the build dir. These included the
manifest.mf files used for creating
ClusterJ jars and the
pom.xml file used by
mvn_install_ndbjtie.sh. In addition,
ndbinfo.sql was written to the build
directory, but marked as output to the source directory in
(Bug #18889568, Bug #72843)
TABLE ... REORGANIZE PARTITION after increasing the
number of data nodes in the cluster from 4 to 16 led to a crash
of the data nodes. This issue was shown to be a regression
caused by previous fix which added a new dump handler using a
dump code that was already in use (7019), which caused the
command to execute two different handlers with different
semantics. The new handler was assigned a new
DUMP code (7024).
References: This bug is a regression of Bug #14220269.
ndbmtd supports multiple parallel receiver
threads, each of which performs signal reception for a subset of
the remote node connections (transporters) with the mapping of
remote_nodes to receiver threads decided at node startup.
Connection control is managed by the multi-instance
TRPMAN block, which is organized as a proxy
and workers, and each receiver thread has a
TRPMAN worker running locally.
QMGR block sends signals to
TRPMAN to enable and disable communications
with remote nodes. These signals are sent to the
TRPMAN proxy, which forwards them to the
workers. The workers themselves decide whether to act on
signals, based on the set of remote nodes they manage.
The current isuue arises because the mechanism used by the
TRPMAN workers for determining which
connections they are responsible for was implemented in such a
way that each worker thought it was responsible for all
connections. This resulted in the
CLOSE_COMREQ being processed multiple times.
The fix keeps
TRPMAN instances (receiver
CLOSE_COMREQ requests. In addition, the
TRPMAN instance is now chosen when
routing from this instance for a specific remote connection.
When running with a very slow main thread, and one or more
transaction coordinator threads, on different CPUs, it was
possible to encounter a timeout when sending a
DIH_SCAN_GET_NODESREQ signal, which could
lead to a crash of the data node. Now in such cases the timeout
During data node failure handling, the transaction coordinator
performing takeover gathers all known state information for any
failed TC instance transactions, determines whether each
transaction has been committed or aborted, and informs any
involved API nodes so that they can report this accurately to
their clients. The TC instance provides this information by
TCKEY_FAILCONF signals to the API nodes as
appropriate top each affected transaction.
In the event that this TC instance does not have a direct
connection to the API node, it attempts to deliver the signal by
routing it through another data node in the same node group as
the failing TC, and sends a
GSN_TCKEY_FAILREFCONF_R signal to TC block
instance 0 in that data node. A problem arose in the case of
multiple transaction cooridnators, when this TC instance did not
have a signal handler for such signals, which led it to fail.
This issue has been corrected by adding a handler to the TC proxy block which in such cases forwards the signal to one of the local TC worker instances, which in turn attempts to forward the signal on to the API node. (Bug #18455971)
A local checkpoint (LCP) is tracked using a global LCP state
c_lcpState), and each
NDB table has a status indicator
which indicates the LCP status of that table
tabLcpStatus). If the global LCP state is
LCP_STATUS_IDLE, then all the tables should
have an LCP status of
When an LCP starts, the global LCP status is
LCP_INIT_TABLES and the thread starts setting
NDB tables to
TLS_ACTIVE. If any tables are not ready for
LCP, the LCP initialization procedure continues with
CONTINUEB signals until all tables have
become available and been marked
When this initialization is complete, the global LCP status is
This bug occurred when the following conditions were met:
An LCP was in the
and some but not all tables had been set to
The master node failed before the global LCP state changed
LCP_STATUS_ACTIVE; that is, before the
LCP could finish processing all tables.
NODE_FAILREP signal resulting from
the node failure was processed before the final
CONTINUEB signal from the LCP
initialization process, so that the node failure was
processed while the LCP remained in the
Following master node failure and selection of a new one, the
new master queries the remaining nodes with a
MASTER_LCPREQ signal to determine the state
of the LCP. At this point, since the LCP status was
LCP_INIT_TABLES, the LCP status was reset to
LCP_STATUS_IDLE. However, the LCP status of
the tables was not modified, so there remained tables with
TLS_ACTIVE. Afterwards, the failed node is
removed from the LCP. If the LCP status of a given table is
TLS_ACTIVE, there is a check that the global
LCP status is not
LCP_STATUS_IDLE; this check
failed and caused the data node to fail.
MASTER_LCPREQ handler ensures that
tabLcpStatus for all tables is updated to
TLS_COMPLETED when the global LCP status is
When performing a copying
TABLE operation, mysqld creates a
new copy of the table to be altered. This intermediate table,
which is given a name bearing the prefix
#sql-, has an updated schema but contains no
data. mysqld then copies the data from the
original table to this intermediate table, drops the original
table, and finally renames the intermediate table with the name
of the original table.
mysqld regards such a table as a temporary
table and does not include it in the output from
mysqldump also ignores an intermediate table.
NDB sees no difference
between such an intermediate table and any other table. This
difference in how intermediate tables are viewed by
mysqld (and MySQL client programs) and by the
NDB storage engine can give rise to problems
when performing a backup and restore if an intermediate table
NDB, possibly left over from a
ALTER TABLE that used copying. If a
schema backup is performed using mysqldump
and the mysql client, this table is not
included. However, in the case where a data backup was done
using the ndb_mgm client's
BACKUP command, the intermediate table was
included, and was also included by
ndb_restore, which then failed due to
attempting to load data into a table which was not defined in
the backed up schema.
To prevent such failures from occurring,
ndb_restore now by default ignores
intermediate tables created during
TABLE operations (that is, tables whose names begin
with the prefix
#sql-). A new option
is added that makes it possible to override the new behavior.
The option's default value is
cause ndb_restore to revert to the old
behavior and to attempt to restore intermediate tables, set this
The logging of insert failures has been improved. This is
intended to help diagnose occasional issues seen when writing to
CHAR column that used
UTF8 character set as a table's
primary key column led to node failure when restarting data
nodes. Attempting to restore a table with such a primary key
also caused ndb_restore to fail.
(Bug #16895311, Bug #68893)
-o) option for the
ndb_select_all utility worked only when
specified as the last option, and did not work with an equals
As part of this fix, the program's
output was also aligned with the
option's correct behavior.
(Bug #64426, Bug #16374870)
NDB data node indicates a
buffer overflow via an empty epoch, the event buffer places an
inconsistent data event in the event queue. When this was
consumed, it was not removed from the event queue as expected,
nextEvent() calls to return
0. This caused event consumption to stall because the
inconsistency remained flagged forever, while event data
accumulated in the queue.
Event data belonging to an empty inconsistent epoch can be found
either at the beginning or somewhere in the middle.
pollEvents() returns 0 for
the first case. This fix handles the second case: calling
nextEvent() call dequeues the inconsistent
event before it returns. In order to benefit from this fix, user
applications must call
nextEvent() even when
pollEvents() returns 0.
returned 1, even when called with a wait time equal to 0, and
there were no events waiting in the queue. Now in such cases it
returns 0 as expected.
Functionality Added or Changed
shortages and statistics has been improved as follows:
The default value of
has been increased from 4 MB to 64 MB.
When this resource is exhausted, a suitable informative message is now printed in the data node log describing possible causes of the problem and suggesting possible solutions.
LongMessageBuffer usage information is
now shown in the
See the description of this table for an example and
The server system variables
ndb_index_stat_freq, which had been
deprecated in a previous MySQL Cluster release series, have now
(Bug #11746486, Bug #26673)
ALTER TABLE statement
changed table schemas without causing a change in the
table's partitioning, the new table definition did not copy
the hash map from the old definition, but used the current
default hash map instead. However, the table data was not
reorganized according to the new hashmap, which made some rows
inaccessible using a primary key lookup if the two hash maps had
To keep this situation from occurring, any
TABLE that entails a hashmap change now triggers a
reorganisation of the table. In addition, when copying a table
definition in such cases, the hashmap is now also copied.
Checking of timeouts is handled by the signal
TIME_SIGNAL. Previously, this signal was
generated by the
NDB kernel block in the main
thread, and sent to the
NDB Kernel Blocks) as needed to
check (respectively) heartbeats, disk writes, and transaction
timeouts. In ndbmtd (as opposed to
ndbd), these blocks all execute in different
threads. This meant that if, for example,
QMGR was actively working and some other
thread was put to sleep, the previously sleeping thread received
a large number of TIME_SIGNAL messages
simultaneously when it was woken up again, with the effect that
effective times moved very quickly in DBLQH
as well as in DBTC. In
DBLQH, this had no noticeable adverse
effects, but this was not the case in DBTC;
the latter block could not work on transactions even though time
was still advancing, leading to a situation in which many
operations appeared to time out because the transaction
coordinator (TC) thread was comparatively slow in answering
In addition, when the TC thread slept for longer than 1500
milliseconds, the data node crashed due to detecting that the
timeout handling loop had not yet stopped. To rectify this
problem, the generation of the
has been moved into the local threads instead of
QMGR; this provides for better control over
TIME_SIGNAL messages are allowed
When certain queries generated signals having more than 18 data words prior to a node failure, such signals were not written correctly in the trace file. (Bug #18419554)
-DWITH_NDBMTD=0 did not function
correctly, which could cause the build to fail on platforms such
as ARM and Raspberry Pi which do not define the memory barrier
functions required to compile ndbmtd.
References: See also Bug #16620938.
ndb_show_tables sometimes failed with the
error message Unable to connect to management
server and immediately terminated, without providing
the underlying reason for the failure. To provide more useful
information in such cases, this program now also prints the most
recent error from the
used to instantiate the connection.
After dropping an
neither the cluster log nor the output of the
MemoryUsage command showed that the
IndexMemory used by that
table had been freed, even though the memory had in fact been
deallocated. This issue was introduced in MySQL Cluster NDB
The block threads managed by the multi-threading scheduler communicate by placing signals in an out queue or job buffer which is set up between all block threads. This queue has a fixed maximum size, such that when it is filled up, the worker thread must wait for the consumer to drain the queue. In a highly loaded system, multiple threads could end up in a circular wait lock due to full out buffers, such that they were preventing each other from performing any useful work. This condition eventually led to the data node being declared dead and killed by the watchdog timer.
To fix this problem, we detect situations in which a circular wait lock is about to begin, and cause buffers which are otherwise held in reserve to become available for signal processing by queues which are highly loaded. (Bug #18229003)
The ndb_mgm client
BACKUP command (see
Commands in the MySQL Cluster Management Client) could
experience occasional random failures when a ping was received
prior to an expected
Now the connection established by this command is not checked
until it has been properly set up.
Data nodes running ndbmtd could stall while performing an online upgrade of a MySQL Cluster containing a great many tables from a version prior to NDB 7.2.5 to version 7.2.5 or later. (Bug #16693068)
When an NDB API client application received a signal with an
invalid block or signal number,
only a very brief error message that did not accurately convey
the nature of the problem. Now in such cases, appropriate
printouts are provided when a bad signal or message is detected.
In addition, the message length is now checked to make certain
that it matches the size of the embedded signal.
Refactoring that was performed in MySQL Cluster NDB 7.2.15
inadvertently introduced a dependency in
Ndb.hpp on a file that is not included in
the distribution, which caused NDB API applications to fail to
compile. The dependency has been removed.
(Bug #18293112, Bug #71803)
References: This bug was introduced by Bug #17647637.
An NDB API application sends a scan query to a data node; the
scan is processed by the transaction coordinator (TC). The TC
LQHKEYREQ request to the
appropriate LDM, and aborts the transaction if it does not
LQHKEYCONF response within the
specified time limit. After the transaction is successfully
aborted, the TC sends a
TCROLLBACKREP to the
NDBAPI client, and the NDB API client processes this message by
cleaning up any
associated with the transaction.
The client receives the data which it has requested in the form
TRANSID_AI signals, buffered for sending
at the data node, and may be delivered after a delay. On
receiving such a signal,
NDB checks the
transaction state and ID: if these are as expected, it processes
the signal using the
Ndb objects associated
with that transaction.
The current bug occurs when all the following conditions are fulfilled:
The transaction coordinator aborts a transaction due to
delays and sends a
to the client, while at the same time a
TRANSID_AI which has been buffered for
delivery at an LDM is delivered to the same client.
The NDB API client considers the transaction complete on
receipt of a
TCROLLBACKREP signal, and
immediately closes the transaction.
The client has a separate receiver thread running concurrently with the thread that is engaged in closing the transaction.
The arrival of the late
interleaves with the closing of the user thread's
transaction such that
processing passes normal checks before
resets the transaction state and invalidates the receiver.
When these conditions are all met, the receiver thread proceeds
to continue working on the
using the invalidated receiver. Since the receiver is already
invalidated, its usage results in a node failure.
Ndb object cleanup done for
TCROLLBACKREP includes invalidation of the
transaction ID, so that, for a given transaction, any signal
which is received after the
arrives does not pass the transaction ID check and is silently
dropped. This fix is also implemented for the
TCKEY_FAILREF signals as well.
See also Operations and Signals, for additional information about NDB messaging. (Bug #18196562)
included an internal header file
ndb_global.h) not found in the MySQL
Cluster binary distribution. The example now uses
instead of this file.
(Bug #18096866, Bug #71409)
Cluster API: ndb_restore could sometimes report Error 701 System busy with other schema operation unnecessarily when restoring in parallel. (Bug #17916243)
Compilation of ndbmtd failed on Solaris 10
and 11 for 32-bit
x86, and the binary was not
included in the binary distributions for these platforms.
Disk Data: When using Disk Data tables and ndbmtd data nodes, it was possible for the undo buffer to become overloaded, leading to a crash of the data nodes. This issue was more likely to be encountered when using Disk Data columns whose size was approximately 8K or larger. (Bug #16766493)
UINT_MAX64 was treated as a signed value by
Visual Studio 2010. To prevent this from happening, the value is
now explicitly defined as unsigned.
References: See also Bug #17647637.
Monotonic timers on several platforms can experience issues which might result in the monotonic clock doing small jumps back in time. This is due to imperfect synchronization of clocks between multiple CPU cores and does not normally have an adverse effect on the scheduler and watchdog mechanisms; so we handle some of these cases by making backtick protection less strict, although we continue to ensure that the backtick is less than 10 milliseconds. This fix also removes several checks for backticks which are thereby made redundant. (Bug #17973819)
Under certain specific circumstances, in a cluster having two SQL nodes, one of these could hang, and could not be accessed again even after killing the mysqld process and restarting it. (Bug #17875885)
References: See also Bug #17934985.
In some cases, with
ndb_join_pushdown enabled, it
was possible to obtain from a valid query the error
Got error 290 'Corrupt key in TC, unable to xfrm'
from NDBCLUSTER even though the data was not
It was determined that a
NULL in a
VARCHAR column could be used to
construct a lookup key, but since
never equal to any other value, such a lookup could simple have
been eliminated instead. This
NULL lookup in
turn led to the spurious error message.
This fix takes advantage of the fact that a key lookup with
NULL never finds any matching rows, and so
NDB does not try to perform the
lookup that would have led to the error.
Poor support or lack of support on some platforms for monotonic timers caused issues with delayed signal handling by the job scheduler for the multithreaded data node. Variances (timer leaps) on such platforms are now handled in the same way the multithreaded data node process that they are by the singlethreaded version. (Bug #17857442)
References: See also Bug #17475425, Bug #17647637.
Use of the
localtime() function in
NDB multithreading code led to
otherwise nondeterministic failures in
ndbmtd. This fix replaces this function,
which on many platforms uses a buffer shared among multiple
localtime_r(), which can have
allocated to it a buffer of its own.
It was theoretically possible in certain cases for a number of
output functions internal to the
NDB code to supply an uninitialized
buffer as output. Now in such cases, a newline character is
(Bug #17775602, Bug #17775772)
When using single-threaded (ndbd) data nodes
enabled, the CPU did not, as intended, temporarily lower its
scheduling priority to normal every 10 milliseconds to give
other, non-realtime threads a chance to run.
During arbitrator selection,
QMGR Block) runs through
a series of states, the first few of which are (in order)
START. A check
for an arbitration selection timeout occurred in the
FIND state, even though the corresponding
timer was not set until
QMGR reached the
Attempting to read the resulting uninitialized timestamp value
could lead to false Could not find an arbitrator,
cluster is not partition-safe warnings.
This fix moves the setting of the timer for arbitration timeout
INIT state, so that the value later
FIND is always initialized.
The global checkpoint lag watchdog tracking the number of times a check for GCP lag was performed using the system scheduler and used this count to check for a timeout condition, but this caused a number of issues. To overcome these limitations, the GCP watchdog has been refactored to keep track of its own start times, and to calculate elapsed time by reading the (real) clock every time it is called.
In addition, any backticks (rare in any case) are now handled by taking the backward time as the new current time and calculating the elapsed time for this round as 0. Finally, any ill effects of a forward leap, which possibly could expire the watchdog timer immediately, are reduced by never calculating an elapsed time longer than the requested delay time for the watchdog timer. (Bug #17647469)
References: See also Bug #17842035.
Timers used in timing scheduler events in the
NDB kernel have been refactored, in
part to insure that they are monotonic on all platforms. In
particular, on Windows, event intervals were previously
calculated using values obtained from
GetSystemTimeAsFileTime(), which reads
directly from the system time (“wall clock”), and
which may arbitrarily be reset backward or forward, leading to
false watchdog or heartbeat alarms, or even node shutdown. Lack
of timer monotonicity could also cause slow disk writes during
backups and global checkpoints. To fix this issue, the Windows
implementation now uses
QueryPerformanceCounters() instead of
GetSystemTimeAsFileTime(). In the event that
a monotonic timer is not found on startup of the data nodes, a
warning is logged.
In addition, on all platforms, a check is now performed at
compile time for available system monotonic timers, and the
build fails if one cannot be found; note that
CLOCK_HIGHRES is now supported as an
CLOCK_MONOTONIC if the latter
is not available.
The length of the interval (intended to be 10 seconds) between
GCP_COMMIT when the GCP progress
watchdog did not detect progress in a global checkpoint was not
always calculated correctly.
In certain rare cases on commit of a transaction, an
Ndb object was released before
the transaction coordinator (
block) sent the expected
NDB failed to send a
COMMIT_ACK signal in response, which caused a
memory leak in the
NDB kernel could later
lead to node failure.
Ndb object is not released until the
COMMIT_CONF signal has actually been
After restoring the database metadata (but not any data) by
-m), SQL nodes would hang while trying to
SELECT from a table in the
database to which the metadata was restored. In such cases the
attempt to query the table now fails as expected, since the
table does not actually exist until
ndb_restore is executed with
Losing its connections to the management node or data nodes
while a query against the
ndbinfo.memoryusage table was
in progress caused the SQL node where the query was issued to
(Bug #14483440, Bug #16810415)
The ndbd_redo_log_reader utility now supports
Using this options causes the program to print basic usage
information, and then to exit.
(Bug #11749591, Bug #36805)
It was possible for an
object to receive signals for handling before it was
initialized, leading to thread interleaving and possible data
node failure when executing a call to
Ndb::init(). To guard against
this happening, a check is now made when it is starting to
receive signals that the
Ndb object is
properly initialized before any signals are actually handled.
Cluster API: Compilation of example NDB API program files failed due to missing include directives. (Bug #17672846, Bug #70759)
Functionality Added or Changed
ExtraSendBufferMemory parameter for
management nodes and API nodes. (Formerly, this parameter was
available only for configuring data nodes.) See
(management nodes), and
(API nodes), for more information.
Performance: In a number of cases found in various locations in the MySQL Cluster codebase, unnecessary iterations were performed; this was caused by failing to break out of a repeating control structure after a test condition had been met. This community-contributed fix removes the unneeded repetitions by supplying the missing breaks. (Bug #16904243, Bug #69392, Bug #16904338, Bug #69394, Bug #16778417, Bug #69171, Bug #16778494, Bug #69172, Bug #16798410, Bug #69207, Bug #16801489, Bug #69215, Bug #16904266, Bug #69393)
The MySQL Cluster installer for Windows provided a nonfunctional
option to install debug symbols (contained in
*.pdb files). This option has been removed
from the installer.
You can obtain the
*.pdb debug files for
a given MySQL Cluster release from the Windows
.zip archive for the same release, such
(Bug #16748308, Bug #69112)
A fix has been made in this release correcting a number of
issues found in some recent
MySQL-Cluster-server RPM packages, including
dependencies on the
mysql-server package and
conflicts with the
mysql-libs package needed
by other common applications. Platforms known to have been
affected by these issues include CentOS 6, Red Hat Enterprise
Linux 6, and Oracle Linux 6.
(Bug #14063590, Bug #14181419, Bug #65534)
The MySQL Cluster Windows installer attempted to use the wrong
path to the
my.ini file and the executables
(Bug #13813120, Bug #64510)
ndb_restore could abort during the last stages of a restore using attribute promotion or demotion into an existing table. This could happen if a converted attribute was nullable and the backup had been run on active database. (Bug #17275798)
DBUTIL data node block is now less strict
about the order in which it receives certain messages from other
not work correctly with data nodes running
The Windows error ERROR_FILE_EXISTS was
not recognized by
treated it as an unknown error.
File system errors occurring during a local checkpoint could sometimes cause an LCP to hang with no obvious cause when they were not handled correctly. Now in such cases, such errors always cause the node to fail. Note that the LQH block always shuts down the node when a local checkpoint fails; the change here is to make likely node failure occur more quickly and to make the original file system error more visible. (Bug #16961443)
DROP TABLE while
DBDIH was updating table checkpoint
information subsequent to a node failure could lead to a data
Maintenance and checking of parent batch completion in the
SPJ block of the
kernel was reimplemented. Among other improvements, the
completion state of all ancestor nodes in the tree are now
The planned or unplanned shutdown of one or more data nodes
while reading table data from the
ndbinfo database caused a
mysql_upgrade failed when upgrading from
MySQL Cluster NDB 7.1.26 to MySQL Cluster NDB 7.2.13 when it
attempted to invoke a stored procedure before the
mysql.proc table had been upgraded.
References: This bug is a regression of Bug #16226274.
Use of an uninitialized variable employed in connection with
error handling in the
DBLQH kernel block
could sometimes lead to a data node crash or other stability
issues for no apparent reason.
In certain cases, when starting a new SQL node, mysqld failed with Error 1427 Api node died, when SUB_START_REQ reached node. (Bug #16840741)
A race condition in the time between the reception of a
execNODE_FAILREP signal by the
QMGR kernel block and its reception by the
blocks could lead to data node crashes during shutdown.
CLUSTERLOG command (see
Commands in the MySQL Cluster Management Client) caused
ndb_mgm to crash on Solaris SPARC systems.
Failure to use container classes specific
NDB during node failure handling
could cause leakage of commit-ack markers, which could later
lead to resource shortages or additional node crashes.
The NDB Error-Reporting Utility
(ndb_error_reporter) failed to include the
cluster nodes' log files in the archive it produced when the
FILE option was set for the parameter
References: See also Bug #11752792, Bug #44082.
The LCP fragment scan watchdog periodically checks for lack of
progress in a fragment scan performed as part of a local
checkpoint, and shuts down the node if there is no progress
after a given amount of time has elapsed. This interval,
formerly hard-coded as 60 seconds, can now be configured using
data node configuration parameter added in this release.
This configuration parameter sets the maximum time the local checkpoint can be stalled before the LCP fragment scan watchdog shuts down the node. The default is 60 seconds, which provides backward compatibility with previous releases.
You can disable the LCP fragment scan watchdog by setting this parameter to 0. (Bug #16630410)
When trying to specify a backup ID greater than the maximum allowed, the value was silently truncated. (Bug #16585455, Bug #68796)
Added the ndb_error_reporter options
which makes it possible to set a timeout for connecting to
which disables scp connections to remote hosts, and
which skips all nodes in a given node group.
References: See also Bug #11752792, Bug #44082.
ndb_mgm treated backup IDs provided to
ABORT BACKUP commands as signed values, so
that backup IDs greater than 231
wrapped around to negative values. This issue also affected
out-of-range backup IDs, which wrapped around to negative values
instead of causing errors as expected in such cases. The backup
ID is now treated as an unsigned value, and
ndb_mgm now performs proper range checking
for backup ID values greater than
(Bug #16585497, Bug #68798)
id WAIT STARTED
id had already been used for a backup
ID, an error caused by the duplicate ID occurred as expected,
but following this, the
START BACKUP command
(Bug #16593604, Bug #68854)
The unexpected shutdown of another data node as a starting data node received its node ID caused the latter to hang in Start Phase 1. (Bug #16007980)
References: See also Bug #18993037.
NDB receive thread waited unnecessarily
for additional job buffers to become available when receiving
data. This caused the receive mutex to be held during this wait,
which could result in a busy wait when the receive thread was
running with real-time priority.
This fix also handles the case where a negative return value
from the initial check of the job buffer by the receive thread
prevented further execution of data reception, which could
possibly lead to communication blockage or configured
When the available job buffers for a given thread fell below the critical threshold, the internal multi-threading job scheduler waited for job buffers for incoming rather than outgoing signals to become available, which meant that the scheduler waited the maximum timeout (1 millisecond) before resuming execution. (Bug #15907122)
Under some circumstances, a race occurred where the wrong
watchdog state could be reported. A new state name
Packing Send Buffers is added for watchdog
state number 11, previously reported as
place. As part of this fix, the state numbers for
states without names are always now reported in such cases.
Creating more than 32 hash maps caused data nodes to fail.
Usually new hashmaps are created only when performing
reorganzation after data nodes have been added or when explicit
partitioning is used, such as when creating a table with the
MAX_ROWS option, or using
BY KEY() PARTITIONS .
When a node fails, the Distribution Handler
DBDIH kernel block) takes steps together
with the Transaction Coordinator (
make sure that all ongoing transactions involving the failed
node are taken over by a surviving node and either committed or
aborted. Transactions taken over which are then committed belong
in the epoch that is current at the time the node failure
occurs, so the surviving nodes must keep this epoch available
until the transaction takeover is complete. This is needed to
maintain ordering between epochs.
A problem was encountered in the mechanism intended to keep the current epoch open which led to a race condition between this mechanism and that normally used to declare the end of an epoch. This could cause the current epoch to be closed prematurely, leading to failure of one or more surviving data nodes. (Bug #14623333, Bug #16990394)
When performing an
ON DUPLICATE KEY UPDATE on an
NDB table where the row to be
inserted already existed and was locked by another transaction,
the error message returned from the
following the timeout was Transaction already
aborted instead of the expected Lock wait
(Bug #14065831, Bug #65130)
When using dynamic listening ports for accepting connections from API nodes, the port numbers were reported to the management server serially. This required a round trip for each API node, causing the time required for data nodes to connect to the management server to grow linearly with the number of API nodes. To correct this problem, each data node now reports all dynamic ports at once. (Bug #12593774)
Formerly, the node used as the coordinator or leader for
distributed decision making between nodes (also known as the
DBDICT Block) was
indicated in the output of the ndb_mgm client
SHOW command as the “master”
node, although this node has no relationship to a master server
in MySQL Replication. (It should also be noted that it is not
necessary to know which node is the leader except when debugging
NDBCLUSTER source code.) To avoid possible
confusion, this label has been removed, and the leader node is
now indicated in
SHOW command output using an
(Bug #11746263, Bug #24880)
ndb_error-reporter did not support the
(Bug #11756666, Bug #48606)
References: See also Bug #11752792, Bug #44082.
START BACKUP WAIT STARTED was run from
the command line using ndb_mgm
the client did not exit until the backup completed.
(Bug #11752837, Bug #44146)
Program execution failed to break out of a loop after meeting a desired condition in a number of internal methods, performing unneeded work in all cases where this occurred. (Bug #69610, Bug #69611, Bug #69736, Bug #17030606, Bug #17030614, Bug #17160263)
ABORT BACKUP in the
ndb_mgm client (see
Commands in the MySQL Cluster Management Client) took an
excessive amount of time to return (approximately as long as the
backup would have required to complete, had it not been
aborted), and failed to remove the files that had been generated
by the aborted backup.
(Bug #68853, Bug #17719439)
Note that converted character data is not checked to conform to any character set.
When performing such promotions, the only other sort of type conversion that can be performed at the same time is between character types and binary types.
ALTER LOGFILE GROUP, and
ALTER TABLESPACE failed with a
syntax error when
INITIAL_SIZE was specified
using letter abbreviations such as
G. In addition,
LOGFILE GROUP failed when
UNDO_BUFFER_SIZE, or both options were
specified using letter abbreviations.
(Bug #13116514, Bug #16104705, Bug #62858)
For each log event retrieved using the MGM API, the log event
simply cast to an
enum type, which resulted
in invalid category values. Now an offset is added to the
category following the cast to ensure that the value does not
fall out of the allowed range.
This change was reverted by the fix for Bug #18354165. See the
MySQL Cluster API Developer documentation for
now supports a pointer or a reference to table as its required
argument. If a null table pointer is used, the method now
returns -1 to make it clear that this is what has occurred.
Functionality Added or Changed
DUMP code 2514, which provides
information about counts of transaction objects per API node.
For more information, see
DUMP 2514. See also
Commands in the MySQL Cluster Management Client.
When ndb_restore fails to find a table, it now includes in the error output an NDB API error code giving the reason for the failure. (Bug #16329067)
Incompatible Change; Cluster Replication:
The default value for the
binlog_format system variable
NDB storage engine is
enabled is now
NDB is not enabled, the default binary
logging format is
STATEMENT, as in the
standard MySQL Server.
The NDB Error-Reporting Utility
(ndb_error_reporter) failed to include the
cluster nodes' log files in the archive it produced when the
FILE option was set for the parameter
References: See also Bug #11752792, Bug #44082.
In some cases a data node could stop with an exit code but no
error message other than
(null) was logged.
(This could occur when using ndbd or
ndbmtd for the data node process.) Now in
such cases the appropriate error message is used instead (see
ndbd Error Messages).
WHERE condition that contained a boolean
test of the result of an
IN subselect was not
mysql_upgrade failed when run on an SQL node where distributed privileges were in use. (Bug #16226274)
Improved handling of lagging row change event subscribers by
setting size of the GCP pool to the value of
fix also introduces a new
data node configuration parameter, which makes it possible to
set a total number of bytes per node to be reserved for
buffering epochs. In addition, a new
code (8013) has been added which causes a list a lagging
subscribers for each node to be printed to the cluster log (see
Purging the binary logs could sometimes cause mysqld to crash. (Bug #15854719)
An error message in
(Bug #14548052, Bug #66518)
The help text for ndb_select_count did not include any information about using table names. (Bug #11755737, Bug #47551)
Functionality Added or Changed
The length of time a management node waits for a heartbeat
message from another management node is now configurable using
management node configuration parameter added in this release.
The connection is considered dead after 3 missed heartbeats. The
default value is 1500 milliseconds, or a timeout of
approximately 6000 ms.
(Bug #17807768, Bug #16426805)
Trying to access old rows while using
role=ndb-caching after restarting the
memached daemon caused the daemon to crash.
The current fix prevents the crash in such cases but does not
provide full support for
Functionality Added or Changed
Following an upgrade to MySQL Cluster NDB 7.2.7 or later, it was
not possible to downgrade online again to any previous version,
due to a change in that version in the default size (number of
LDM threads used) for
hash maps. The fix for this issue makes the size configurable,
with the addition of the
To retain compatibility with an older release that does not
support large hash maps, you can set this parameter in the
config.ini file to the value
used in older releases (240) before performing an upgrade, so
that the data nodes continue to use smaller hash maps that are
compatible with the older release. You can also now employ this
parameter in MySQL Cluster NDB 7.0 and MySQL Cluster NDB 7.1 to
enable larger hash maps prior to upgrading to MySQL Cluster NDB
7.2. For more information, see the description of the
References: See also Bug #14645319.
Important Change; Cluster API:
When checking—as part of evaluating an
if predicate—which error codes should
be propagated to the application, any error code less than 6000
caused the current row to be skipped, even those codes that
should have caused the query to be aborted. In addition, a scan
that aborted due to an error from
no rows had been sent to the API caused
to send a
SCAN_FRAGCONF signal rather than a
SCAN_FRAGREF signal to
DBTC. This caused
time out waiting for a
that was never sent, and the scan was never closed.
As part of this fix, the default
value used by
has been changed from 899 (Rowid already
allocated) to 626 (Tuple did not
exist). The old value continues to be supported for
backward compatibility. User-defined values in the range
6000-6999 (inclusive) are also now supported. You should also
keep in mind that the result of using any other
ErrorCode value not mentioned here is not
defined or guaranteed.
When using tables having more than 64 fragments in a MySQL
Cluster where multiple TC threads were configured (on data nodes
running ndbmtd, using
memory could be freed prematurely, before scans relying on these
objects could be completed, leading to a crash of the data node.
References: See also Bug #13799800. This bug was introduced by Bug #14143553.
When started with
-f) option, ndb_mgmd
removed the old configuration cache before verifying the
configuration file. Now in such cases,
ndb_mgmd first checks for the file, and
continues with removing the configuration cache only if the
configuration file is found and is valid.
DUMP 2304 command during a data
node restart could cause the data node to crash with a
Pointer too large error.
Including a table as a part of a pushed join should be rejected if there are outer joined tables in between the table to be included and the tables with which it is joined with; however the check as performed for any such outer joined tables did so by checking the join type against the root of the pushed query, rather than the common ancestor of the tables being joined. (Bug #16199028)
References: See also Bug #16198866.
Data nodes could fail during a system restart when the host ran
short of memory, due to signals of the wrong types
TRANSID_AI_R) being sent to the
DBSPJ kernel block.
Some queries were handled differently with
ndb_join_pushdown enabled, due
to the fact that outer join conditions were not always pruned
correctly from joins before they were pushed down.
References: See also Bug #16199028.
Attempting to perform additional operations such as
COLUMN as part of an
[ONLINE | OFFLINE] TABLE ... RENAME ... statement is
not supported, and now fails with an
Due to a known issue in the MySQL Server, it is possible to drop
PERFORMANCE_SCHEMA database. (Bug
#15831748) In addition, when executed on a MySQL Server acting
as a MySQL Cluster SQL node,
DATABASE caused this database to be dropped on all SQL
nodes in the cluster. Now, when executing a distributed drop of
NDB does not delete
tables that are local only. This prevents MySQL system databases
from being dropped in such cases.
When performing large numbers of DDL statements (100 or more) in
succession, adding an index to a table sometimes caused
mysqld to crash when it could not find the
NDB. Now when this problem
occurs, the DDL statement should fail with an appropriate error.
DUMP 1000 command (see
DUMP 1000) that
contained extra or malformed arguments could lead to data node
under heavy load could cause data nodes running
ndbmtd to fail.
The ndb_mgm client
command did not show the complete syntax for the
method performs a
malloc() if no buffer is
provided for it to use. However, it was assumed that the memory
thus returned would always be suitably aligned, which is not
always the case. Now when
malloc() provides a
buffer to this method, the buffer is aligned after it is
allocated, and before it is used.
Functionality Added or Changed
Added several new columns to the
transporters table and
counters for the
table of the
information database. The information provided may help in
troublehsooting of transport overloads and problems with send
buffer memory allocation. For more information, see the
descriptions of these tables.
To provide information which can help in assessing the current
state of arbitration in a MySQL Cluster as well as in diagnosing
and correcting arbitration problems, 3 new
been added to the
NDB table grew to contain
approximately one million rows or more per partition, it became
possible to insert rows having duplicate primary or unique keys
into it. In addition, primary key lookups began to fail, even
when matching rows could be found in the table by other means.
This issue was introduced in MySQL Cluster NDB 7.0.36, MySQL Cluster NDB 7.1.26, and MySQL Cluster NDB 7.2.9. Signs that you may have been affected include the following:
Rows left over that should have been deleted
Rows unchanged that should have been updated
Rows with duplicate unique keys due to inserts or updates (which should have been rejected) that failed to find an existing row and thus (wrongly) inserted a new one
This issue does not affect simple scans, so you can see all rows
in a given
SELECT * FROM
and similar queries
that do not depend on a primary or unique key.
Upgrading to or downgrading from an affected release can be troublesome if there are rows with duplicate primary or unique keys in the table; such rows should be merged, but the best means of doing so is application dependent.
In addition, since the key operations themselves are faulty, a merge can be difficult to achieve without taking the MySQL Cluster offline, and it may be necessary to dump, purge, process, and reload the data. Depending on the circumstances, you may want or need to process the dump with an external application, or merely to reload the dump while ignoring duplicates if the result is acceptable.
Another possibility is to copy the data into another table
without the original table' unique key constraints or
primary key (recall that
TABLE t2 SELECT * FROM t1 does not by default copy
t1's primary or unique key definitions
t2). Following this, you can remove the
duplicates from the copy, then add back the unique constraints
and primary key definitions. Once the copy is in the desired
state, you can either drop the original table and rename the
copy, or make a new dump (which can be loaded later) from the
(Bug #16023068, Bug #67928)
The management client command
BackupStatus failed with an error when used with data
nodes having multiple LQH worker threads
(ndbmtd data nodes). The issue did not effect
form of this command.
The multi-threaded job scheduler could be suspended prematurely when there were insufficient free job buffers to allow the threads to continue. The general rule in the job thread is that any queued messages should be sent before the thread is allowed to suspend itself, which guarantees that no other threads or API clients are kept waiting for operations which have already completed. However, the number of messages in the queue was specified incorrectly, leading to increased latency in delivering signals, sluggish response, or otherwise suboptimal performance. (Bug #15908684)
The setting for the
API node configuration parameter was ignored, and the default
value used instead.
Node failure during the dropping of a table could lead to the node hanging when attempting to restart.
When this happened, the
internal dictionary (
DBDICT) lock taken by
the drop table operation was held indefinitely, and the logical
global schema lock taken by the SQL the drop table operation
from which the drop operation originated was held until the
NDB internal operation timed out. To aid in
debugging such occurrences, a new dump code,
which dumps the contents of the
queue, has been added in the ndb_mgm client.
The recently added LCP fragment scan watchdog occasionally reported problems with LCP fragment scans having very high table id, fragment id, and row count values.
This was due to the watchdog not accounting for the time spent draining the backup buffer used to buffer rows before writing to the fragment checkpoint file.
Now, in the final stage of an LCP fragment scan, the watchdog switches from monitoring rows scanned to monitoring the buffer size in bytes. The buffer size should decrease as data is written to the file, after which the file should be promptly closed. (Bug #14680057)
During an online upgrade, certain SQL statements could cause the server to hang, resulting in the error Got error 4012 'Request ndbd time-out, maybe due to high load or communication problems' from NDBCLUSTER. (Bug #14702377)
Job buffers act as the internal queues for work requests (signals) between block threads in ndbmtd and could be exhausted if too many signals are sent to a block thread.
Performing pushed joins in the
block can execute multiple branches of the query tree in
parallel, which means that the number of signals being sent can
increase as more branches are executed. If
DBSPJ execution cannot be completed before
the job buffers are filled, the data node can fail.
This problem could be identified by multiple instances of the message sleeploop 10!! in the cluster out log, possibly followed by job buffer full. If the job buffers overflowed more gradually, there could also be failures due to error 1205 (Lock wait timeout exceeded), shutdowns initiated by the watchdog timer, or other timeout related errors. These were due to the slowdown caused by the 'sleeploop'.
Normally up to a 1:4 fanout ratio between consumed and produced signals is permitted. However, since there can be a potentially unlimited number of rows returned from the scan (and multiple scans of this type executing in parallel), any ratio greater 1:1 in such cases makes it possible to overflow the job buffers.
The fix for this issue defers any lookup child which otherwise would have been executed in parallel with another is deferred, to resume when its parallel child completes one of its own requests. This restricts the fanout ratio for bushy scan-lookup joins to 1:1. (Bug #14709490)
References: See also Bug #14648712.
Under certain rare circumstances, MySQL Cluster data nodes could
crash in conjunction with a configuration change on the data
nodes from a single-threaded to a multi-threaded transaction
coordinator (using the
configuration parameter for ndbmtd). The
problem occurred when a mysqld that had been
started prior to the change was shut down following the rolling
restart of the data nodes required to effect the configuration
Functionality Added or Changed
Important Change; ClusterJ:
A new CMake option
WITH_NDB_JAVA is introduced. When
this option is enabled, the MySQL Cluster build is configured to
include Java support, including support for
ClusterJ. If the JDK cannot be found,
CMake fails with an error. This option is
enabled by default; if you do not wish to
compile Java support into MySQL Cluster, you must now set this
explicitly when configuring the build, using
Added 3 new columns to the
transporters table in the
ndbinfo database. The
bytes_received columns help to provide an
overview of data transfer across the transporter links in a
MySQL Cluster. This information can be useful in verifying
system balance, partitioning, and front-end server load
balancing; it may also be of help when diagnosing network
problems arising from link saturation, hardware faults, or other
Data node logs now provide tracking information about arbitrations, including which nodes have assumed the arbitrator role and at what times. (Bug #11761263, Bug #53736)
A slow filesystem during local checkpointing could exert undue
DBDIH kernel block file page
buffers, which in turn could lead to a data node crash when
these were exhausted. This fix limits the number of table
definition updates that
DBDIH can issue
The management server process, when started with
sometimes hang during shutdown.
The output from ndb_config
--configinfo now contains the
same information as that from ndb_config
--xml, including explicit
indicators for parameters that do not require restarting a data
--initial to take effect.
ndb_config indicated incorrectly
node configuration parameter requires an initial node restart to
take effect, when in fact it does not; this error was also
present in the MySQL Cluster documentation, where it has also
Attempting to restart more than 5 data nodes simultaneously could cause the cluster to crash. (Bug #14647210)
In MySQL Cluster NDB 7.2.7, the size of the hash map was
increased to 3840 LDM threads. However, when upgrading a MySQL
Cluster from a previous release, existing tables could not use
or be modified online to take advantage of the new size, even
when the number of fragments was increased by (for example)
adding new data nodes to the cluster. Now in such cases,
following an upgrade and (once the number of fragments has been
increased), you can run
TABLE ... REORGANIZE PARTITION on tables that were
created in MySQL Cluster NDB 7.2.6 or earlier, after which they
can use the larger hash map size.
ALTER TABLE with other
DML statements on the same NDB table returned Got
error -1 'Unknown error code' from NDBCLUSTER.
Receiver threads could wait unnecessarily to process incomplete signals, greatly reducing performance of ndbmtd. (Bug #14525521)
On platforms where epoll was not available, setting multiple
receiver threads with the
caused ndbmtd to fail.
CPU consumption peaked several seconds after the forced termination an NDB client application due to the fact that the DBTC kernel block waited for any open transactions owned by the disconnected API client to be terminated in a busy loop, and did not break between checks for the correct state. (Bug #14550056)
--connect-delay startup options for
ndbd and ndbmtd.
--connect-retries (default 12) controls how
many times the data node tries to connect to a management server
before giving up; setting it to -1 means that the data node
never stops trying to make contact.
--connect-delay sets the number of seconds to
wait between retries; the default is 5.
(Bug #14329309, Bug #66550)
It was possible in some cases for two transactions to try to
drop tables at the same time. If the master node failed while
one of these operations was still pending, this could lead
either to additional node failures (and cluster shutdown) or to
new dictionary operations being blocked. This issue is addressed
by ensuring that the master will reject requests to start or
stop a transaction while there are outstanding dictionary
takeover requests. In addition, table-drop operations now
correctly signal when complete, as the
kernel block could not confirm node takeovers while such
operations were still marked as pending completion.
Following a failed
TABLE ... REORGANIZE PARTITION statement, a subsequent
execution of this statement after adding new data nodes caused a
failure in the
DBDIH kernel block which led
to an unplanned shutdown of the cluster.
DUMP code 7019 was added as part of this fix.
It can be used to obtain diagnostic information relating to a
failed data node. See
DUMP 7019, for more
References: See also Bug #18550318.
DBSPJ kernel block had no information
about which tables or indexes actually existed, or which had
been modified or dropped, since execution of a given query
DBSPJ might submit dictionary
requests for nonexistent tables or versions of tables, which
could cause a crash in the
This fix introduces a simplified dictionary into the
DBSPJ kernel block such that
DBSPJ can now check reliably for the
existence of a particular table or version of a table on which
it is about to request an operation.
Previously, it was possible to store a maximum of 46137488 rows in a single MySQL Cluster partition. This limitation has now been removed. (Bug #13844405, Bug #14000373)
References: See also Bug #13436216.
When using ndbmtd and performing joins, data
nodes could fail where ndbmtd processes were
configured to use a large number of local query handler threads
(as set by the
configuration parameter), the tables accessed by the join had a
large number of partitions, or both.
(Bug #13799800, Bug #14143553)
Disk Data: Concurrent DML and DDL operations against the same NDB table could cause mysqld to crash. (Bug #14577463)
An improvement introduced in MySQL Cluster NDB 7.2.7 saves memory used by buffers for communications between threads. One way in which this was implemented was by not allocating buffers between threads which were assumed not to communicate, including communication between local query handler (LQH or LDM) threads.
BACKUP kernel block is used by
the LDM thread, and during
native backup the first instance of this block used by the first
LDM thread acts as a client coordinator, and thus attempts to
communicate with other LDM threads. This caused the
START BACKUP command to fail when using
ndbmtd configured with multiple LDM threads.
The fix for this issue restores the buffer used for communication between LDM threads in such cases. (Bug #14489398)
When reloading the redo log during a node or system restart, and
greater than or equal to 42, it was possible for metadata to be
read for the wrong file (or files). Thus, the node or nodes
involved could try to reload the wrong set of data.
NDB table was created
during a data node restart, the operation was rolled back in the
NDB engine, but not on the SQL node where it
was executed. This was due to the table
.FRM files not being cleaned up following
the operation that was rolled back by
Now in such cases these files are removed.
FILE was used for the value of the
without also specifying the
filename, the log
file name defaulted to
logger.log. Now in
such cases, the name defaults to
(Bug #11764570, Bug #57417)
Packaging: Some builds on Solaris 64-bit failed because the packages exceeded the 2GB limit for the SVR4 installation layout. Now such packages are built without the embedded versions of mysqltest and mysql-client_test to save space. (Bug #14058643)
If the Transaction Coordinator aborted a transaction in the “prepared” state, this could cause a resource leak. (Bug #14208924)
When attempting to connect using a socket with a timeout, it was possible (if the timeout was exceeded) for the socket not to be set back to blocking. (Bug #14107173)
An error handling routine in the local query handler used the wrong code path, which could corrupt the transaction ID hash, causing the data node process to fail. This could in some cases possibly lead to failures of other data nodes in the same node group when the failed node attempted to restart. (Bug #14083116)
A shortage of scan fragment records in
resulted in a leak of concurrent scan table records and key
Attempting to add both a column and an index on that column in
the same online
statement caused mysqld to fail. Although
this issue affected only the mysqld shipped
with MySQL Cluster, the table named in the
TABLE could use any storage engine for which online
operations are supported.
Packaging; Cluster API:
was missing from the
clusterjpa JAR file.
This could cause setting
ndb” to be rejected.
References: See also Bug #52106.
libndbclient did not include the
When an NDB API application called
again after the previous call had returned end-of-file (return
code 1), a transaction object was leaked. Now when this happens,
NDB returns error code 4210 (Ndb sent more info than
length specified); previouslyu in such cases, -1 was
returned. In addition, the extra transaction object associated
with the scan is freed, by returning it to the transaction
coordinator's idle list.
TABLE ... REORGANIZE PARTITION statement can be used
to create new table partitions after new empty nodes have been
added to a MySQL Cluster. Usually, the number of partitions to
create is determined automatically, such that, if no new
partitions are required, then none are created. This behavior
can be overridden by creating the original table using the
MAX_ROWS option, which indicates that extra
partitions should be created to store a large number of rows.
However, in this case
ALTER ONLINE TABLE ... REORGANIZE
PARTITION simply uses the
value specified in the original
TABLE statement to determine the number of partitions
required; since this value remains constant, so does the number
of partitions, and so no new ones are created. This means that
the table is not rebalanced, and the new data nodes remain
To solve this problem, support is added for
ONLINE TABLE ...
newvalue is greater than the value
MAX_ROWS in the original
CREATE TABLE statement. This larger
MAX_ROWS value implies that more partitions
are required; these are allocated on the new data nodes, which
restores the balanced distribution of the table data.
When a fragment scan occurring as part of a local checkpoint (LCP) stopped progressing, this kept the entire LCP from completing, which could result it redo log exhaustion, write service outage, inability to recover nodes, and longer system recovery times. To help keep this from occurring, MySQL Cluster now implements an LCP watchdog mechanism, which monitors the fragment scans making up the LCP and takes action if the LCP is observed to be delinquent.
This is intended to guard against any scan related system-level I/O errors or other issues causing problems with LCP and thus having a negative impact on write service and recovery times. Each node independently monitors the progress of local fragment scans occurring as part of an LCP. If no progress is made for 20 seconds, warning logs are generated every 10 seconds thereafter for up to 1 minute. At this point, if no progress has been made, the fragment scan is considered to have hung, and the node is restarted to enable the LCP to continue.
In addition, a new ndbd exit code
NDBD_EXIT_LCP_SCAN_WATCHDOG_FAIL is added
to identify when this occurs. See
LQH Errors, for more information.
An error handling routine in the local query handler
DBLQH) used the wrong code path, which could
corrupt the transaction ID hash, causing the data node process
to fail. This could in some cases possibly lead to failures of
other data nodes in the same node group when the failed node
attempted to restart.
It could sometimes happen that a query pushed down to the data nodes could refer to buffered rows which had been released, and possibly overwritten by other rows. Such rows, if overwritten, could lead to incorrect results from a pushed query, and possibly even to failure of one or more data nodes or SQL nodes. (Bug #14010406)
Pushed joins performed as part of a stored procedure or trigger could cause spurious Out of memory errors on the SQL node where they were executed. (Bug #13945264)
References: See also Bug #13944272.
DUMP 2303 in the ndb_mgm
client now includes the status of the single fragment scan
record reserved for a local checkpoint.
ndbmemcached exited unexpectedly when more
than 128 clients attempted to connect concurrently using
prefixes. In addition, a NOT FOUND error
was returned when the
encountered a temporary error from
NDB; now the error No
Ndb Instances in freelist is returned instead.
(Bug #13890064, Bug #13891085)
References: See also Bug #13945264.
The performance of
ndbmemcache with a
workload that consisted mostly of primary key reads became
(Bug #13868787, Bug #64713)
When upgrading or downgrading between a MySQL Cluster version supporting distributed pushdown joins (MySQL Cluster NDB 7.2 and later) and one that did not, queries that the later MySQL Cluster version tried to push down could cause data nodes still running the earlier version to fail. Now the SQL nodes check the version of the software running on the data nodes, so that queries are not pushed down if there are any data nodes in the cluster that do not support pushdown joins. (Bug #13894817)
The memcached server failed to build correctly on 64-bit Solaris/SPARC. (Bug #13854122)
ndbmtd failed to restart when the size of a table definition exceeded 32K.
(The size of a table definition is dependent upon a number of factors, but in general the 32K limit is encountered when a table has 250 to 300 columns.) (Bug #13824773)
An initial start using ndbmtd could sometimes hang. This was due to a state which occurred when several threads tried to flush a socket buffer to a remote node. In such cases, to minimize flushing of socket buffers, only one thread actually performs the send, on behalf of all threads. However, it was possible in certain cases for there to be data in the socket buffer waiting to be sent with no thread ever being chosen to perform the send. (Bug #13809781)
In some cases, restarting data nodes spent a very long time in
Start Phase 101, when API nodes must connect to the starting
when the API nodes trying to connect failed in a live-lock
scenario. This connection process uses a handshake during which
a small number of messages are exchanged, with a timeout used to
detect failures during the handshake.
Prior to this fix, this timeout was set such that, if one API node encountered the timeout, all other nodes connecting would do the same. The fix also decreases this timeout. This issue (and the effects of the fix) are most likely to be observed on relatively large configurations having 10 or more data nodes and 200 or more API nodes. (Bug #13825163)
TABLE failed when a
When trying to use ndb_size.pl
to connect to a MySQL server running on a nonstandard port, the
port argument was ignored.
(Bug #13364905, Bug #62635)
server system variable was inadvertently removed from the NDB
7.2 codebase prior to General Availability. This fix restores
(Bug #64697, Bug #13891116, Bug #13947227)
An assert in
NDB could cause
memcached to fail.
Important Change: A number of changes have been made in the configuration of transporter send buffers.
The data node configuration parameter
is now deprecated, and thus subject to removal in a future
MySQL Cluster release.
ReservedSendBufferMemory has been
non-functional since it was introduced and remains so.
TotalSendBufferMemory now works correctly
with data nodes using ndbmtd.
A new data node configuration parameter
is introduced. Its purpose is to control how much additional
memory can be allocated to the send buffer over and above
that specified by
SendBufferMemory. The default setting
(0) allows up to 16MB to be allocated automatically.
(Bug #13633845, Bug #11760629, Bug #53053)
Several instances in the NDB code affecting the operation of
multi-threaded data nodes, where
SendBufferMemory was associated with a
specific thread for an unnecessarily long time, have been
identified and fixed, by minimizing the time that any of these
buffers can be held exclusively by a given thread (send buffer
memory being critical to operation of the entire node).
LIKE ... ESCAPE on
NDB tables failed when pushed down
to the data nodes. Such queries are no longer pushed down,
regardless of the value of
(Bug #13604447, Bug #61064)
To avoid TCP transporter overload, an overload flag is kept in
the NDB kernel for each data node; this flag is used to abort
key requests if needed, yielding error 1218 Send
Buffers overloaded in NDB kernel in such cases.
Scans can also put significant pressure on transporters,
especially where scans with a high degree of parallelism are
executed in a configuration with relatively small send buffers.
However, in these cases, overload flags were not checked, which
could lead to node failures due to send buffer exhaustion. Now,
overload flags are checked by scans, and in cases where
returning sufficient rows to match the batch size
--ndb-batch-size server option)
would cause an overload, the number of rows is limited to what
can be accommodated by the send buffer.
See also Configuring MySQL Cluster Send Buffer Parameters. (Bug #13602508)
A data node crashed when more than 16G fixed-size memory was
DBTUP to one fragment (because
DBACC kernel block was not prepared to
accept values greater than 32 bits from it, leading to an
overflow). Now in such cases, the data node returns Error 889
Table fragment fixed data reference has reached
maximum possible value.... When this happens, you
can work around the problem by increasing the number of
partitions used by the table (such as by using the
MAXROWS option with
References: See also Bug #11747870, Bug #34348.
ndb_engine.so with only 2 API slots
available in the cluster configuration file,
memcached attempted to make a third
connection and crashed when this failed.
References: See also Bug #13608135.
memcached started with the
ndb_engine.so plugin failed to start if the
ndbmemcache database was missing.
A node failure and recovery while performing a scan on more than 32 partitions led to additional node failures during node takeover. (Bug #13528976)
option now causes ndb_mgmd to skip checking
for the configuration directory, and thus to skip creating it in
the event that it does not exist.
Functionality Added or Changed
A mysqld process joining a MySQL Cluster
where distributed privileges are in use now automatically
PRIVILEGES as part of the connection process, so that
the cluster's distributed privileges take immediate effect
on the new SQL node.
RPM distributions of MySQL Cluster NDB 7.1 contained a number of
packages which in MySQL Cluster NDB 7.2 have been merged into
MySQL-Cluster-server RPM. However, the
MySQL Cluster NDB 7.2
RPM did not actually obsolete these packages, which meant that
they had to be removed manually prior to performing an upgrade
from a MySQL Cluster NDB 7.1 RPM installation.
These packages include the
For more information, see Installing MySQL Cluster from RPM. (Bug #13545589)
Accessing a table having a
column but no primary key following a restart of the SQL node
failed with Error 1 (Unknown error code).
At the beginning of a local checkpoint, each data node marks its local tables with a “to be checkpointed” flag. A failure of the master node during this process could cause either the LCP to hang, or one or more data nodes to be forcibly shut down. (Bug #13436481)
A node failure while a
TABLE statement was executing resulted in a hung
connection (and the user was not informed of any error that
would cause this to happen).
References: See also Bug #13407848.
Functionality Added or Changed
data node configuration parameter to enable control of multiple
threads and CPUs when using ndbmtd, by
assigning threads of one or more specified types to execute on
one or more CPUs. This can provide more precise and flexible
control over multiple threads than can be obtained using the
data node configuration parameter, which specifies a percentage
of data node resources to hold in reserve for restarts. The
resources monitored are
IndexMemory, and any
MAX_ROWS settings (see
CREATE TABLE Syntax). The default value of
MinFreePct is 5, which
means that 5% from each these resources is now set aside for
TRUNCATE TABLE on
mysql.procs_priv, when these tables had been
converted to MySQL Cluster distributed grant tables, caused
mysqld to crash.
Restarting an SQL node configured for distributed grants could sometimes result in a crash. (Bug #13340819)
Previously, forcing simultaneously the shutdown of multiple data
SHUTDOWN -F in the
ndb_mgm management client could cause the
entire cluster to fail. Now in such cases, any such nodes are
forced to abort immediately.
Functionality Added or Changed
Added the ndbinfo_select_all utility.
Cluster API: Added support for the Memcache API using ndbmemcache, a loadable storage engine for memcached version 1.6 and later, which can be used to provide a persistent MySQL Cluster data store, accessed using the memcache protocol.
The standard memcached caching engine is now included in the MySQL Cluster distribution. Each memcached server, in addition to providing direct access to data stored in a MySQL Cluster, is able to cache data locally and serve (some) requests from this local cache.
The memcached server can also provide an interface to existing MySQL Cluster tables that is strictly defined, so that an administrator can control exactly which tables and columns are referenced by particular memcache keys and values, and which operations are allowed on these keys and values.
For more information, see
ndbmemcache—Memcache API for MySQL Cluster.
When adding data nodes online, if the SQL nodes were not restarted before starting the new data nodes, the next query to be executed crashed the SQL node on which it was run. (Bug #13715216, Bug #62847)
References: This bug was introduced by Bug #13117187.
include/storage directory, where the
header files supplied for use in compiling MySQL Cluster
applications are normally located, was missing from MySQL
Cluster release packages for Windows.
Functionality Added or Changed
Added support for pushing down joins to the NDB kernel for
parallel execution across the data nodes, which can speed up
execution of joins across
tables by a factor of 20 or more in some cases. Some
restrictions apply on the types of joins that can be optimized
in this way; in particular, columns to be joined must use
exactly the same data type, and cannot be any of the
TEXT types. In addition, columns
to be joined must be part of a table index or primary key.
Support for this feature can be enabled or disabled using the
ndb_join_pushdown server system
variable (enabled by default); see the description of this
variable for more information and examples.
As part of this improvement, the status variables
Ndb_pushed_reads also been
introduced for monitoring purposes. You can also see whether a
given join is pushed down using
EXPLAIN. In addition, several new
counters relating to push-down join performance have been added
counters table in the
ndbinfo database. For more
information, see the descriptions of the status variables
The default values for a number of MySQL Cluster data node
configuration parameters have changed, to provide more
resiliency to environmental issues and better handling of some
potential failure scenarios, and to perform more reliably with
increases in memory and other resource requirements brought
about by recent improvements in join handling by
NDB. The affected parameters are
Default increased from 1500 ms to 5000 ms.
Default increased from 3000 ms to 7500 ms.
Now effectively disabled by default (default changed from
4000 ms to 0).
Default increased from 20MB to 128MB.
Default increased from 32 to 256.
In addition, when
MaxNoOfLocalScans is not
specified, the value computed for it automatically has been
increased by a factor of 4 (that is, to 4 times
times the number of data nodes in the cluster).
MySQL user privileges can now be distributed automatically
across all MySQL servers (SQL nodes) connected to the same MySQL
Cluster. Previously, each MySQL Server's user privilege
tables were required to use the
MyISAM storage engine, which meant
that a user account and its associated privileges created on one
SQL node were not available on any other SQL node without manual
intervention. MySQL Cluster now provides an SQL script file
ndb_dist_priv.sql that can be found in
share/mysql under the MySQL installation
directory; loading this script creates a stored procedure
mysql_cluster_move_privileges that can be
used following initial installation to convert the privilege
tables to the
NDB storage engine,
so that any time a MySQL user account is created, dropped, or
has its privileges updated on any SQL node, the changes take
effect immediately on all other MySQL servers attached to the
cluster. Note that you may need to execute
PRIVILEGES on any SQL nodes with connected MySQL
clients (or to disconnect and then reconnect the clients) in
order for those clients to be able to see the changes in
mysql_cluster_move_privileges has been
executed successfully, all MySQL user privileges are distributed
across all connected MySQL Servers. MySQL Servers that join the
cluster after this automatically participate in the privilege
ndb_dist_priv.sql also provides stored
routines that can be used to verify that the privilege tables
have been distributed successfully, and to perform other tasks.
For more information, see Distributed MySQL Privileges for MySQL Cluster.
By default, data nodes now exhibit fail-fast behavior whenever
they encounter corrupted tuples—in other words, a data
node forcibly shuts down whenever it detects a corrupted tuple.
To override this behavior, you can disable the
data node configuration parameter, which is enabled by default.
This is a change from MySQL Cluster NDB 7.0 and MySQL Cluster
NDB 7.1, where
was disabled (so that data nodes ignored tuple corruption) by
It is now possible to filter the output from
ndb_config so that it displays only system,
data node, or connection parameters and values, using one of the
In addition, it is now possible to specify from which data node
the configuration data is obtained, using the
that is added in this release.
For more information, see ndb_config — Extract MySQL Cluster Configuration Information. (Bug #11766870)
Incompatible Change; Cluster API: Restarting a machine hosting data nodes, SQL nodes, or both, caused such nodes when restarting to time out while trying to obtain node IDs.
As part of the fix for this issue, the behavior and default
values for the NDB API
method have been improved. Due to these changes, the version
number for the included NDB client library
libndbclient.so) has been increased from
4.0.0 to 5.0.0. For NDB API applications, this means that as
part of any upgrade, you must do both of the following:
Review and possibly modify any NDB API code that uses the
method, in order to take into account its changed default
Recompile any NDB API applications using the new version of the client library.
Also in connection with this issue, the default value for each
of the two mysqld options
--ndb-wait-setup has been
increased to 30 seconds (from 0 and 15, respectively). In
addition, a hard-coded 30-second delay was removed, so that the
now handled correctly in all cases.
AUTO_INCREMENT values were not set correctly
IGNORE statements affecting
NDB tables. This could lead such
statements to fail with Got error 4350 'Transaction
already aborted' from NDBCLUSTER when inserting
multiple rows containing duplicate values.
(Bug #11755237, Bug #46985)