Documentation Home
MySQL 5.6 Reference Manual
Related Documentation Download this Manual
PDF (US Ltr) - 31.6Mb
PDF (A4) - 31.6Mb
PDF (RPM) - 30.5Mb
HTML Download (TGZ) - 7.6Mb
HTML Download (Zip) - 7.6Mb
HTML Download (RPM) - 6.5Mb
Man Pages (TGZ) - 170.9Kb
Man Pages (Zip) - 280.0Kb
Info (Gzip) - 3.0Mb
Info (Zip) - 3.0Mb
Excerpts from this Manual

MySQL 5.6 Reference Manual  /  ...  /  Performing a Rolling Restart of an NDB Cluster

18.5.5 Performing a Rolling Restart of an NDB Cluster

This section discusses how to perform a rolling restart of an NDB Cluster installation, so called because it involves stopping and starting (or restarting) each node in turn, so that the cluster itself remains operational. This is often done as part of a rolling upgrade or rolling downgrade, where high availability of the cluster is mandatory and no downtime of the cluster as a whole is permissible. Where we refer to upgrades, the information provided here also generally applies to downgrades as well.

There are a number of reasons why a rolling restart might be desirable. These are described in the next few paragraphs.

Configuration change.  To make a change in the cluster's configuration, such as adding an SQL node to the cluster, or setting a configuration parameter to a new value.

NDB Cluster software upgrade or downgrade.  To upgrade the cluster to a newer version of the NDB Cluster software (or to downgrade it to an older version). This is usually referred to as a rolling upgrade (or rolling downgrade, when reverting to an older version of NDB Cluster).

Change on node host.  To make changes in the hardware or operating system on which one or more NDB Cluster node processes are running.

System reset (cluster reset).  To reset the cluster because it has reached an undesirable state. In such cases it is often desirable to reload the data and metadata of one or more data nodes. This can be done in any of three ways:

Resource Recovery.  To free memory previously allocated to a table by successive INSERT and DELETE operations, for re-use by other NDB Cluster tables.

The process for performing a rolling restart may be generalized as follows:

  1. Stop all cluster management nodes (ndb_mgmd processes), reconfigure them, then restart them. (See Rolling restarts with multiple management servers.)

  2. Stop, reconfigure, then restart each cluster data node (ndbd process) in turn.

  3. Stop, reconfigure, then restart each cluster SQL node (mysqld process) in turn.

The specifics for implementing a given rolling upgrade depend upon the changes being made. A more detailed view of the process is presented here:

Figure 18.24 NDB Cluster Rolling Restarts By Type

Content is described in the surrounding text.

In the previous diagram, the Stop and Start steps indicate that the process must be stopped completely using a shell command (such as kill on most Unix systems) or the management client STOP command, then started again from a system shell by invoking the ndbd or ndb_mgmd executable as appropriate. On Windows, you can also use the system NET START and NET STOP commands or the Windows Service Manager to start and stop nodes which have been installed as Windows services (see Section, “Installing NDB Cluster Processes as Windows Services”).

Restart indicates that the process may be restarted using the ndb_mgm management client RESTART command.

NDB Cluster supports a flexible order for upgrading nodes. When upgrading an NDB Cluster, you may upgrade API nodes (including SQL nodes) before upgrading the management nodes, data nodes, or both. In other words, you are permitted to upgrade the API and SQL nodes in any order. This is subject to the following provisions:

  • This functionality is intended for use as part of an online upgrade only. A mix of node binaries from different NDB Cluster releases is neither intended nor supported for continuous, long-term use in a production setting.

  • All management nodes must be upgraded before any data nodes are upgraded. This remains true regardless of the order in which you upgrade the cluster's API and SQL nodes.

  • Features specific to the new version must not be used until all management nodes and data nodes have been upgraded.

    This also applies to any MySQL Server version change that may apply, in addition to the NDB engine version change, so do not forget to take this into account when planning the upgrade. (This is true for online upgrades of NDB Cluster in general.)

See also Bug #48528 and Bug #49163.


It is not possible for any API node to perform schema operations (such as data definition statements) during a node restart.

Rolling restarts with multiple management servers.  When performing a rolling restart of an NDB Cluster with multiple management nodes, you should keep in mind that ndb_mgmd checks to see if any other management node is running, and, if so, tries to use that node's configuration data. To keep this from occurring, and to force ndb_mgmd to reread its configuration file, perform the following steps:

  1. Stop all NDB Cluster ndb_mgmd processes.

  2. Update all config.ini files.

  3. Start a single ndb_mgmd with --reload, --initial, or both options as desired.

  4. If you started the first ndb_mgmd with the --initial option, you must also start any remaining ndb_mgmd processes using --initial.

    Regardless of any other options used when starting the first ndb_mgmd, you should not start any remaining ndb_mgmd processes after the first one using --reload.

  5. Complete the rolling restarts of the data nodes and API nodes as normal.

When performing a rolling restart to update the cluster's configuration, you can use the config_generation column of the ndbinfo.nodes table to keep track of which data nodes have been successfully restarted with the new configuration. See Section, “The ndbinfo nodes Table”.

User Comments
  Posted by David Lotts on December 20, 2007
I am not sure if the following is documented, so I am putting in here. We used Server version: 5.0.45 MySQL Community Server (GPL)

It appears that ndbd rolling restarts wait for transactions to commit, but eventually time out and close the uncommited connection.

I think this is good behavior, it cleans up stalled transactions that would probably fail anyway, and then starts the node.

This came out of a lot of interesting scenarios spanning rolling restarts. We only start and stop ndbd. We have 2 ndb notes containing all the data, node 6 and 7.
I used two terminals, indicated below by prompts ndb_mgm> and mysql>. Read above how "restart -n" is like stop, but it can be started later from the ndb_mgm. Here is how I did the experiment:

ndb_mgm> 7 start
ndb_mgm> 6 restart -n
### This effectively stops node 6
ndb_mgm> show
### Observe node 6 is down, 7 is up

mysql> start transaction;
mysql> insert ...;

ndb_mgm> 6 start
ndb_mgm> show
### Node 6 six will stay in a "Starting" state for several minutes.
### if you commit the transaction, 6 will immediately start.
### if you wait until node 6 starts, 5 or 10 minutes, the following happens in the mysql client terminal:

mysql> select * from ...;
ERROR 2006 (HY000): MySQL server has gone away
No connection. Trying to reconnect...
Connection id: 165066
Current database: *** NONE ***

### Observe that the insert transaction was rolled back.
### That's it.

Speculative conclusion: rolling restarts wait for commits, but eventually time out and close the uncommitted connection.

  Posted by Tristan Sloughter on October 11, 2011
This fails to mention what you must do if you make a change to the number of replicas. You update can be applied 'online'.
Sign Up Login You must be logged in to post a comment.