In the following section, we answer questions that are frequently
asked about MySQL Cluster and the
A.10.1: Which versions of the MySQL software support Cluster? Do I have to compile from source?
A.10.2: What do “NDB” and “NDBCLUSTER” mean?
A.10.3: What is the difference between using MySQL Cluster versus using MySQL Replication?
A.10.4: Do I need any special networking to run MySQL Cluster? How do computers in a cluster communicate?
A.10.5: How many computers do I need to run a MySQL Cluster, and why?
A.10.6: What do the different computers do in a MySQL Cluster?
When I run the
SHOW command in the MySQL
Cluster management client, I see a line of output that looks
id=2 @10.100.10.32 (Version: 5.6.20-ndb-7.3.7 Nodegroup: 0, *)
What does the
* mean? How is this node
different from the others?
A.10.8: With which operating systems can I use MySQL Cluster?
A.10.9: What are the hardware requirements for running MySQL Cluster?
A.10.10: How much RAM do I need to use MySQL Cluster? Is it possible to use disk memory at all?
A.10.11: What file systems can I use with MySQL Cluster? What about network file systems or network shares?
A.10.12: Can I run MySQL Cluster nodes inside virtual machines (such as those created by VMWare, Parallels, or Xen)?
I am trying to populate a MySQL Cluster database. The loading
process terminates prematurely and I get an error message like
ERROR 1114: The table 'my_cluster_table' is
Why is this happening?
A.10.14: MySQL Cluster uses TCP/IP. Does this mean that I can run it over the Internet, with one or more nodes in remote locations?
A.10.15: Do I have to learn a new programming or query language to use MySQL Cluster?
A.10.16: What programming languages and APIs are supported by MySQL Cluster?
A.10.17: Does MySQL Cluster include any management tools?
A.10.18: How do I find out what an error or warning message means when using MySQL Cluster?
A.10.19: Is MySQL Cluster transaction-safe? What isolation levels are supported?
A.10.20: What storage engines are supported by MySQL Cluster?
A.10.21: In the event of a catastrophic failure—say, for instance, the whole city loses power and my UPS fails—would I lose all my data?
Is it possible to use
FULLTEXT indexes with
A.10.23: Can I run multiple nodes on a single computer?
A.10.24: Are there any limitations that I should be aware of when using MySQL Cluster?
A.10.25: Does MySQL Cluster support foreign keys?
A.10.26: How do I import an existing MySQL database into a MySQL Cluster?
A.10.27: How do MySQL Cluster nodes communicate with one another?
A.10.28: What is an arbitrator?
A.10.29: What data types are supported by MySQL Cluster?
A.10.30: How do I start and stop MySQL Cluster?
A.10.31: What happens to MySQL Cluster data when the MySQL Cluster is shut down?
A.10.32: Is it a good idea to have more than one management node for a MySQL Cluster?
A.10.33: Can I mix different kinds of hardware and operating systems in one MySQL Cluster?
A.10.34: Can I run two data nodes on a single host? Two SQL nodes?
A.10.35: Can I use host names with MySQL Cluster?
A.10.36: Does MySQL Cluster support IPv6?
A.10.37: How do I handle MySQL users in a MySQL Cluster having multiple MySQL servers?
A.10.38: How do I continue to send queries in the event that one of the SQL nodes fails?
A.10.39: How do I back up and restore a MySQL Cluster?
A.10.40: What is an “angel process”?
Questions and Answers
MySQL Cluster is not supported in standard MySQL Server 5.6 releases. Instead, MySQL Cluster is provided as a separate product. Currently, the following MySQL Cluster release series are available for production use:
MySQL Cluster NDB 6.3. This series is still maintained and available for use but is not recommended for new deployments. The most recent MySQL Cluster NDB 6.3 release can be obtained from http://dev.mysql.com/downloads/cluster/.
MySQL Cluster NDB 7.0. This series is still available for use in production, although MySQL Cluster NDB 7.2 is recommended for new deployments. The most recent MySQL Cluster NDB 7.0 release can be obtained from http://dev.mysql.com/downloads/cluster/.
MySQL Cluster NDB 7.1. This series is a previous General Availability (GA) version of MySQL Cluster, still available for production, although we recommend that new deployments use MySQL Cluster NDB 7.3. The most recent MySQL Cluster NDB 7.1 release can be obtained from http://dev.mysql.com/downloads/cluster/.
MySQL Cluster NDB 7.2. This series is a General Availability (GA) version of MySQL Cluster, still available for production, although we recommend that new deployments use the latest MySQL Cluster NDB 7.3 release. The most recent MySQL Cluster NDB 7.2 release can be obtained from http://dev.mysql.com/downloads/cluster/.
MySQL Cluster NDB 7.3.
This series is the latest Generally Available (GA) version
of MySQL Cluster, based on version 7.3 of the
NDB storage engine and MySQL
Server 5.6. New deployments should use the latest release
in this series. The most recent MySQL Cluster NDB 7.3
release can be obtained from
You should use MySQL Cluster NDB 7.2 or MySQL Cluster NDB 7.3 for any new deployments; if you are using an older version of MySQL Cluster, you should upgrade to one of these soon as possible. For an overview of improvements made in MySQL Cluster NDB 7.2, see MySQL Cluster Development in MySQL Cluster NDB 7.2; for information about improvements made in MySQL Cluster NDB 7.3, see Section 18.104.22.168, “MySQL Cluster Development in MySQL Cluster NDB 7.3”.
“NDB” stands for
NDBCLUSTER are both names for the
storage engine that enables clustering support in MySQL. While
our developers prefer
name is correct; both names appear in our documentation, and
either name can be used in the
CREATE TABLE statement for
creating a MySQL Cluster table.
In traditional MySQL replication, a master MySQL server updates
one or more slaves. Transactions are committed sequentially, and
a slow transaction can cause the slave to lag behind the master.
This means that if the master fails, it is possible that the
slave might not have recorded the last few transactions. If a
transaction-safe engine such as
InnoDB is being used, a transaction
will either be complete on the slave or not applied at all, but
replication does not guarantee that all data on the master and
the slave will be consistent at all times. In MySQL Cluster, all
data nodes are kept in synchrony, and a transaction committed by
any one data node is committed for all data nodes. In the event
of a data node failure, all remaining data nodes remain in a
In short, whereas standard MySQL replication is asynchronous, MySQL Cluster is synchronous.
Asynchronous replication is also available in MySQL Cluster. MySQL Cluster Replication (also sometimes known as “geo-replication”) includes the capability to replicate both between two MySQL Clusters, and from a MySQL Cluster to a non-Cluster MySQL server. See Section 18.6, “MySQL Cluster Replication”.
MySQL Cluster is intended to be used in a high-bandwidth environment, with computers connecting using TCP/IP. Its performance depends directly upon the connection speed between the cluster's computers. The minimum connectivity requirements for MySQL Cluster include a typical 100-megabit Ethernet network or the equivalent. We recommend you use gigabit Ethernet whenever available.
The faster SCI protocol is also supported, but requires special hardware. See Section 18.3.5, “Using High-Speed Interconnects with MySQL Cluster”, for more information about SCI.
A minimum of three computers is required to run a viable cluster. However, the minimum recommended number of computers in a MySQL Cluster is four: one each to run the management and SQL nodes, and two computers to serve as data nodes. The purpose of the two data nodes is to provide redundancy; the management node must run on a separate machine to guarantee continued arbitration services in the event that one of the data nodes fails.
To provide increased throughput and high availability, you should use multiple SQL nodes (MySQL Servers connected to the cluster). It is also possible (although not strictly necessary) to run multiple management servers.
A MySQL Cluster has both a physical and logical organization, with computers being the physical elements. The logical or functional elements of a cluster are referred to as nodes, and a computer housing a cluster node is sometimes referred to as a cluster host. There are three types of nodes, each corresponding to a specific role within the cluster. These are:
Management node. This node provides management services for the cluster as a whole, including startup, shutdown, backups, and configuration data for the other nodes. The management node server is implemented as the application ndb_mgmd; the management client used to control MySQL Cluster is ndb_mgm. See Section 18.4.4, “ndb_mgmd — The MySQL Cluster Management Server Daemon”, and Section 18.4.5, “ndb_mgm — The MySQL Cluster Management Client”, for information about these programs.
This type of node stores and replicates data. Data node
functionality is handled by instances of the
NDB data node process
ndbd. For more information, see
Section 18.4.1, “ndbd — The MySQL Cluster Data Node Daemon”.
This is simply an instance of MySQL Server
(mysqld) that is built with support for
NDBCLUSTER storage engine
and started with the
to enable the engine and the
--ndb-connectstring option to enable it
to connect to a MySQL Cluster management server. For more
about these options, see
Section 22.214.171.124, “MySQL Server Options for MySQL Cluster”.
An API node is any application that makes direct use of Cluster data nodes for data storage and retrieval. An SQL node can thus be considered a type of API node that uses a MySQL Server to provide an SQL interface to the Cluster. You can write such applications (that do not depend on a MySQL Server) using the NDB API, which supplies a direct, object-oriented transaction and scanning interface to MySQL Cluster data; see MySQL Cluster API Overview: The NDB API, for more information.
id=2 @10.100.10.32 (Version: 5.6.20-ndb-7.3.7 Nodegroup: 0, *)
What does the
* mean? How is this node
different from the others?
The simplest answer is, “It's not something you can control, and it's nothing that you need to worry about in any case, unless you're a software engineer writing or analyzing the MySQL Cluster source code”.
If you don't find that answer satisfactory, here's a longer and more technical version:
A number of mechanisms in MySQL Cluster require distributed coordination among the data nodes. These distributed algorithms and protocols include global checkpointing, DDL (schema) changes, and node restart handling. To make this coordination simpler, the data nodes “elect” one of their number to act as leader. (This node was once referred to as a “master”, but this terminology was dropped to avoid confusion with master server in MySQL Replication.) There is no user-facing mechanism for influencing this selection, which is completely automatic; the fact that it is automatic is a key part of MySQL Cluster's internal architecture.
When a node acts as the “leader” for any of these mechanisms, it is usually the point of coordination for the activity, and the other nodes act as “followers”, carrying out their parts of the activity as directed by the leader. If the node acting as leader fails, then the remaining nodes elect a new leader. Tasks in progress that were being coordinated by the old leader may either fail or be continued by the new leader, depending on the actual mechanism involved.
It is possible for some of these different mechanisms and
protocols to have different leader nodes, but in general the
same leader is chosen for all of them. The node indicated as the
leader in the output of
SHOW in the
management client is known internally as the
DICT manager (see
DBDICT Block, in the
MySQL Cluster API Developer Guide, for
more information), responsible for coordinating DDL and metadata
MySQL Cluster is designed in such a way that the choice of leader has no discernable effect outside the cluster itself. For example, the current leader does not have significantly higher CPU or resource usage than the other data nodes, and failure of the leader should not have a significantly different impact on the cluster than the failure of any other data node.
MySQL Cluster is supported on most Unix-like operating systems. MySQL Cluster NDB 7.2 is also supported in production settings on Microsoft Windows operating systems.
For more detailed information concerning the level of support which is offered for MySQL Cluster on various operating system versions, operating system distributions, and hardware platforms, please refer to http://www.mysql.com/support/supportedplatforms/cluster.html.
MySQL Cluster should run on any platform for which
NDB-enabled binaries are available.
For data nodes and API nodes, faster CPUs and more memory are
likely to improve performance, and 64-bit CPUs are likely to be
more effective than 32-bit processors. There must be sufficient
memory on machines used for data nodes to hold each node's share
of the database (see How much RAM do I
Need? for more information). For a computer which is
used only for running the MySQL Cluster management server, the
requirements are minimal; a common desktop PC (or the
equivalent) is generally sufficient for this task. Nodes can
communicate through the standard TCP/IP network and hardware.
They can also use the high-speed SCI protocol; however, special
networking hardware and software are required to use SCI (see
Section 18.3.5, “Using High-Speed Interconnects with MySQL Cluster”).
Formerly MySQL Cluster was in-memory only. MySQL 5.1 and later also provide the ability to store MySQL Cluster on disk. (Note that we have no plans to backport this capability to previous releases.) See Section 18.5.12, “MySQL Cluster Disk Data Tables”, for more information.
NDB tables, you can use the
following formula for obtaining a rough estimate of how much RAM
is needed for each data node in the cluster:
(SizeofDatabase × NumberOfReplicas × 1.1 ) / NumberOfDataNodes
To calculate the memory requirements more exactly requires determining, for each table in the cluster database, the storage space required per row (see Section 11.7, “Data Type Storage Requirements”, for details), and multiplying this by the number of rows. You must also remember to account for any column indexes as follows:
Each primary key or hash index created for an
NDBCLUSTER table requires
21–25 bytes per record. These indexes use
Each ordered index requires 10 bytes storage per record,
Creating a primary key or unique index also creates an
ordered index, unless this index is created with
USING HASH. In other words:
A primary key or unique index on a Cluster table normally takes up 31 to 35 bytes per record.
However, if the primary key or unique index is created
USING HASH, then it requires
only 21 to 25 bytes per record.
Note that creating MySQL Cluster tables with
HASH for all primary keys and unique indexes will
generally cause table updates to run more quickly—in some
cases by a much as 20 to 30 percent faster than updates on
USING HASH was not used in
creating primary and unique keys. This is due to the fact that
less memory is required (because no ordered indexes are
created), and that less CPU must be utilized (because fewer
indexes must be read and possibly updated). However, it also
means that queries that could otherwise use range scans must be
satisfied by other means, which can result in slower selects.
When calculating Cluster memory requirements, you may find
useful the ndb_size.pl utility which is
available in recent MySQL 5.6 releases. This Perl
script connects to a current (non-Cluster) MySQL database and
creates a report on how much space that database would require
if it used the
engine. For more information, see
Section 18.4.25, “ndb_size.pl — NDBCLUSTER Size Requirement Estimator”.
It is especially important to keep in mind that every
MySQL Cluster table must have a primary key. The
NDB storage engine creates a
primary key automatically if none is defined; this primary key
is created without
You can determine how much memory is being used for storage of
MySQL Cluster data and indexes at any given time using the
REPORT MEMORYUSAGE command in the
ndb_mgm client; see
Section 18.5.2, “Commands in the MySQL Cluster Management Client”, for more
information. In addition, warnings are written to the cluster
log when 80% of available
IndexMemory is in use,
and again when usage reaches 85%, 90%, and so on.
Generally, any file system that is native to the host operating system should work well with MySQL Cluster. If you find that a given file system works particularly well (or not so especially well) with MySQL Cluster, we invite you to discuss your findings in the MySQL Cluster Forums.
For Windows, we recommend that you use
file systems for MySQL Cluster, just as we do for standard
MySQL. We do not test MySQL Cluster with
VFAT file systems. Because of this, we do
not recommend their use with MySQL or MySQL Cluster.
MySQL Cluster is implemented as a shared-nothing solution; the idea behind this is that the failure of a single piece of hardware should not cause the failure of multiple cluster nodes, or possibly even the failure of the cluster as a whole. For this reason, the use of network shares or network file systems is not supported for MySQL Cluster. This also applies to shared storage devices such as SANs.
MySQL Cluster is supported for use in virtual machines beginning with MySQL Cluster NDB 7.2. We currently support and test using Oracle VM.
Some MySQL Cluster users have successfully deployed MySQL Cluster using other virtualization products; in such cases, Oracle can provide MySQL Cluster support, but issues specific to the virtual environment must be referred to that product's vendor.
I am trying to populate a MySQL Cluster database. The loading
process terminates prematurely and I get an error message like
ERROR 1114: The table 'my_cluster_table' is
Why is this happening?
The cause is very likely to be that your setup does not provide
sufficient RAM for all table data and all indexes,
including the primary key required by the
NDB storage engine and
automatically created in the event that the table definition
does not include the definition of a primary key.
It is also worth noting that all data nodes should have the same amount of RAM, since no data node in a cluster can use more memory than the least amount available to any individual data node. For example, if there are four computers hosting Cluster data nodes, and three of these have 3GB of RAM available to store Cluster data while the remaining data node has only 1GB RAM, then each data node can devote at most 1GB to MySQL Cluster data and indexes.
In some cases it is possible to get Table is
full errors in MySQL client applications even when
ndb_mgm -e "ALL REPORT MEMORYUSAGE" shows
DataMemory. You can
NDB to create extra
partitions for MySQL Cluster tables and thus have more memory
available for hash indexes by using the
MAX_ROWS option for
CREATE TABLE. In general, setting
MAX_ROWS to twice the number of rows that you
expect to store in the table should be sufficient.
For similar reasons, you can also sometimes encounter problems
with data node restarts on nodes that are heavily loaded with
data. In MySQL Cluster NDB 7.1 and later, the addition of the
helps with this issue by reserving a portion (5% by default) of
IndexMemory for use in
restarts. This reserved memory is not available for storing
NDB tables or data.
It is very unlikely that a cluster would perform reliably under such conditions, as MySQL Cluster was designed and implemented with the assumption that it would be run under conditions guaranteeing dedicated high-speed connectivity such as that found in a LAN setting using 100 Mbps or gigabit Ethernet—preferably the latter. We neither test nor warrant its performance using anything slower than this.
Also, it is extremely important to keep in mind that communications between the nodes in a MySQL Cluster are not secure; they are neither encrypted nor safeguarded by any other protective mechanism. The most secure configuration for a cluster is in a private network behind a firewall, with no direct access to any Cluster data or management nodes from outside. (For SQL nodes, you should take the same precautions as you would with any other instance of the MySQL server.) For more information, see Section 18.5.11, “MySQL Cluster Security Issues”.
No. Although some specialized commands are used to manage and configure the cluster itself, only standard (My)SQL statements are required for the following operations:
Creating, altering, and dropping tables
Inserting, updating, and deleting table data
Creating, changing, and dropping primary and unique indexes
Some specialized configuration parameters and files are required to set up a MySQL Cluster—see Section 18.3.2, “MySQL Cluster Configuration Files”, for information about these.
A few simple commands are used in the MySQL Cluster management client (ndb_mgm) for tasks such as starting and stopping cluster nodes. See Section 18.5.2, “Commands in the MySQL Cluster Management Client”.
MySQL Cluster supports the same programming APIs and languages as the standard MySQL Server, including ODBC, .Net, the MySQL C API, and numerous drivers for popular scripting languages such as PHP, Perl, and Python. MySQL Cluster applications written using these APIs behave similarly to other MySQL applications; they transmit SQL statements to a MySQL Server (in the case of MySQL Cluster, an SQL node), and receive responses containing rows of data. For more information about these APIs, see Chapter 23, Connectors and APIs.
MySQL Cluster also supports application programming using the
NDB API, which provides a low-level C++ interface to MySQL
Cluster data without needing to go through a MySQL Server. See
The NDB API. In addition, many
NDBCLUSTER management functions are
exposed by the C-language MGM API; see
The MGM API, for more information.
MySQL Cluster (NDB 7.1 and later) also supports Java application programming using ClusterJ, which supports a domain object model of data using sessions and transactions. See Java and MySQL Cluster, for more information.
MySQL Cluster (NDB 7.2 and later) also supports
memcached, allowing developers to access data
stored in MySQL Cluster using the
interface; for more information, see
ndbmemcache—Memcache API for MySQL Cluster.
MySQL Cluster NDB 7.3 adds adapters supporting NoSQL
applications written against
MySQL Cluster as the data store. See
MySQL Cluster includes a command line client for performing basic management functions. See Section 18.4.5, “ndb_mgm — The MySQL Cluster Management Client”, and Section 18.5.2, “Commands in the MySQL Cluster Management Client”.
MySQL Cluster NDB 7.0 and later is also supported by MySQL Cluster Manager, a separate product providing an advanced command line interface that can automate many MySQL Cluster management tasks such as rolling restarts and configuration changes. For more information about MySQL Cluster Manager, see MySQL™ Cluster Manager 1.3.2 User Manual.
MySQL Cluster NDB 7.3 introduces a graphical, browser-based Auto-Installer for setting up and deploying MySQL Cluster, as part of the MySQL Cluster software distribution. For more information, see Section 18.2.1, “The MySQL Cluster Auto-Installer”.
There are two ways in which this can be done:
Clustering with MySQL is supported only by the
NDB storage engine. That is, in
order for a table to be shared between nodes in a MySQL Cluster,
the table must be created using
(or the equivalent option
It is possible to create tables using other storage engines
MyISAM) on a MySQL server being
used with a MySQL Cluster, but since these tables do not use
NDB, they do not participate in
clustering; each such table is strictly local to the individual
MySQL server instance on which it is created.
All committed transactions are logged. Therefore, although it is possible that some data could be lost in the event of a catastrophe, this should be quite limited. Data loss can be further reduced by minimizing the number of operations per transaction. (It is not a good idea to perform large numbers of operations per transaction in any case.)
It is possible but not always advisable. One of the chief reasons to run a cluster is to provide redundancy. To obtain the full benefits of this redundancy, each node should reside on a separate machine. If you place multiple nodes on a single machine and that machine fails, you lose all of those nodes. For this reason, if you do run multiple data nodes on a single machine, it is extremely important that they be set up in such a way that the failure of this machine does not cause the loss of all the data nodes in a given node group.
Given that MySQL Cluster can be run on commodity hardware loaded with a low-cost (or even no-cost) operating system, the expense of an extra machine or two is well worth it to safeguard mission-critical data. It also worth noting that the requirements for a cluster host running a management node are minimal. This task can be accomplished with a 300 MHz Pentium or equivalent CPU and sufficient RAM for the operating system, plus a small amount of overhead for the ndb_mgmd and ndb_mgm processes.
It is acceptable to run multiple cluster data nodes on a single host that has multiple CPUs, cores, or both. MySQL Cluster NDB 7.0 and later also provide a multi-threaded version of the data node binary intended for use on such systems. For more information, see Section 18.4.3, “ndbmtd — The MySQL Cluster Data Node Daemon (Multi-Threaded)”.
It is also possible in some cases to run data nodes and SQL nodes concurrently on the same machine; how well such an arrangement performs is dependent on a number of factors such as number of cores and CPUs as well as the amount of disk and memory available to the data node and SQL node processes, and you must take these factors into account when planning such a configuration.
NDB tables in MySQL
MySQL Cluster NDB 7.3 include the following:
Temporary tables are not supported; a
TEMPORARY TABLE statement using
ENGINE=NDBCLUSTER fails with an error.
The only types of user-defined partitioning supported for
NDBCLUSTER tables are
Trying to create an
NDB table using any
other partitioning type fails with an error.
FULLTEXT indexes are not supported.
Index prefixes are not supported. Only complete columns may be indexed.
Spatial indexes are not supported (although spatial columns can be used). See Section 11.5, “Extensions for Spatial Data”.
Support for partial transactions and partial rollbacks is
comparable to that of other transactional storage engines
InnoDB that can roll
back individual statements.
The maximum number of attributes allowed per table is 512. Attribute names cannot be any longer than 31 characters. For each table, the maximum combined length of the table and database names is 122 characters.
The maximum size for a table row is 14 kilobytes, not
There is no set limit for the number of rows per
NDB table. Limits on table size depend on
a number of factors, in particular on the amount of RAM
available to each data node.
For a complete listing of limitations in MySQL Cluster, see Section 18.1.6, “Known Limitations of MySQL Cluster”. See also Section 126.96.36.199, “Previous MySQL Cluster Issues Resolved in MySQL Cluster NDB 7.3”.
MySQL Cluster NDB 7.3 adds support for foreign key constraints,
comparable to that found in the
InnoDB storage engine; see
Section 188.8.131.52, “FOREIGN KEY Constraints”, for more detailed
information, as well as
Section 184.108.40.206, “Using FOREIGN KEY Constraints”. Applications
requiring foreign key support should use MySQL Cluster NDB 7.3
You can import databases into MySQL Cluster much as you would
with any other version of MySQL. Other than the limitations
mentioned elsewhere in this FAQ, the only other special
requirement is that any tables to be included in the cluster
must use the
NDB storage engine.
This means that the tables must be created with
It is also possible to convert existing tables that use other
storage engines to
one or more
statement. However, the definition of the table must be
compatible with the
storage engine prior to making the conversion. In MySQL
5.6, an additional workaround is also required; see
Section 18.1.6, “Known Limitations of MySQL Cluster”, for details.
Cluster nodes can communicate through any of three different transport mechanisms: TCP/IP, SHM (shared memory), and SCI (Scalable Coherent Interface). Where available, SHM is used by default between nodes residing on the same cluster host; however, this is considered experimental. SCI is a high-speed (1 gigabit per second and higher), high-availability protocol used in building scalable multi-processor systems; it requires special hardware and drivers. See Section 18.3.5, “Using High-Speed Interconnects with MySQL Cluster”, for more about using SCI as a transport mechanism for MySQL Cluster.
If one or more data nodes in a cluster fail, it is possible that not all cluster data nodes will be able to “see” one another. In fact, it is possible that two sets of data nodes might become isolated from one another in a network partitioning, also known as a “split-brain” scenario. This type of situation is undesirable because each set of data nodes tries to behave as though it is the entire cluster. An arbitrator is required to decide between the competing sets of data nodes.
When all data nodes in at least one node group are alive,
network partitioning is not an issue, because no single subset
of the cluster can form a functional cluster on its own. The
real problem arises when no single node group has all its nodes
alive, in which case network partitioning (the
“split-brain” scenario) becomes possible. Then an
arbitrator is required. All cluster nodes recognize the same
node as the arbitrator, which is normally the management server;
however, it is possible to configure any of the MySQL Servers in
the cluster to act as the arbitrator instead. The arbitrator
accepts the first set of cluster nodes to contact it, and tells
the remaining set to shut down. Arbitrator selection is
controlled by the
configuration parameter for MySQL Server and management server
nodes. In MySQL Cluster NDB 7.0.7 and later, you can also use
ArbitrationRank configuration parameter
to control the arbitrator selection process. For more
information about these parameters, see
Section 220.127.116.11, “Defining a MySQL Cluster Management Server”.
The role of arbitrator does not in and of itself impose any heavy demands upon the host so designated, and thus the arbitrator host does not need to be particularly fast or to have extra memory especially for this purpose.
MySQL Cluster supports all of the usual MySQL data types,
including those associated with MySQL's spatial extensions;
NDB storage engine
does not support spatial indexes. (Spatial indexes are supported
Section 11.5, “Extensions for Spatial Data”, for more information.) In
addition, there are some differences with regard to indexes when
MySQL Cluster Disk Data tables (that is, tables created with
TABLESPACE ... STORAGE DISK ENGINE=NDB or
TABLESPACE ... STORAGE DISK
ENGINE=NDBCLUSTER) have only fixed-width rows. This
means that (for example) each Disk Data table record
column requires space for 255 characters (as required for the
character set and collation being used for the table),
regardless of the actual number of characters stored therein.
See Section 18.1.6, “Known Limitations of MySQL Cluster”, for more information about these issues.
It is necessary to start each node in the cluster separately, in the following order:
Start the management node, using the ndb_mgmd command.
Start each data node with the ndbd command.
Start each MySQL Server (SQL node) using your preferred startup script, such as mysqld_safe.
Each MySQL Server must be started with the
These options cause mysqld to enable
NDBCLUSTER storage engine
support and how to connect to the management server.
Each of these commands must be run from a system shell on the
machine housing the affected node. (You do not have to be
physically present at the machine—a remote login shell can
be used for this purpose.) You can verify that the cluster is
running by starting the
management client ndb_mgm on the machine
housing the management node and issuing the
To shut down a running cluster, issue the command
SHUTDOWN in the management client.
Alternatively, you may enter the following command in a system
ndb_mgm -e "SHUTDOWN"
(The quotation marks in this example are optional, since there
are no spaces in the command string following the
-e option; in addition, the
SHUTDOWN command, like other management
client commands, is not case-sensitive.)
For more information, see Section 18.5.2, “Commands in the MySQL Cluster Management Client”, and Section 18.2.7, “Safe Shutdown and Restart of MySQL Cluster”.
The data that was held in memory by the cluster's data nodes is written to disk, and is reloaded into memory the next time that the cluster is started.
It can be helpful as a fail-safe. Only one management node controls the cluster at any given time, but it is possible to configure one management node as primary, and one or more additional management nodes to take over in the event that the primary management node fails.
See Section 18.3.2, “MySQL Cluster Configuration Files”, for information on how to configure MySQL Cluster management nodes.
Yes, as long as all machines and operating systems have the same “endianness” (all big-endian or all little-endian).
It is also possible to use software from different MySQL Cluster releases on different nodes. However, we support this only as part of a rolling upgrade procedure (see Section 18.5.5, “Performing a Rolling Restart of a MySQL Cluster”).
Yes, it is possible to do this. In the case of multiple data nodes, it is advisable (but not required) for each node to use a different data directory. If you want to run multiple SQL nodes on one machine, each instance of mysqld must use a different TCP/IP port. However, in MySQL 5.6, running more than one cluster node of a given type per machine is generally not encouraged or supported for production use.
Yes, it is possible to use DNS and DHCP for cluster hosts. However, if your application requires “five nines” availability, you should use fixed (numeric) IP addresses, since making communication between Cluster hosts dependent on services such as DNS and DHCP introduces additional potential points of failure.
IPv6 is supported for connections between SQL nodes (MySQL servers), but connections between all other types of MySQL Cluster nodes must use IPv4.
In practical terms, this means that you can use IPv6 for replication between MySQL Clusters, but connections between nodes in the same MySQL Cluster must use IPv4. For more information, see Section 18.6.3, “Known Issues in MySQL Cluster Replication”.
MySQL user accounts and privileges are normally not automatically propagated between different MySQL servers accessing the same MySQL Cluster. Beginning with MySQL Cluster NDB 7.2, MySQL Cluster provides support for distributed privileges. While privilege distribution is not enabled automatically, you can activate it by following a procedure provided in the MySQL Cluster documentation. See Section 18.5.14, “Distributed MySQL Privileges for MySQL Cluster”, for more information.
MySQL Cluster does not provide any sort of automatic failover between SQL nodes. Your application must be prepared to handlethe loss of SQL nodes and to fail over between them.
You can use the NDB native backup and restore functionality in the MySQL Cluster management client and the ndb_restore program. See Section 18.5.3, “Online Backup of MySQL Cluster”, and Section 18.4.20, “ndb_restore — Restore a MySQL Cluster Backup”.
You can also use the traditional functionality provided for this purpose in mysqldump and the MySQL server. See Section 4.5.4, “mysqldump — A Database Backup Program”, for more information.
This process monitors and, if necessary, attempts to restart the data node process. If you check the list of active processes on your system after starting ndbd, you can see that there are actually 2 processes running by that name, as shown here (we omit the output from ndb_mgmd and ndbd for brevity):
ps aux | grep ndbme 23002 0.0 0.0 122948 3104 ? Ssl 14:14 0:00 ./ndb_mgmd me 23025 0.0 0.0 5284 820 pts/2 S+ 14:14 0:00 grep ndb shell>
./ndbd -c 127.0.0.1 --initialshell>
ps aux | grep ndbme 23002 0.0 0.0 123080 3356 ? Ssl 14:14 0:00 ./ndb_mgmd me 23096 0.0 0.0 35876 2036 ? Ss 14:14 0:00 ./ndbd -c 127.0.0.1 --initial me 23097 1.0 2.4 524116 91096 ? Sl 14:14 0:00 ./ndbd -c 127.0.0.1 --initial me 23168 0.0 0.0 5284 812 pts/2 R+ 14:15 0:00 grep ndb
The ndbd process showing 0 memory and CPU
usage is the angel process. It actually does use a very small
amount of each, of course. It simply checks to see if the main
ndbd process (the primary data node process
that actually handles the data) is running. If permitted to do
so (for example, if the
configuration parameter is set to false—see
Section 18.104.22.168, “MySQL Cluster Data Node Configuration Parameters”), the angel process
tries to restart the primary data node process.