All memory allocation for a data node is performed when the node
is started. This ensures that the data node can run in a stable
manner without using swap memory, so that
NDB can be used for
latency-sensitive (realtime) applications. The following types
of memory are allocated on data node startup:
Shared global memory
Redo log buffers
Page cache for disk data records
Schema transaction memory
Undo log buffer
Block data structures
Long signal memory
Shared memory communication buffers
NDB memory manager, which
regulates most data node memory, handles the following memory
Data Memory (
Redo log buffers (
Transaction memory (
Disk access records
Each of these resources is set up with a reserved memory area and a maximum memory area. The reserved memory area can be used only by the resource for which it is reserved and cannot be shared with other resources; a given resource can never allocate more than the maximum memory allowed for the resource. A resource that has no maximum memory can expand to use all the shared memory in the memory manager.
The size of the global shared memory for these resources is
controlled by the
configuration parameter (default: 128 MB).
Data memory is always reserved and never acquires any memory
from shared memory. It is controlled using the
parameter, whose maximum is 16384 GB.
DataMemory is where records are stored,
including hash indexes (approximately 15 bytes per row), ordered
indexes (10-12 bytes per row per index), and row headers (16-32
bytes per row).
Redo log buffers also use reserved memory only; this is
controlled by the
parameter, which sets the size of the redo log buffer per LDM
thread. This means that the actual amount of memory used is the
value of this parameter multiplied by the number of LDM threads
in the data node.
Job buffers use reserved memory only; the size of this memory is
NDB, based on the
numbers of threads of various types.
Send buffers have a reserved part but can also allocate an additional 25% of shared global memory. The send buffer reserved size is calculated in two steps:
Use the value of the
TotalSendBufferMemoryconfiguration parameter (no default value) or the sum of the individual send buffers used by all individual connections to the data node. A data node is connected to all other data nodes, to all API nodes, and to all management nodes. This means that, in a cluster with 2 data nodes, 2 management nodes, and 10 API nodes each data node has 13 node connections. Since the default value for
SendBufferMemoryfor a data node connection is 2 MByte, this works out to 26 MB total.
To obtain the total reserved size for the send buffer, the value of the
ExtraSendBufferMemoryconfiguration parameter, if any (default value 0). is added to the value obtained in the previous step.
In other words, if
been set, the send buffer size is
+ ExtraSendBufferMemory; otherwise, the size of the
send buffer is equal to
number of node
connections] * SendBufferMemory) +
The page cache for disk data records uses a reserved resource
only; the size of this resource is controlled by the
configuration parameter (default 64 MB). Memory for 32 KB disk
page entries is also allocated; the number of these is
determined by the
configuration parameter (default 10).
Transaction memory has a reserved part that either is calculated
NDB, or is set explicitly using the
configuration parameter, introduced in NDB 8.0 (previously, this
value was always calculated by
NDB); transaction memory can also
use an unlimited amount of shared global memory. Transaction
memory is used for all operational resources handling
transactions, scans, locks, scan buffers, and trigger
operations. It also holds table rows as they are updated, before
the next commit writes them to data memory.
Previously, operational records used dedicated resources whose
sizes were controlled by a number of configuration parameters.
In NDB 8.0, these are all allocated from a common transaction
memory resource and can also use resources from global shared
memory. the size of this resource can be controlled using a
Reserved memory for undo log buffers can be set using the
configuration parameter. If an undo log buffer is created as
part of a
CREATE LOGFILE GROUP
SQL statement, the memory is taken from the transaction memory.
A number of resources relating to metadata for Disk Data resources also have no reserved part, and use shared global memory only. Shared global shared memory is thus shared between send buffers, transaction memory, and Disk Data metadata.
not set, it is calculated based on the following parameters:
is set explicitly, none of the configuration parameters just
listed are used to calculate memory size. In addition, the
are incompatible with
cannot be set concurrently with it; if
TransactionMemory is set and any of these
four parameters are also set in the
config.ini configuration file, the
management server cannot start. These four parameters are
deprecated in NDB 8.0; expect them to be removed from a future
release of MySQL NDB Cluster.
The transaction memory resource contains a large number of memory pools. Each memory pool represents an object type and contains a set of objects; each pool includes a reserved part allocated to the pool at startup; this reserved memory is never returned to shared global memory. Reserved records are found using a data structure having only a single level for fast retrieval, which means that a number of records in each pool should be reserved. The number of reserved records in each pool has some impact on performance and reserved memory allocation, but is generally necessary only in certain very advanced use cases to set the reserved sizes explicitly.
The size of the reserved part of the pool can be controlled by setting the following configuration parameters:
For any of the parameters just listed that is not set explicitly
config.ini, the reserved setting is
calculated as 25% of the corresponding maximum setting. For
example, if unset,
calculated as 25% of
ReservedLocalScans is calculated as 25%
ReservedTransactionBufferMemory is not
set, it is calculated as 25% of
The number of reserved records is per data node; these records
are split among the threads handling them (LDM and TC threads)
on each node. In most cases, it is sufficient to set
TransactionMemory alone, and to allow the
number of records in pools to be governed by its value.
limits the number of concurrent scans that can be active in each
TC thread. This is important in guarding against cluster
limits the number of operations that can be active at any one
time in updating transactions. (Simple reads are not affected by
this parameter.) This number needs to be limited because it is
necessary to preallocate memory for node failure handling, and a
resource must be available for handling the maximum number of
active operations in one TC thread when contending with node
failures. It is imperative that
MaxNoOfConcurrentOperations be set to the
same number on all nodes (this can be done most easily by
setting a value for it once, in the
default] section of the
config.ini global configuration file).
While its value can be increased using a rolling restart (see
Section 23.6.5, “Performing a Rolling Restart of an NDB Cluster”), decreasing it
in this way is not considered safe due to the possibility of a
node failure occurring during the rolling restart.
It is possible to limit the size of a single transaction in NDB
Cluster through the
parameter. If this is not set, the size of one transaction is
this parameter limits the total number of concurrent operations
per TC thread.
Schema memory size is controlled by the following set of configuration parameters:
The number of nodes and the number of LDM threads also have a major impact on the size of schema memory since the number of partitions in each table and each partition (and its fragment replicas) have to be represented in schema memory.
In addition, a number of other records are allocated during startup. These are relatively small. Each block in each thread contains block objects that use memory. This memory size is also normally quite small compared to the other data node memory structures.