NDB memcache engine configuration options.
The NDB engine supports the following configuration options for
use with memcache
Section 6.3, “memcached command line options”):
Enables writing of debug tracing output to
stderror the memcached log file, as shown in this example:
$> memcached -E lib/ndb_engine.so -e "debug=true"
Because the debug output can be quite large, you should enable this option as a diagnostic tool only, and not in production.
By default, this option is
This option takes as its value an NDB Cluster connection string (see NDB Cluster Connection Strings) pointing to the primary NDB Cluster—that is, the NDB Cluster in which the
ndbmemcacheconfiguration database is stored, as shown here:
shell > memcached -E lib/ndb_engine.so -e "connectstring=sam:1186;debug=true"
The default value is
Enables online reconfiguration (reloading of the configuration stored in the
This option is enabled (
true) by default.
Sets the role assumed by this memcached server. A role corresponds to a set of key-prefix mappings described in the
ndbmemcacheconfiguration database, identified by a
role_namefound in the
The default role is
An example is shown here:
$> memcached -E lib/ndb_engine.so -e "role=db-only"
This option controls some advanced aspects of how the NDB engine sends requests to NDB Cluster. The
scheduler_nameof the default scheduler or S-scheduler is
S. An S-scheduler option takes the form of a single letter followed by a number; multiple S-scheduler options are separated by commas. In most cases, the default value
These S-scheduler options are described in the following list:
c: Number of connections to
NDB. Possible values are in the range 0-4 inclusive, with 0 (the default) causing this number to be calculated automatically. Using 1, 2, 3, or 4 causes that number of connections to be created.
f: Can be either 0 or 1; setting to 1 enables force-send. The default is 0 (force-send disabled).
t: Sets the send-thread timer to 1-10 milliseconds (inclusive). The default is 1.
Initial Configuration. When a the NDB engine starts up, its most important command-line arguments are the cluster connection string and server role. The connection string is used to connect to a particular cluster, called the primary cluster, which contains a configuration schema. The tables in the configuration schema are read to retrieve a set of key-prefix mappings for the given server role (see the ndbmemcache configuration schema). Those mappings instruct the server how to respond to memcache operations on particular keys, based on the leftmost part of the key. For instance, they may specify that data is stored in particular columns of a certain table. This table may be stored in the same cluster as the configuration schema, or in a different cluster. A memcache server may have connections to several different clusters, and many memcache servers may connect to a single cluster but with a variety of roles.
The ndbmemcache configuration schema. When the memcache NDB engine starts up, it connects to a cluster, and looks for the ndbmemcache configuration schema there. If the schema is not found, it shuts down.
The schema is described (with full comments) in the file ndb_memcache_metadata.sql
The main concept of the schema is a key-prefix mapping. This takes a prefix of a memcache key and maps it to a specific container table, on a particular cluster, with a particular cache policy.
A server role is defined as a set of key-prefix mappings that a memcached server will implement.
Whenever a memcached server is started with a particular server role (from the command-line arguments), that server role must exist in the ndbmemcache.server_roles table.
The following table lists table names and descriptions for tables
that belong to the
Table 6.1 ndbmemcache configuration schema, table names and descriptions
||The meta table describes the version number of the ndbmemcache tables. It should be considered as a read-only table.|
For each cluster, this table holds a numeric cluster-id and a connection string. The microsec_rtt column is used for performance tuning. It is recommended to use the default value of this column. See Autotuning.
This table maps a policy name to a set of get, set,
delete, and flush policies. The
Additional information about cache policies can found in the text following the table.
The containers table describes how the memcached server can use a database table to store data.
Additional information about containers can found in the text following the table.
This table also has an update_timestamp column. This column can be updated to enable online reconfiguration. See Online reconfiguration.
Additional information about server roles can found in the text following the table.
In this table, the leftmost part of a memcache key is paired with a cluster ID, container, and cache policy to make a key prefix mapping.
Additional information about key prefix mappings can found in the text following the table.
There are four policy types:
flush_from_db. These are described in the
get_policy determines how the memcached server
GET commands. Possible values and
their meanings are shown in the following list:
cache_only: The server searches in its local cache only.
ndb_only: The server searches in the NDB Cluster database only.
caching: The server searches the local cache first, then the NDB Cluster database.
GETcommands are not permitted.
set_policy determines how the memcached
set_policy values and their
meanings are listed here:
cache_only: The server updates the value in its local cache only.
ndb_only: The server updates the value stored in NDB Cluster only.
caching: The server updates the value stored in NDB Cluster, and then stores a copy of that value in its local cache.
REPLACEcommands are not allowed.
delete_policy describes how the memcached
DELETE commands. It can take
on the values shown and described in the following list:
cache_only: The server deletes the value from its local cache only.
ndb_only: The server deletes the value from the NDB Cluster database only.
caching: The server deletes the value from both the database and its local cache.
DELETEoperations are not allowe.
flush_from_db determines how the memcached
server interprets a
FLUSH_ALL command with
regard to data stored in the NDB Cluster database, as shown here:
FLUSH_ALLcommands cause data to be deleted from the NDB Cluster database.
FLUSH_ALLcommands do not affect the NDB Cluster database.
containers table columns.
The columns in the
containers table are
described in the following list:
name: Name of container; primary key of table.
db_schema: Name of database (schema) holding container table.
db_table: table name of container table.
key_columns: List of columns that map to the memcache key. Most keys are one-part keys, but a key can have up to four parts, in which case multiple columns are listed and separated by commas.
value_columns: List of columns that map to the memcache value. It can also contain a comma-separated list of up to 16 value columns.
flags: Currently unimplemented; it is intended hold either a numeric value which is used as the memcache
FLAGSvalue for the entire container, or the name of that column of the container table used to store this value.
increment_column: Name of the column in the container table which stores the numeric value used in memcached
DECRoperations. If set, this must be a
cas_columnName of the column in the container table storing the memcache CAS value. If set, it must be a
expire_time_column: Currently unimplemented.
server_role_id is a numeric server role identifier which references the memcache_server_roles table
key_prefix is a string that corresponds to the leftmost part of the memcache key. If this string is empty, then the defined prefix will be the "default prefix". The default prefix matches any memcache key that does not match some more specific prefix.
cluster_id is an int that references the ndb_clusters table
policy is a string that references a policy name in the cache_policies table
container is a container name that references the containers table
The following table lists table names and descriptions for
ndbmemcache logging and
Table 6.2 ndbmemcache logging and container tables not for configuration, with descriptions
This table is not part of the configuration schema, but is an informative logging table. It records the most recent login time of each memcached server using the configuration.
||demo_table is the container table used with default key prefix in the default server role. It is used to demonstrate SET and GET operations as well as INCR, DECR, and CAS, with one key column and one value column.|
||demo_table_tabs is the container table for the "demo_tabs" container, which is used with the key prefix "t:" in the default server role. It is used to demonstrate one key column with multiple value columns. In memcache operations, the value columns are represented as a tab-separated list of values.|
Predefined configuration objects
Predefined clusters. A single ndb_cluster record is predefined, referring to the primary cluster (the one where configuration data is stored) as cluster id 0. Id 0 should always be reserved for the primary cluster.
Predefined cache policies
"memcache-only" : a policy in which all memcache operations are to use local cache only
"ndb-only" : a policy in which all memcache operations use the NDB Cluster database, except for FLUSH_ALL, which is disabled
"caching" : a policy with get_policy, set_policy, and delete_policy all set to "caching". FLUSH_ALL is disabled.
"caching-with-local-deletes": a policy in which get_policy and set_policy are set to caching, but delete_policy is set to "cache-only", and FLUSH_ALL is disabled.
"ndb-read-only": a policy in which get_policy is set to ndb_only, so that memcache GET operations use the database, but all other memcache operations are disabled
"ndb-test": a policy like "ndb-only" with the difference that FLUSH_ALL is allowed (flush_from_db) is true. This is the only predefined policy with flush_from_db enabled. This policy is enabled by default for the default server role, so taht the entire memcache command set can be demonstrated.
"demo_table": a container using the table ndbmemcache.demo_table as a container table
"demo_tabs": a container using the table ndbmemcache.demo_table_tabs as a container table
Predefined memcache server roles and their key prefixes
"default_role" (role id 0)
"": The empty (default) prefix uses the ndb-test policy and the demo_table container
"mc:" Memcache keys beginning with "mc:" are treated according to the memcache-only cache policy
"t:" Memcache keys beginning with "t:" use the ndb-test cache policy and the demo_tabs container
The "db-only" role (role id 1)
"": the empty (default) prefix uses the ndb-only role and demo_table container
The "t:" prefix uses the ndb-only role and demo_tabs container
The "mc-only" role (role id 2)
"": The empty (default) prefix uses local caching only for all keys
The "ndb-caching" role (role id 3)
"": The empty (default) prefix uses the "caching" cache policy and "demo_table" container for all keys
Configuration versioning and upgrade. The configuration schema is versioned, and the version number is stored in the ndbmemcache.meta table. The NDB Engine begins the configuration process by reading the schema version number from this table. As a rule, newer versions of the NDB engine will remain compatible with older versions of the configuration schema.
STABILITY NOTE: consider this section "unstable" & subject to change
Performance Tuning. Two parameters are used to tune performance of the NDB memcache engine. The parameters are stored in the configuration schema: the "usec_rtt" value of a particular cluster, and the "max_tps" value of a memcache server role. These values are currently used in two ways: to configure the number of connections to each cluster, and to configure a particular fixed number of concurrent operations supported from each connection.
Autotuning. Autotuning uses an estimated round trip time between cluster data nodes and a target rate of throughput to determine the ideal number of cluster connections and transactions per connection for a given workload. Autotuning parameters are described in the next few paragraphs.
usec_rtt: The round trip time, in microseconds, between cluster nodes. The default value is 250, which is typical for an NDB Cluster on a local switched ethernet. To represent a cluster with higher inter-node latency (wider area), a higher value should be used.
max_tps: The desired throughput from a server. This value is a heuristic, and does not in any way express either a floor or a ceiling on the actual throughput obtained. The default value (100000) is reasonable in most cases.
These values are used, as described in the next few paragraphs, to calculate an optimum number of cluster connections with a given transactions-per-second capacity..
Number of cluster connections. The NDB Engine scheduler attempts to open 1 cluster connection per 50000 transactions per second (TPS). This behavior can be overridden by using a scheduler configuration string (see Section 6.4, “NDB Engine Configuration”.) If the scheduler fails to open a second or subsequent connection to a cluster—for example, because a node id is not available—this is not a fatal error; it will run with only the connections actually opened.
Number of transactions per connection.
We assume that a transaction takes 5 times the cluster round
trip time to complete. We can obtain the total number of
in-flight transactions by dividing the server's
5 * rtt (in
seconds). These in-flight transaction objects are evenly
distributed among the cluster connections.
The following example starts with the default values
usec_rtt = 250 and
= 100000, and assumes a memcached server with 4 worker threads.
100000 TPS divided by 50000 is 2, and the server opens two NDB cluster connections.
Transaction time in microseconds = 250 µs round trip time * 5 round trips = 1250 µs.
Transactions per connection per second = 1000000 / tx_time_in_µsec = 1000000 / 1250 = 800.
Total Ndb objects = max_tps / tx_per_ndb_per_sec = 100000 / 800 = 125.
125 Ndb objects / 2 connections = 63 Ndb objects per connection (rounding upward).
(Rounding upward once more) each of 4 worker threads gets 32 Ndb objects
It is possible to reconfigure the key-prefix mappings of a
running NDB engine without restarting it. This is done by
committing a change to the configuration schema, and then
update_timestamp column of a
particular server role in the memcache server roles table. The
updating of the timestamp causes an event trigger to fire, so
that the memcache server receives notification of the event.
Online reconfiguration can be disabled by using the
reconf=false option on the command line.
Online reconfiguration can be used to connect to new clusters and to create new key-prefix mappings. However, it cannot be used to reset autotuning values on existing connections.
Online reconfiguration is a risky operation that could result in memcache server crashes or data corruption, and is used extensively in the mysql test suite. However, it is not recommended for reconfiguring a production server under load.
stats reconf command can be run before and
after online reconfiguration to verify that the version number of
the running configuration has increased. Verification of
reconfiguration is also written into the memcached log file.