6.4 NDB Engine Configuration

NDB memcache engine configuration options.  The NDB engine supports the following configuration options for use with memcache -e (see Section 6.3, “memcached command line options”):

• debug={true|false}

  Enables writing of debug tracing output to stderr or the memcached log file, as shown in this example:

  $> memcached -E lib/ndb_engine.so -e "debug=true"

  Because the debug output can be quite large, you should enable this option as a diagnostic tool only, and not in production.

  By default, this option is false.

• connectstring=connect_string

  This option takes as its value an NDB Cluster connection string (see NDB Cluster Connection Strings) pointing to the primary NDB Cluster—that is, the NDB Cluster in which the ndbmemcache configuration database is stored, as shown here:

  shell > memcached -E lib/ndb_engine.so -e "connectstring=sam:1186;debug=true"

  The default value is localhost:1186.

• reconf={true|false}

  Enables online reconfiguration (reloading of the configuration stored in the ndbmemcache information database).

  This option is enabled (true) by default.

• role=role_name

  Sets the role assumed by this memcached server. A role corresponds to a set of key-prefix mappings described in the ndbmemcache configuration database, identified by a role_name found in the ndbmemcache.memcache_server_roles table.

  The default role is default_role.

  An example is shown here:

  $> memcached -E lib/ndb_engine.so -e "role=db-only"

• scheduler=scheduler_name:scheduler_options

  This option controls some advanced aspects of how the NDB engine sends requests to NDB Cluster. The scheduler_name of the default scheduler or S-scheduler is S. An S-scheduler option takes the form of a single letter followed by a number; multiple S-scheduler options are separated by commas. In most cases, the default value S:c0,f0,t1 is sufficient.

  These S-scheduler options are described in the following list:

  • c: Number of connections to NDB. Possible values are in the range 0-4 inclusive, with 0 (the default) causing this number to be calculated automatically. Using 1, 2, 3, or 4 causes that number of connections to be created.

  • f: Can be either 0 or 1; setting to 1 enables force-send. The default is 0 (force-send disabled).

  • t: Sets the send-thread timer to 1-10 milliseconds (inclusive). The default is 1.

Initial Configuration.  When a the NDB engine starts up, its most important command-line arguments are the cluster connection string and server role. The connection string is used to connect to a particular cluster, called the primary cluster, which contains a configuration schema. The tables in the configuration schema are read to retrieve a set of key-prefix mappings for the given server role (see the ndbmemcache configuration schema). Those mappings instruct the server how to respond to memcache operations on particular keys, based on the leftmost part of the key. For instance, they may specify that data is stored in particular columns of a certain table. This table may be stored in the same cluster as the configuration schema, or in a different cluster. A memcache server may have connections to several different clusters, and many memcache servers may connect to a single cluster but with a variety of roles.

The ndbmemcache configuration schema.  When the memcache NDB engine starts up, it connects to a cluster, and looks for the ndbmemcache configuration schema there. If the schema is not found, it shuts down.

The schema is described (with full comments) in the file ndb_memcache_metadata.sql

The main concept of the schema is a key-prefix mapping. This takes a prefix of a memcache key and maps it to a specific container table, on a particular cluster, with a particular cache policy.

A server role is defined as a set of key-prefix mappings that a memcached server will implement.

Whenever a memcached server is started with a particular server role (from the command-line arguments), that server role must exist in the ndbmemcache.server_roles table.

The following table lists table names and descriptions for tables that belong to the ndbmemcache configuration schema.

Table 6.1 ndbmemcache configuration schema, table names and descriptions

Table Name	Description
meta	The meta table describes the version number of the ndbmemcache tables. It should be considered as a read-only table.
ndb_clusters	For each cluster, this table holds a numeric cluster-id and a connection string. The microsec_rtt column is used for performance tuning. It is recommended to use the default value of this column. See Autotuning.
cache_policies	This table maps a policy name to a set of get, set, delete, and flush policies. The policy_name column is used as the key (there is no numeric policy id). Additional information about cache policies can found in the text following the table.
containers	The containers table describes how the memcached server can use a database table to store data. Additional information about containers can found in the text following the table.
memcache_server_roles	The memcache_server_roles table maps a role name to a numeric ID and a max_tps specifier, which is used for performance tuning. See Autotuning. It is recommended to use the default value. This table also has an update_timestamp column. This column can be updated to enable online reconfiguration. See Online reconfiguration. Additional information about server roles can found in the text following the table.
key_prefixes	In this table, the leftmost part of a memcache key is paired with a cluster ID, container, and cache policy to make a key prefix mapping. Additional information about key prefix mappings can found in the text following the table.

Cache policies.  There are four policy types: get_policy, set_policy, delete_policy, and flush_from_db. These are described in the following paragraphs.

get_policy determines how the memcached server interprets GET commands. Possible values and their meanings are shown in the following list:

• cache_only: The server searches in its local cache only.

• ndb_only: The server searches in the NDB Cluster database only.

• caching: The server searches the local cache first, then the NDB Cluster database.

• disabled: GET commands are not permitted.

The set_policy determines how the memcached server interprets SET, INSERT, and REPLACE commands. Possible set_policy values and their meanings are listed here:

• cache_only: The server updates the value in its local cache only.

• ndb_only: The server updates the value stored in NDB Cluster only.

• caching: The server updates the value stored in NDB Cluster, and then stores a copy of that value in its local cache.

• disabled: SET, INSERT, and REPLACE commands are not allowed.

delete_policy describes how the memcached server interprets DELETE commands. It can take on the values shown and described in the following list:

• cache_only: The server deletes the value from its local cache only.

• ndb_only: The server deletes the value from the NDB Cluster database only.

• caching: The server deletes the value from both the database and its local cache.

• disabled: DELETE operations are not allowe.

flush_from_db determines how the memcached server interprets a FLUSH_ALL command with regard to data stored in the NDB Cluster database, as shown here:

• true: FLUSH_ALL commands cause data to be deleted from the NDB Cluster database.

• false: FLUSH_ALL commands do not affect the NDB Cluster database.

containers table columns.  The columns in the containers table are described in the following list:

• name: Name of container; primary key of table.

• db_schema: Name of database (schema) holding container table.

• db_table: table name of container table.

• key_columns: List of columns that map to the memcache key. Most keys are one-part keys, but a key can have up to four parts, in which case multiple columns are listed and separated by commas.

• value_columns: List of columns that map to the memcache value. It can also contain a comma-separated list of up to 16 value columns.

• flags: Currently unimplemented; it is intended hold either a numeric value which is used as the memcache FLAGS value for the entire container, or the name of that column of the container table used to store this value.

• increment_column: Name of the column in the container table which stores the numeric value used in memcached INCR and DECR operations. If set, this must be a BIGINT UNSIGNED column.

• cas_column Name of the column in the container table storing the memcache CAS value. If set, it must be a BIGINT UNSIGNED column.

• expire_time_column: Currently unimplemented.

Key mappings. 

• server_role_id is a numeric server role identifier which references the memcache_server_roles table

• key_prefix is a string that corresponds to the leftmost part of the memcache key. If this string is empty, then the defined prefix will be the "default prefix". The default prefix matches any memcache key that does not match some more specific prefix.

• cluster_id is an int that references the ndb_clusters table

• policy is a string that references a policy name in the cache_policies table

• container is a container name that references the containers table

The following table lists table names and descriptions for non-configuration ndbmemcache logging and container tables.

Table 6.2  ndbmemcache logging and container tables not for configuration, with descriptions

Table Name	Description
last_memcached_signon	This table is not part of the configuration schema, but is an informative logging table. It records the most recent login time of each memcached server using the configuration. ndb_node_id is an int recording the API node id of the server hostname is the hostname of the memcached server server_role is the role assigned to the server at signon time signon_time is a timestamp recording the memcached startup time In the case of online reconfiguration, signon_time records the time of the latest reconfiguration, not the time of startup. This is an unintended consequence and might be considered a bug.
demo_table	demo_table is the container table used with default key prefix in the default server role. It is used to demonstrate SET and GET operations as well as INCR, DECR, and CAS, with one key column and one value column.
demo_table_tabs	demo_table_tabs is the container table for the "demo_tabs" container, which is used with the key prefix "t:" in the default server role. It is used to demonstrate one key column with multiple value columns. In memcache operations, the value columns are represented as a tab-separated list of values.

Predefined configuration objects

Predefined clusters.  A single ndb_cluster record is predefined, referring to the primary cluster (the one where configuration data is stored) as cluster id 0. Id 0 should always be reserved for the primary cluster.

Predefined cache policies

• "memcache-only" : a policy in which all memcache operations are to use local cache only

• "ndb-only" : a policy in which all memcache operations use the NDB Cluster database, except for FLUSH_ALL, which is disabled

• "caching" : a policy with get_policy, set_policy, and delete_policy all set to "caching". FLUSH_ALL is disabled.

• "caching-with-local-deletes": a policy in which get_policy and set_policy are set to caching, but delete_policy is set to "cache-only", and FLUSH_ALL is disabled.

• "ndb-read-only": a policy in which get_policy is set to ndb_only, so that memcache GET operations use the database, but all other memcache operations are disabled

• "ndb-test": a policy like "ndb-only" with the difference that FLUSH_ALL is allowed (flush_from_db) is true. This is the only predefined policy with flush_from_db enabled. This policy is enabled by default for the default server role, so taht the entire memcache command set can be demonstrated.

Predefined containers

• "demo_table": a container using the table ndbmemcache.demo_table as a container table

• "demo_tabs": a container using the table ndbmemcache.demo_table_tabs as a container table

Predefined memcache server roles and their key prefixes

• "default_role" (role id 0)

  "": The empty (default) prefix uses the ndb-test policy and the demo_table container

  "mc:" Memcache keys beginning with "mc:" are treated according to the memcache-only cache policy

  "t:" Memcache keys beginning with "t:" use the ndb-test cache policy and the demo_tabs container

• The "db-only" role (role id 1)

  "": the empty (default) prefix uses the ndb-only role and demo_table container

  The "t:" prefix uses the ndb-only role and demo_tabs container

• The "mc-only" role (role id 2)

  "": The empty (default) prefix uses local caching only for all keys

• The "ndb-caching" role (role id 3)

  "": The empty (default) prefix uses the "caching" cache policy and "demo_table" container for all keys

Configuration versioning and upgrade.  The configuration schema is versioned, and the version number is stored in the ndbmemcache.meta table. The NDB Engine begins the configuration process by reading the schema version number from this table. As a rule, newer versions of the NDB engine will remain compatible with older versions of the configuration schema.

STABILITY NOTE: consider this section "unstable" & subject to change

Performance Tuning.  Two parameters are used to tune performance of the NDB memcache engine. The parameters are stored in the configuration schema: the "usec_rtt" value of a particular cluster, and the "max_tps" value of a memcache server role. These values are currently used in two ways: to configure the number of connections to each cluster, and to configure a particular fixed number of concurrent operations supported from each connection.

Autotuning.  Autotuning uses an estimated round trip time between cluster data nodes and a target rate of throughput to determine the ideal number of cluster connections and transactions per connection for a given workload. Autotuning parameters are described in the next few paragraphs.

• usec_rtt: The round trip time, in microseconds, between cluster nodes. The default value is 250, which is typical for an NDB Cluster on a local switched ethernet. To represent a cluster with higher inter-node latency (wider area), a higher value should be used.

• max_tps: The desired throughput from a server. This value is a heuristic, and does not in any way express either a floor or a ceiling on the actual throughput obtained. The default value (100000) is reasonable in most cases.

These values are used, as described in the next few paragraphs, to calculate an optimum number of cluster connections with a given transactions-per-second capacity..

Number of cluster connections.  The NDB Engine scheduler attempts to open 1 cluster connection per 50000 transactions per second (TPS). This behavior can be overridden by using a scheduler configuration string (see Section 6.4, “NDB Engine Configuration”.) If the scheduler fails to open a second or subsequent connection to a cluster—for example, because a node id is not available—this is not a fatal error; it will run with only the connections actually opened.

Number of transactions per connection.  We assume that a transaction takes 5 times the cluster round trip time to complete. We can obtain the total number of in-flight transactions by dividing the server's max_tps by 5 * rtt (in seconds). These in-flight transaction objects are evenly distributed among the cluster connections.

Tuning example.  The following example starts with the default values usec_rtt = 250 and max_tps = 100000, and assumes a memcached server with 4 worker threads.

• 100000 TPS divided by 50000 is 2, and the server opens two NDB cluster connections.

• Transaction time in microseconds = 250 µs round trip time * 5 round trips = 1250 µs.

• Transactions per connection per second = 1000000 / tx_time_in_µsec = 1000000 / 1250 = 800.

• Total Ndb objects = max_tps / tx_per_ndb_per_sec = 100000 / 800 = 125.

• 125 Ndb objects / 2 connections = 63 Ndb objects per connection (rounding upward).

• (Rounding upward once more) each of 4 worker threads gets 32 Ndb objects

Online reconfiguration.  It is possible to reconfigure the key-prefix mappings of a running NDB engine without restarting it. This is done by committing a change to the configuration schema, and then updating the update_timestamp column of a particular server role in the memcache server roles table. The updating of the timestamp causes an event trigger to fire, so that the memcache server receives notification of the event.

Online reconfiguration can be disabled by using the -e reconf=false option on the command line.

Online reconfiguration can be used to connect to new clusters and to create new key-prefix mappings. However, it cannot be used to reset autotuning values on existing connections.

Online reconfiguration is a risky operation that could result in memcache server crashes or data corruption, and is used extensively in the mysql test suite. However, it is not recommended for reconfiguring a production server under load.

The stats reconf command can be run before and after online reconfiguration to verify that the version number of the running configuration has increased. Verification of reconfiguration is also written into the memcached log file.