In this blog post we will highlight some of the details of the MCM 1.4.2 release.
Progress reporting
First, some eye candy. With MCM 1.4.2 we added simple progress reporting. There is now a simple way to get some sense of progress, especially interesting for long-running commands. There is a plain variant, and one which includes an ascii-art progressbar.
|
1
2
3
4
5
6
7
8
9
10
11
12
13
|
mcm> show status --progress mycluster; +---------------+-----------+----------+ | Command | Status | Progress | +---------------+-----------+----------+ | start cluster | executing | 64% | +---------------+-----------+----------+ mcm> show status --progressbar mycluster; +---------------+-----------+----------------------------+ | Command | Status | Progress | +---------------+-----------+----------------------------+ | start cluster | executing | 78% [############### ] | +---------------+-----------+----------------------------+ |
Combine this with something like the linux/unix watch command, and you can easily monitor cluster health directly in the CLI. Please note that monitoring using a very short timer, combined with a log-level of debug/info, will cause excessive logging:
|
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
|
mcmd@shark1:/foo/install$ watch -n 1 "./mcm1.4.2/bin/mcm --table -e 'show status -r mycluster;show status --progressbar mycluster'; echo '\n'; ./cluster/bin/ndb_mgm -t 1 -e 'all report memory'" Every 1.0s: ./mcm1.4.2/bin/mcm --table -e ... shark1: Wed Feb 22 12:29:52 2017 MySQL Cluster Manager client started. This wrapper will spawn the mysql client to connect to mcmd mcm> show status --process mycluster; +--------+----------+--------+---------+-----------+---------+ | NodeId | Process | Host | Status | Nodegroup | Package | +--------+----------+--------+---------+-----------+---------+ | 49 | ndb_mgmd | cod | running | | 7.5.5 | | 1 | ndbmtd | shark1 | running | 0 | 7.5.5 | | 2 | ndbmtd | shark2 | running | 0 | 7.5.5 | | 50 | mysqld | whale1 | running | | 7.5.5 | | 51 | mysqld | whale2 | running | | 7.5.5 | | 52 | ndbapi | * | added | | | +--------+----------+--------+---------+-----------+---------+ +---------------+----------+-----------------------------+ | Command | Status | Progress | +---------------+----------+-----------------------------+ | start cluster | finished | 100% [####################] | +---------------+----------+-----------------------------+ Connected to Management Server at: localhost:1186 Node 1: Data usage is 0%(23 32K pages of total 2560) Node 1: Index usage is 0%(20 8K pages of total 2336) Node 2: Data usage is 0%(23 32K pages of total 2560) Node 2: Index usage is 0%(20 8K pages of total 2336) |
Update process
Like it or not, one of the main motivations for running a distributed system in production is the ability to handle disasters and disaster recovery. In production it can be more important to make your service available again, than exactly how. In desperation the missing process is restarted on the commandline to restore availability. When the dust finally settles, you clean up.
In earlier versions there was no way for MCM to (re)discover the restarted process, meaning yet another restart was needed for MCM to resume management of the lost process. With very large databases this could mean an unwanted service and durability degradation, since a datanode may take a significant amount of time to restart.
With MCM 1.4.2 we add the ability to reattach a process started manually on the commandline, and put it under MCM control once again. The new functionality builds on the import cluster functionality of MCM and impose similar restrictions on the reattached process. Please have a look at the documentation for details and limitations.
|
1 |
update process --pid=12345 mycluster |
Example:
Assume your mission critical cluster is up and running in production. You did ensure your backups are good, actually are restoreable, and stored in a safe place, right?
|
1
2
3
4
5
6
7
8
9
10
11
12
|
mcm> show status --process mycluster; +--------+----------+--------+---------+-----------+---------+ | NodeId | Process | Host | Status | Nodegroup | Package | +--------+----------+--------+---------+-----------+---------+ | 49 | ndb_mgmd | cod | running | | 7.5.5 | | 1 | ndbmtd | shark1 | running | 0 | 7.5.5 | | 2 | ndbmtd | shark2 | running | 0 | 7.5.5 | | 50 | mysqld | whale1 | running | | 7.5.5 | | 51 | mysqld | whale2 | running | | 7.5.5 | | 52 | ndbapi | * | added | | | +--------+----------+--------+---------+-----------+---------+ 6 rows in set (0.09 sec) |
Disaster strikes, at the worst time possible, and host shark1 does not survive the carnage for some reason. Necessary actions are taken to quickly replace the hardware, and the failed datanode is restarted on the commandline to restore availability. All looks well from the ndb_mgm commandline client. Phew!
|
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
|
mcmd@shark1:/foo/install$ ps -ef | grep ndbmtd mcmd 5742 1 1 09:17 pts/18 00:00:03 /foo/install/cluster/bin/ndbmtd --ndb-connectstring=cod:1186 --ndb-nodeid=1 --foreground=true mcmd@shark1:/foo/install$ ./cluster/bin/ndb_mgm -c cod:1186 -- NDB Cluster -- Management Client -- ndb_mgm> show Cluster Configuration --------------------- [ndbd(NDB)] 2 node(s) id=1 @10.0.0.1 (mysql-5.7.17 ndb-7.5.5, Nodegroup: 0, *) id=2 @10.0.0.2 (mysql-5.7.17 ndb-7.5.5, Nodegroup: 0) [ndb_mgmd(MGM)] 1 node(s) id=49 @10.0.0.3 (mysql-5.7.17 ndb-7.5.5) [mysqld(API)] 3 node(s) id=50 @10.0.0.4 (mysql-5.7.17 ndb-7.5.5) id=51 @10.0.0.5 (mysql-5.7.17 ndb-7.5.5) id=52 (not connected, accepting connect from any host) |
… but, hang on. MCM doesn’t see it!
|
1
2
3
4
5
6
7
8
9
10
11
12
|
mcm> show status --process mycluster; +--------+----------+--------+---------+-----------+---------+ | NodeId | Process | Host | Status | Nodegroup | Package | +--------+----------+--------+---------+-----------+---------+ | 49 | ndb_mgmd | cod | running | | 7.5.5 | | 1 | ndbmtd | shark1 | failed | 0 | 7.5.5 | | 2 | ndbmtd | shark2 | running | 0 | 7.5.5 | | 50 | mysqld | whale1 | running | | 7.5.5 | | 51 | mysqld | whale2 | running | | 7.5.5 | | 52 | ndbapi | * | added | | | +--------+----------+--------+---------+-----------+---------+ 6 rows in set (0.05 sec) |
You have to let MCM know the pid of the restarted process using the new update process command:
|
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
|
mcm> update process --pid=5742 1 mycluster; +------------------------------+ | Command result | +------------------------------+ | Process updated successfully | +------------------------------+ 1 row in set (3.82 sec) mcm> show status --process mycluster; +--------+----------+--------+---------+-----------+---------+ | NodeId | Process | Host | Status | Nodegroup | Package | +--------+----------+--------+---------+-----------+---------+ | 49 | ndb_mgmd | cod | running | | 7.5.5 | | 1 | ndbmtd | shark1 | running | 0 | 7.5.5 | | 2 | ndbmtd | shark2 | running | 0 | 7.5.5 | | 50 | mysqld | whale1 | running | | 7.5.5 | | 51 | mysqld | whale2 | running | | 7.5.5 | | 52 | ndbapi | * | added | | | +--------+----------+--------+---------+-----------+---------+ 6 rows in set (0.08 sec) |
The restarted process has been successfully reattached to mcm and service fully restored.
Do note that the failed datanode must be restarted using the --foreground=true option, as to restart without an angel.
Collect logs
There are some significant improvements to the existing collect logs feature in MCM 1.4.2.
The port number used by the collector thread is now user configurable using the --copy-port=4567 option, or as a setting in the agent’s defaults-file. This enables log collection in setups where a firewall would block access to the random port number designated by previous versions of MCM.
With MCM 1.4.2 the sender threads on each agent send more metadata to the collector thread about the files being collected. This improves progress tracking and detection of partially transferred file sets, or partially transferred files.
Reinitializing mysqld datadir
Assume your mysqld datadir was corrupted, and the mysqld refuses to start. Earlier versions required manual intervention to reinitialize the mysqlds datadir, but with MCM 1.4.2 this may now be done using
start process --initial. It requires datadir to be empty for this command to succeed. If there are files left in datadir, MCM will refuse the reinitialization and throw an error. Do note that mcmd does not rerun any restore operation on non-NDB tables, so you still need to manually restore any non-system InnoDB tables that you need, from your safely stored backups…
MCM 1.4.1 also apply the same automation if you fully replace a mysqld host or disk. When mcmd rejoins an existing site without any configuration metadata, it automatically recovers the configuration metadata from the other members and restarts any processes that should run on the replaced host. MCM 1.4.2 will automatically reinitialize any missing mysqld datadirs on the replaced host, as if you ran start process --initial .
More details
We have also done a number of smaller improvements, and as always fixed bugs. More details are available in the the MCM 1.4.2 Release Notes.
And updated documentation is available here.
Enjoy!