We started our series on MySQL Docker deploments by showing how to deploy and use MySQL locally with docker-compose in Docker Compose and App Deployment with MySQL. Docker-compose itself is limited to one machine and it does not solve cross-node networking or span multiple datacenters. This is a job for so called cluster schedulers, i.e. a software component managing deployment of containers across a cluster. A cluster manager’s main goal is to provide an abstraction layer across individual computing nodes (especially cloud-provisioned ones) and achieve better resource utilisation while promoting the idea of immutable deployment artifacts (containers).
HashiCorp products have gained quite some popularity and cover a wide range of DevOps areas. Terraform allows for writing, planning, and creating infrastructure as code. Consul handles service discovery and configuration management. Vault is a “tool for managing secrets.” HashiCorp Nomad is a cluster scheduler that will schedule containers (or executables) across nodes, environments, and datacenters. The setup and maintenance of a high-availability cluster is clearly out of scope so in this post we follow up and show how to use the mysql-server Docker image for application deployment with Nomad running on a local machine. While we are still limited to a local machine the main idea is to show how containers can be scheduled in a production-like system. It will be possible to schedule our examples to a remote Nomad server just as well.
The full example is available here (and works out of the box for Linux): Source files for example on Github
In order to run it we require a working Docker environment and both the HashiCorp Consul binary and the HashiCorp Nomad binary. The binaries can be installed via make, we’ll show how later.
Cluster Schedulers
Cluster schedulers have become increasingly popular in recent years. The basic job of a scheduler is to abstract away actual hardware and even cloud providers. A scheduler places workloads on a group of machines as efficiently as possible. While scalability is definitely a factor here, it can be argued that the impact on deployment is just as important an aspect and as such may make sense even in less large-scale settings. Deployment can be and is done in many ways, many would agree though that environment parity (I want my staging environment as similar to my production environment), speed (I want to deploy faster), and lifecycle questions (how do I redeploy and what happens to my traffic), as well as deployment of immutable artifacts (I want to schedule (docker) containers) are central requirements here. And schedulers help with all of those. A somewhat more in-depth evaluation of pros and cons of cluster schedulers focusing on Nomad is given in Cluster Schedulers. In addition building deployment pipelines towards a cluster scheduler rather than a particular cloud provider’s services is a nice way to avoid vendor lock-in in the time of the cloud.
There are multiple projects out there, Kubernetes is the most popular but alternatives such as Mesos/Marathon or Nomad often share many technical aspects. One thing they all have in common is that a production-scale cluster is not trivial to either set up or maintain. This is a task left to either one’s infrastructure department or even better to cloud providers since, for example, hosted Kubernetes installations can be purchased from multiple major players now.
The advantages of Nomad are that it is maybe a bit less monolithic and less complex to set up than the alternatives and that it plays very well together with other Hashicorp products such as Consul and Vault.
In any case, similar concepts as the one mentioned here exist in other schedulers as well.
Our example makes use of both Consul and Nomad because they are easy to set up and to a large extent follow the unix philosophy of doing one thing well (and we do like that philosophy). In combination they can solve many of the problems inherent to large-scale distributed deployment systems. Consul is used as a backend for service registry, i.e. as a central stateful component which is always aware of everything running in our environment. This is a central requirement of any cluster scheduler.
Running Basic Infrastructure
To run our example, we do the following:
- Run Consul locally for service discovery (requires the consul binary)
- Run Nomad locally (requires the nomad binary)
- Schedule jobs on our local machine (send .nomad files to the nomad server)
Running the example requires make, wget, and unzip, as well as a working Docker installation on your machine. First, check out the repo:
git clone https://github.com/neumayer/mysql-nomad-examples
cd mysql-nomad-examples/mysql-serverDownload Consul and Nomad binaries:
makeFirst, we start Consul on the host machine (which Nomad will use as a backend). It will run on 127.0.0.1:
./consul agent -server -client 127.0.0.1 -advertise 127.0.0.1 -data-dir /tmp/consul -ui -bootstrapVerify that Consul is up and running by pointing your browser to:
http://127.0.0.1:8500This shows an overview of running services and should so far only include Consul itself.
Next, in another terminal, we run Nomad (both client and server mode, this means the same process will handle server state and client allocations, also running on 127.0.0.1:
./nomad agent -config nomad.confWe now have Nomad running on that interface, and can verify by checking the Nomad gui:
http://127.0.0.1:4646/We can now go back to the Consul ui and see that both Nomad (the component keeping state and making scheduling decisions), and nomad-client (the component running the actual jobs) have been registered.
Our Nomad cluster is ready to accept jobs now. Jobs are specified in .nomad files and sent to the Nomad server to be placed in the cluster (nomad run job.nomad).
In this example we will run the following jobs:
- MySQL server
- Example application talking to the db container (via Consul)
A Nomad Job for MySQL
Configuration of a a MySQL Nomad job largely corresponds to the docker-compose example but uses different syntax:
group "mysql-server" {
count = 1
task "mysql-server" {
driver = "docker"
config {
image = "mysql/mysql-server:8.0"
port_map {
db = 3306
}
volumes = [
"docker-entrypoint-initdb.d/:/docker-entrypoint-initdb.d/",
]
}
template {
data = <<EOH
CREATE DATABASE dbwebappdb;
CREATE USER 'dbwebapp'@'%' IDENTIFIED BY 'dbwebapp';
GRANT ALL PRIVILEGES ON dbwebappdb.* TO 'dbwebapp'@'%';
EOH
destination = "/docker-entrypoint-initdb.d/db.sql"
}
resources {
cpu = 500
memory = 1024
network {
mbits = 10
port "db" {}
}
}
}We use three important blocks here:
- driver/config
- template
- resources
The driver block specifies whether to use docker or a standard executable (exec driver). Both will use cgroups internally and provide a similar level of isolation). Additional parameters can be passed to the container via environment variables.
The template block is used to place a file on the instance the job will be allocated to. In this case we just write a static file. The real strength of this, however lies in its ability to re-render the file based on interpolation, we will show an example in the example app .nomad file.
Nomad jobs can also be assigned resources on the node, i.e. it is possible to specify how much network bandwidth, cpu, or memory a job is assigned.
One of the main questions in the context of schedulers when it comes to database deployments is how state is handled. After all we would rather not lose our database files. State in Nomad is handled via the concept of sticky volumes. When a task is allocated, it can make use of an ephemeral disk. This disk can be marked sticky which means that future allocations of the same task will happen on the node with the volume. In addition it has a migrate flag. If that one is set to true, the scheduler will make an effort to sync the volume to another node. This is relevant in the case that the task can not be scheduled on the same node. In our example we specify the following ephemeral disk:
ephemeral_disk {
migrate = true
size = 300
sticky = true
} We consider experimentation on how well this works in practice out of scope for this blog post, but have plans of doing a more large-scale evaluation of the volume problem in the future.
A Nomad Job for Our Example App
The image for the example app is available on Docker Hub under neumayer/dbwebapp. For the purpose of this post it is enough to simply pull the container from there (or rather let Nomad pull the container for you). If you want to build it yourself, the full sources for the example application are available on Github as Dbwebapp. It is a very simple application that just connects to a given database on start and reports itself healthy if it can.
group "dbwebapp" {
count = 2
task "dbwebapp" {
driver = "docker"
env {
DBUSER = "dbwebapp"
DBPASS = "dbwebapp"
}
template {
data = <<EOH
HOST="{{ range service "mysql-server" }}{{ .Address }}{{ end }}"
PORT="{{ range service "mysql-server" }}{{ .Port }}{{ end }}"
destination = "mysql-server.env"
env = true
}
config {
image = "neumayer/dbwebapp"
port_map {
web = 8080
}
}
}This config is similar to the one for the mysql-server job. One important difference is that the template block is dynamically rendered using Consul. In our case we use it to look up the address and port of the mysql-server container. If the values in Consul change, for example the mysql-server container gets scheduled on another client, the template is re-rendered and the task restarted. Another advantage of this mechanism is its integration with Vault (the HashiCorp secret management solution).
Running jobs
We are now ready to schedule jobs to our local scheduler. Nomad will register these services with consul. All the network workarounds so far guarantee that the rest of the examples an as well be deployed to a real production cluster. All jobs will be started on the local machine inside the Docker bridge network.
nomad run mysql-server.nomad
nomad run dbwebapp.nomadBoth the Nomad and Consul gui are updated to reflect running containers and services, respectively.
We can verify that the dbwebapp actually can connect to its db by running:
nomad logs -stderr -job dbwebappThis will output all the logs for the given job, and look like this:
2017/12/04 09:06:36 Initialising dbwebapp.
2017/12/04 09:06:36 Connecting to dbwebapp:xxxx@tcp(x.x.x.x:28098)/
2017/12/04 09:06:36 Pinging db x.x.x.x.
2017/12/04 09:06:36 Connected to db.
2017/12/04 09:06:36 Starting dbwebapp server.For now we scheduled one instance of our example app. Nomad makes it easy to scale horizontally. Each task group is assigned a count value which directly corresponds to the scheduled number of instances. To achieve this in our example we can change:
group "dbwebapp" {
count = 1to:
group "dbwebapp" {
count = 8and rerun the job:
nomad run dbwebapp.nomadAs a result we now run eight containers instead of one. In our example we are limited to one machine and still have the limitations of that machine. In a real production setup there would be tens or more Nomad client machines on which this workload would be distributed.
Production Setting Considerations and Outlook
We showed how to use Nomad to schedule MySQL containers locally and how to access them in a dynamic environment. Our example was specific to local test deployment; however, all job files in use here could be deployed to a real production cluster just as well. While the particular choice of scheduler naturally has a large impact on any the overall evaluation of this way of deploying databases, many of the underlying concepts should be reasonably similar.
This is an example only, and when running a similar setup in production, the following issues should be addressed:
- Consul should be a cluster (i.e. three or five instances)
- Nomad should be a cluster (i.e. three or five server instances and a dynamic number of client instances)
- Networking should be planned wrt. to both security and routing/proxying
- Backup for MySQL data (before considering any kind of container deployment for MySQL we strongly recommend to have both backups and restore processes in place)
- Proper risk assessment and evaluation of your storage/volume requirements
An area deserving particular scrutiny is of course volume handling. For this post we did no testing and experimentation other than seeing a container scheduled on the same host reusing and existing volume (since there only is one host involved). An in-depth evaluation would be needed and Nomad guarantees should be compared to others’.