Traditionally infrastructure management is a manual task, with sysadmins managing rather static servers. The automation capabilities of modern cloud platforms have changed this way of working: Infrastructure is often described “as code,” e.g. in a git repository, and changes are made by infrastructure management systems automatically. As a result infrastructure is much less static and turnaround times are much shorter.
Most commonly infrastructure testing frameworks are used to verify the state of machine images (Amazon Machine Images, Google Compute Images, or Oracle OCI Images). With the advent of containers infrastructure testing becomes relevant here as well—analogous to the machine image case.
Not unlike many others we have a lot of infrastructure at MySQL, and we increasingly use containers instead of real (virtual) machines. Also, more and more of our core infrastructure is run on Oracle’s Cloud Infrastructure (OCI). This calls for automation on multiple levels, and infrastructure testing can be leveraged to verify the state of our servers (or virtual machines or containers). Infrastructure testing is also used to verify the state of some of our release artifacts.
In this blog post we will focus on how to use automated infrastructure testing to verify the MySQL Server Docker images. We will compare three different frameworks that can be used for container testing and show example code for all of them.
Automated Infrastructure Testing
The term infrastructure testing relates to testing the state of one’s infrastructure. It is used to answer questions like: Is my Apache server listening on port 80 as it should? Did I configure the DNS servers correctly and are those settings correctly propagated to the resolv.conf file? Are all the binaries in place that I was trying to install on a machine image?
This kind of testing could be part of the bash script so often used for configuration tasks or it could be a manual step at the end of a (manual) instance creation. Automated infrastructure testing goes one step further. It accepts that there is simply too much infrastructure to check and script correctly. It assumes the dynamic nature of modern cloud environments is simply too much to handle in a manual way.
Very often infrastructure testing tools work in combination with provisioning tools like Ansible, Puppet, or Chef. The provisioning tool installs software on a machine — the testing framework makes sure that it actually works. Then everything can be specified in code and done automatically using the right tools.
Our focus here is testing of docker images and for us this is rather low-level. As all our docker images mainly consist of thin layers of already released and tested yum packages on top of a very solid OS layer, we mainly want to verify that the packages are installed in the correct versions and that the desired binaries are there and functional. Network glitches and partial installations can happen at build time, and this is exactly the kind of case we want to catch with automatic testing.
Two concepts are important in the context of evaluating tools:
- Configuration language, i.e. what one wants to test for (packages available, files in place, etc.)
- Test execution, i.e. how to run the testing (local/ssh/container)
We will focus on these two aspects when going through the individual tools below. The most common tools in that area seem to be:
- InSpec/Serverspec
- Goss
- Container Structure Test
In the following we will give a short overview of each of them.
InSpec
InSpec is based on the ruby RSpec testing framework and is built on the experiences with Serverspec (which also builds on RSpec and is widely adopted). It is part of and integrated into the Chef ecosystem for provisioning and testing infrastructure. Its configuration is stored in a ruby file.
- Configuration language is extensive via resources
- Test execution realised via targets, i.e. local/ssh/docker
Goss
Goss markets itself as a quick and easy alternative to Serverspec and is a server testing/validation framework written in Go. Its configuration is stored in a yaml file and this configuration file can conveniently be generated from current system state.
- Configuration language quite extensive
- Test execution supported locally and in docker containers (via the dgoss wrapper script)
Container Structure Test
Container Structure Test is a framework to validate the structure of container images. Like Goss it is written in go and uses a yaml configuration file. This project got released earlier this year and has gained some traction since. The scope is narrower than for the alternatives (only containers) but it offers enough to test images comprehensively.
- Configuration language quite basic
- Test execution limited to local containers
Example: MySQL Server Images
In the following we show how to install the required tools, explain the individual test configs, and run the tests locally. We run the tests against the newest MySQL Server container (latest or 8.0 tags). In the interest of brevity we skip the building step, i.e. we download the container from the public registry and test it locally. In our actual build pipelines we build containers first, run tests, and only push to the public registry on success. One can pull the latest version of the mysql-server image by typing:
docker pull mysql/mysql-server
By and large we want to test two things:
- container exists on host machine with correct metadata
- container contains all our packages and binaries
Prerequisites
In addition to a working Docker environment, running the example requires InSpec, Goss, and Container Structure Test to be installed locally.
InSpec instructions can be found here: https://downloads.chef.io/inspec , and on Linux platforms the Goss and Container Structure Test binaries can be installed by running:
curl -L https://github.com/aelsabbahy/goss/releases/download/v0.3.6/goss-linux-amd64 -o goss && chmod +x goss
curl -L https://raw.githubusercontent.com/aelsabbahy/goss/master/extras/dgoss/dgoss -o dgoss && chmod +x dgoss
curl -L https://storage.googleapis.com/container-structure-test/latest/container-structure-test-linux-amd64 -o container-structure-test && chmod +x container-structure-test
Once all binaries (inspec, goss/dgoss, container-structure-test) are in place and on your path the tests can be run via the shell scripts.
Test Configurations
In order to show the differences in both configuration and test execution we provide
example files to test MySQL Server Docker images for all three frameworks:
https://github.com/neumayer/mysql-server-image-tests
It can be cloned by typing:
git clone https://github.com/neumayer/mysql-server-image-tests.git
The repo contains configuration files:
- mysql-server-inspec.rb
- goss.yaml
- mysql-server-container-structure-test.yml
Let’s look at these files in turn, starting with the InSpec config:
control 'container' do
impact 0.5
describe docker_container('mysql-server') do
it { should exist }
it { should be_running }
its('repo') { should eq 'mysql/mysql-server' }
its('ports') { should eq '3306/tcp, 33060/tcp' }
its('command') { should match '/entrypoint.sh mysqld' }
end
end
control 'server-package' do
impact 0.5
describe package('mysql-community-server-minimal') do
it { should be_installed }
its ('version') { should match '8.0.12.*' }
end
end
control 'shell-package' do
impact 0.5
describe package('mysql-shell') do
it { should be_installed }
its ('version') { should match '8.0.12.*' }
end
end
Without going into too much detail on the fine-grained control mechanisms (controls and versions) InSpec tests are organised via profiles and controls where a control is the smaller unit, roughly being a set of tests around a given topic. The first control is called container and is run against the host machine, talking to the Docker daemon running on localhost to verify that a container is running. The other two controls check for packages inside the container. This distinction is important, specifically because the last two controls can be run against localhost, an ssh host, or a Docker container. In our case we use it for the latter. This allows very good re-usability and flexibility. While we only use the Docker and package resources in this example, controls can use any of the existing InSpec resources.
This leads to the following workflow:
- Start container
- Run “container” control against localhost
- Run remaining controls against the container
In our script file this looks like this:
docker run -d --name mysql-server mysql/mysql-server
inspec exec mysql-server-inspec.rb --controls container
inspec exec mysql-server-inspec.rb -t docker://mysql-server --controls server-package
InSpec will output the following on a successful run:
Profile: tests from mysql-server-inspec.rb (tests from mysql-server-inspec.rb)
Version: (not specified)
Target: local://
✔ container: Docker Container mysql-server
✔ Docker Container mysql-server should exist
✔ Docker Container mysql-server should be running
✔ Docker Container mysql-server repo should eq "mysql/mysql-server"
✔ Docker Container mysql-server ports should eq "3306/tcp, 33060/tcp"
✔ Docker Container mysql-server command should match "/entrypoint.sh mysqld"
Profile Summary: 1 successful control, 0 control failures, 0 controls skipped
Test Summary: 5 successful, 0 failures, 0 skipped
Profile: tests from mysql-server-inspec.rb (tests from mysql-server-inspec.rb)
Version: (not specified)
Target: docker://d06da2588b80a4ee9b839b55c2f719ab9e860904eeb831b71488704f50f8b994
✔ server-package: System Package mysql-community-server-minimal
✔ System Package mysql-community-server-minimal should be installed
✔ System Package mysql-community-server-minimal version should match "8.0.12.*"
Profile Summary: 1 successful control, 0 control failures, 0 controls skipped
Test Summary: 2 successful, 0 failures, 0 skipped
For Goss, the config file looks like this:
file:
/usr/sbin/mysqld:
exists: true
contains: []
package:
mysql-community-server-minimal:
installed: true
mysql-shell:
installed: true
port:
tcp6:3306:
listening: true
ip: []
tcp6:33060:
listening: true
ip: []
user:
mysql:
exists: true
process:
mysqld:
running: true
In addition to the mysqld file being in place we check for the packages to be installed, correct ports being exposed, and the process to be running. One important difference here is that Goss starts the container for us:
GOSS_SLEEP=10 dgoss run -p 3306:3306 mysql/mysql-server
The GOSS_SLEEP is set so our server has time to finish initialisation, the remainder of the parameters is passed on to docker run. The output looks like this:
INFO: Starting docker container
INFO: Container ID: 75bc8869
INFO: Sleeping for 10
INFO: Running Tests
File: /usr/sbin/mysqld: exists: matches expectation: [true]
User: mysql: exists: matches expectation: [true]
Process: mysqld: running: matches expectation: [true]
Port: tcp6:33060: listening: matches expectation: [true]
Port: tcp6:33060: ip: matches expectation: [[]]
Port: tcp6:3306: listening: matches expectation: [true]
Port: tcp6:3306: ip: matches expectation: [[]]
Package: mysql-shell: installed: matches expectation: [true]
Package: mysql-community-server-minimal: installed: matches expectation: [true]
Total Duration: 0.038s
Count: 9, Failed: 0, Skipped: 0
INFO: Deleting container
The Container Structure Test configuration is specified in the following yaml snippet:
schemaVersion: "2.0.0"
metadataTest:
exposedPorts: [ "3306", "33060" ]
entrypoint: [ "/entrypoint.sh" ]
cmd: [ "mysqld" ]
volumes: [ "/var/lib/mysql" ]
commandTests:
- name: "mysqlsh"
command: "mysqld"
args:
- "--version"
expectedOutput:
- "8.0.12"
- name: "mysqlsh"
command: "mysqlsh"
args:
- "--version"
expectedOutput:
- "8.0.12"
fileExistenceTests:
- name: "mysqld"
path: "/usr/sbin/mysqld"
Also here we check for the correct ports to be exposed and make sure the files are in place but directly run the binaries to verify that they are in place instead of using internal wrappers like with the other tools.
container-structure-test --image mysql/mysql-server test --config mysql-server-container-structure-test.yml
Similar to Goss the invocation is easy, all it needs is the image name and the config file.
==================================================================
====== Test file: mysql-server-container-structure-test.yml ======
==================================================================
INFO: stdout: /usr/sbin/mysqld Ver 8.0.12 for Linux on x86_64 (MySQL Community Server - GPL)
=== RUN: Command Test: mysqlsh
--- PASS
stdout: /usr/sbin/mysqld Ver 8.0.12 for Linux on x86_64 (MySQL Community Server - GPL)
INFO: stdout: mysqlsh Ver 8.0.12 for Linux on x86_64 - for MySQL 8.0.12 (MySQL Community Server (GPL))
=== RUN: Command Test: mysqlsh
--- PASS
stdout: mysqlsh Ver 8.0.12 for Linux on x86_64 - for MySQL 8.0.12 (MySQL Community Server (GPL))
INFO: File Existence Test: mysqld
=== RUN: File Existence Test: mysqld
--- PASS
=== RUN: Metadata Test
--- PASS
===================================================================
============================= RESULTS =============================
===================================================================
Passes: 4
Failures: 0
Total tests: 4
PASS
Container structure test feels slickest in that respect — it provides everything to make sure containers are in good shape, run fast and are easy to invoke. The tradeoff is that it can only be used for containers. In many cases that’s perfectly fine and it will make sure the containers behave the way they should.
The tests mentioned above can be run via scripts from the root of the repo:
- ./inspec.sh
- ./goss.sh
- ./container-structure-test.sh
Container Testing at MySQL
We have started testing all our docker images with InSpec. From the next MySQL release (8.0.13) basic InSpec testing will be part of the automated release process for the MySQL Server, MySQL Cluster, and MySQL Router docker images. The decision do so was mostly based on:
- it having the largest scope (ssh/local/docker) which opens for further internal use,
- its extensive set of resources,
- its dependency mechanism,
- the fact that it is widely adopted because of both its connection to the Chef ecosystem and its similarity to Serverspec
There exist disadvantages too:
- due to the ruby calls it feels slower than the Go alternatives (we don’t consider this to be critical though)
- InSpec is more generic — for good and for bad: the tradeoff is slightly more scripting to run the tests
We run InSpec as part of our automated release pipeline, i.e. we produce no artifacts if any of the tests fail. Our QA process consists of many other step such as the aforementioned separate testing of the rpm packages that are used in the MySQL docker images.
Links
https://cloud.oracle.com/cloud-infrastructure
https:://www.chef.io
https://www.inspec.io/docs/reference/resources/
https://www.inspec.io/docs/reference/inspec_and_friends/
https://goss.rocks/
https://github.com/GoogleContainerTools/container-structure-test
Official MySQL docker images on docker hub