If you haven’t been paying attention, we’ve got an early Christmas present: containers are coming to Fly Machines. The tl;dr is that you can run multiple images/containers all in one Fly Machine. Your app and log shipper can now live happily ever after, right next to each other.
The push for containers was inspired by Fly Kubernetes (a.k.a FKS), our managed Kubernetes service. We always felt our Kubernetes was incomplete without sidecars, after all Kubernetes and sidecars are two peas in a pod. Additionally, we were left without init containers which are just as popular. After getting the pieces working throughout our platform, with our shiny new init, aptly named Pilot, we got to work supporting it in FKS.
Why it matters?
Another present we’ve got in the works is an upcoming managed databases product. We’re building it on top of Percona, which utilises Kubernetes under the hood. When we started working on this, we hacked around the shortcomings of Fly Kubernetes in order to get it to run. With support with sidecars and init containers, we’re steps closer to running Percona as intended. In doing so, we’re closer to a stock standard deployment, ensuring a more stable and reliable service.
How did we do it?
99% of the heavy lifting is done by Pilot. FKS is a regular user of the Fly Machines API. Each container is specified in the containers field in the configuration. As an example
{
"config": {
"containers": [
{
"name": "go-httpbin",
"image": "mccutchen/go-httpbin:latest",
"env": {"PORT": "10240"}
},
{
"name": "nginx",
"image": "nginx:latest"
}
]
}
}
And that’s all there is to it. Flyd (our in-house orchestrator) does all the magic of pulling images, making them accessible to Pilot which is responsible for running the respective containers and managing their lifecycle.
Init containers are interesting. They are a Kubernetes concept used to perform initialisation (e.g writing configuration files to a specific location) before the main application containers are run. Init containers run sequentially, having to exit successfully before the next one begins.
This sequence can be modeled as a dependency graph, where each container depends on the previous container running to completion. Internally, Pilot keeps track of dependencies using a directed graph (using the wonderful Rust library, petgraph). This is particularly neat as we can support arbitrary topologies of dependencies between containers. Init containers are easily represented in our graph, with each node pointing to the next with a successful exit as a condition. The main container(s) all depend on the successful exit of the final init container.
When run, Pilot walks through all the dependencies, adding the respective nodes and the edges between them. It starts containers with no dependencies first. As they pass through their lifecycle, we record the conditions they’ve fulfilled, regardless if they are depended on or not. For example, when a container starts, we record that condition and check if there a new containers eligible to start which required it to start. Similarly when healthchecks pass (if configured) and when a container exits. Through this mechanism, we traverse the entire graph, running each container as necessary.
With the Machines API, this is easy enough to configure. There’s a depends_on field in the containers definition. Other than the container we depend on, it allows us to define the condition that container must fulfill. Examples of this would be: the container started, it exited successfully or its health checks are passing. For example, we can make our nginx container depend on go-httpbin being started.
{
"config": {
"containers": [
{
"name": "go-httpbin",
"image": "mccutchen/go-httpbin:latest",
"env": {"PORT": "10240"}
},
{
"name": "nginx",
"image": "nginx:latest",
"depends_on": [
{
"name": "go-httpbin",
"condition": "started"
}
]
}
]
}
}