Scale-to-Zero/Wake-on-Request Docker on Fly

Questions / Help
1

Hello! I’m trying to use Machines to implement a version of the Fly Docker daemon that scales to zero and wakes on request.

I have been using a Docker daemon Fly app for my build server, but it is my most costly app when left running nonstop. I can scale to zero and manually scale up when I need to do a build, but I am trying to automate the process now. I figured the scale-to-zero feature of Fly machines seemed like a good fit.

I was able to successfully create a machine-ified version of the Docker daemon, and when it’s running, it responds to requests via the IPv6 address that comes up when you run flyctl machines list. This is not an IPv6 address obtained with flyctl ips allocate-v6 --private.

When I stop the machine and attempt to start a build, I would expect the Docker traffic to wake it up, but nothing happens. The build fails because it can’t connect to the Docker daemon, and the machine stays stopped. I tried allocating a Flycast IPv6 address with allocate-v6 --private, and setting DOCKER_HOST=tcp://[allocated-ip]:2375, but this still fails to wake up the Machine, and the build server just waits indefinitely. Furthermore, if I manually start the machine and try to kick off a build, the Docker daemon no longer functions correctly, with the following error message in the build server logs:

 2022-11-18T22:08:44.440 app[0e8d88b4] otp [info] time="2022-11-18T22:08:44Z" level=error msg="cannot ping the docker daemon" error="error during connect: Get \"http://[allocated-ip]:2375/_ping\": read tcp [machine-ip]:34010->[allocated-ip]:2375: read: connection reset by peer"

Is there anything additional I need to configure to get wake-on-request functioning? Does it not work for raw tcp traffic like Docker is sending?

Thanks in advance for any help!

1 Like
2

@mat Have you tried specifying a service? I’ve found that wake-on-request won’t work without one. In case you’re using a fly.toml, this should do the trick:

[[services]]
  protocol = "tcp"
  internal_port = 2375
  [[services.ports]]
    port = 2375
3 Likes
3

@lukas-w I tried adding this to the config, as well as specifying it directly on the command line with --port for flyctl machine run, but it still doesn’t seem to wake up when the machine is stopped and my build server tries to kick off a build (edited to add bolded part). Any thoughts on how I could debug this?

4

fly m run is to create new machines. Try: fly m list -a <app-name>, note the ids; then fly m start <id> -a <app-name>, and see what happens?

nb, in my experience, logging in with fly ssh console -s -a <app-name> also wakes the machine up.

1 Like
5

The machine is fine once I manually start with eg flyctl m start, I was just using run to create a machine with the port mentioned above. Even with the port, wake-on-request doesn’t seem to work. ssh console, and anything that requires a manual step, is a non-starter here, because my goal is to start the machine when my build server tries to kick off a build. Since the build server runs its commands on the Docker host, it has to be awake before I can run something like flyctl m start or flyctl ssh console

6

Gotcha.

If what lukas-w suggested doesn’t work (and it looks like it should work… well…), see if you can at least connect to the docker-builder after a fly m start?

Alternatively, you can opt to ping Kurt or Jerome, who seem to be the main code contributors.

7

The Docker daemon definitely does work when manually started, it’s exclusively the wake-on-request functionality that doesn’t seem to work.

@kurt Would you have any ideas for debugging here? Is there any e.g. metric or log I can watch to judge if wake-on-request should be working?

2 Likes
8

I’ve also been trying the exact same thing - I’m hoping to have a build server based on the Fly docker daemon that I can wake-on-request.

Did you make any modifications to that repo before launching the machine?

9

You can see the exact fly machine I’m running here: fly-apps/docker at main - fly-apps - git.mat.services: Gitea for us

Initially I did a git subtree of this fork in my own repo. The main difference is that I used flyctl to redeploy the docker image as a machine.

2 Likes
10

Thank you! I’m trying the forked repo out and I’m experiencing the same issue - it works when I start the machine myself, but wake-on-request times out.

11

This seems to have fixed the issue for me - the machine was waking-on-request for a month or two but then stopped - adding a services stanza seems to have fixed it

1 Like