Summary
Hello!
I have a Docker image that works on my local machine using Podman. However, when it’s on a Fly machine, the server reports command not found for any command my server script tries to call.
Attempts to Fix
My script uploads the Docker image to registry.fly.io using Skopeo, and then deploys a Fly app from that image.
"$DOCKER_IMAGE_STREAM" | gzip --fast | skopeo --insecure-policy copy --dest-creds="x:$FLY_API_TOKEN" "docker-archive:/dev/stdin" "docker://registry.fly.io/$FLY_APP_NAME:$DOCKER_IMAGE_TAG"
flyctl deploy -c "$FLY_CONFIG" -i "registry.fly.io/$FLY_APP_NAME:$DOCKER_IMAGE_TAG"
Fly seems to successfully load the image, however the Fly machine won’t stay on.
I see some command not found errors from inside the server script in the flyctl logs.
2025-01-08T23:26:14Z app[185241dc107068] den [info]2025-01-08T23:26:14.358821070 [01JH43PM0KBA5TPPEJVYP1RY3T:main] Running Firecracker v1.7.0
2025-01-08T23:26:14Z app[185241dc107068] den [info] INFO Starting init (commit: 1df1d0a0)...
2025-01-08T23:26:15Z app[185241dc107068] den [info] INFO Preparing to run: `/nix/store/9jnifcq70bqmmsa99iw4mhpc2vla6m30-fly-vpn-server-0.0.0/bin/fly-vpn-server-0.0.0` as root
2025-01-08T23:26:15Z app[185241dc107068] den [info] INFO [fly api proxy] listening at /.fly/api
2025-01-08T23:26:15Z app[185241dc107068] den [info]/nix/store/9jnifcq70bqmmsa99iw4mhpc2vla6m30-fly-vpn-server-0.0.0/bin/fly-vpn-server-0.0.0: line 30: id: command not found
2025-01-08T23:26:15Z app[185241dc107068] den [info]/nix/store/9jnifcq70bqmmsa99iw4mhpc2vla6m30-fly-vpn-server-0.0.0/bin/fly-vpn-server-0.0.0: line 146: sudo: command not found
2025-01-08T23:26:15Z runner[185241dc107068] den [info]Machine started in 809ms
2025-01-08T23:26:15Z app[185241dc107068] den [info]2025/01/08 23:26:15 INFO SSH listening listen_address=[fdaa:2:439f:a7b:9:5554:5b3d:2]:22 dns_server=[fdaa::3]:53
2025-01-08T23:26:16Z app[185241dc107068] den [info] INFO Main child exited normally with code: 127
2025-01-08T23:26:16Z app[185241dc107068] den [info] INFO Starting clean up.
2025-01-08T23:26:16Z app[185241dc107068] den [info] WARN could not unmount /rootfs: EINVAL: Invalid argument
2025-01-08T23:26:16Z app[185241dc107068] den [info][ 1.661012] reboot: Restarting system
2025-01-08T23:26:16Z runner[185241dc107068] den [info]machine has reached its max restart count of 10
So I try to use Podman to upload the image instead, thinking there could be a bug with Skopeo.
podman login -u x -p "$FLY_API_TOKEN" -v registry.fly.io
"$DOCKER_IMAGE_STREAM" | podman load
podman push --format v2s2 "localhost/$DOCKER_IMAGE_NAME:$DOCKER_IMAGE_TAG" "docker://registry.fly.io/$FLY_APP_NAME:$DOCKER_IMAGE_TAG"
flyctl deploy -c "$FLY_CONFIG" -i "registry.fly.io/$FLY_APP_NAME:$DOCKER_IMAGE_TAG"
The behavior persists.
So I try to download and run the image from registry.fly.io to see if it is copied correctly.
Podman successfully loads the image from registry.fly.io, and the image runs flawlessly.
> podman login -u x -p "$FLY_API_TOKEN" -v registry.fly.io
Used: /run/user/1000/containers/auth.json
Login Succeeded!
> podman pull registry.fly.io/flyvpn:0.0.0
Trying to pull registry.fly.io/flyvpn:0.0.0...
Getting image source signatures
Copying blob 507ec11f5a1f skipped: already exists
Copying blob 3375c174bfc7 skipped: already exists
Copying blob 10e45524ca1c skipped: already exists
Copying blob 10e45524ca1c skipped: already exists
Copying blob 10e45524ca1c skipped: already exists
Copying blob 10e45524ca1c skipped: already exists
Copying blob 10e45524ca1c skipped: already exists
Copying blob 3375c174bfc7 skipped: already exists
Copying blob 507ec11f5a1f skipped: already exists
Copying blob cb52c0b7d468 skipped: already exists
Copying blob 87197915c18d skipped: already exists
Copying blob e07f43e582c2 skipped: already exists
Copying blob 95b18c5e2fb6 skipped: already exists
Copying blob 52a004ba8547 skipped: already exists
Copying blob 2666773469b4 skipped: already exists
Copying blob 1196b72764b9 skipped: already exists
Copying blob b145809b4cc3 done |
Copying blob faa67288214d done |
Copying blob beb7e054375b skipped: already exists
Copying config 954759158d done |
Writing manifest to image destination
954759158d565e2ffde2b38ff1989b893d86197f09086ec9c7b1e3ddec38d19c
> podman container run --tty --interactive registry.fly.io/flyvpn:0.0.0
{ ... SERVER OUTPUT ... }
So the copy of the image is correctly loaded into registry.fly.io.
Next I thought that perhaps the issue is specifically with the commands sudo or id.
So I comment out the part of my server script that uses these commands, then re-deploy the image.
2025-01-08T23:59:29Z app[185241dc107068] den [info]2025-01-08T23:59:29.372307289 [01JH45J70AFX4KZJ597EF02DF1:main] Running Firecracker v1.7.0
2025-01-08T23:59:29Z app[185241dc107068] den [info] INFO Starting init (commit: 1df1d0a0)...
2025-01-08T23:59:30Z app[185241dc107068] den [info] INFO Preparing to run: `/nix/store/2cpwn105kaf4zxfl3vhm24m41lk6bavx-fly-vpn-server-0.0.0/bin/fly-vpn-server-0.0.0` as root
2025-01-08T23:59:30Z app[185241dc107068] den [info] INFO [fly api proxy] listening at /.fly/api
2025-01-08T23:59:30Z app[185241dc107068] den [info]/nix/store/2cpwn105kaf4zxfl3vhm24m41lk6bavx-fly-vpn-server-0.0.0/bin/fly-vpn-server-0.0.0: line 37: mkdir: command not found
2025-01-08T23:59:30Z app[185241dc107068] den [info]/nix/store/d7p2bql11nbs4jbmslzl3c52knlj8bwm-softether-4.41-9782-beta/bin/vpnserver: line 2: /var/lib/softether/vpnserver/vpnserver: No such file or directory
2025-01-08T23:59:30Z runner[185241dc107068] den [info]Machine started in 775ms
2025-01-08T23:59:30Z app[185241dc107068] den [info]2025/01/08 23:59:30 INFO SSH listening listen_address=[fdaa:2:439f:a7b:9:5554:5b3d:2]:22 dns_server=[fdaa::3]:53
2025-01-08T23:59:31Z app[185241dc107068] den [info] INFO Main child exited normally with code: 127
2025-01-08T23:59:31Z app[185241dc107068] den [info] INFO Starting clean up.
2025-01-08T23:59:31Z app[185241dc107068] den [info] WARN could not unmount /rootfs: EINVAL: Invalid argument
2025-01-08T23:59:31Z app[185241dc107068] den [info][ 1.647670] reboot: Restarting system
2025-01-08T23:59:31Z runner[185241dc107068] den [info]machine has reached its max restart count of 10
With sudo and id removed, the server also can’t find mkdir or the direct path to a bin directory with my server in it.
So not only are commands provided by PATH not resolved, even direct paths to executable files in the image are not able to be resolved.
Again, the image works perfectly fine when loaded by Podman.
I’m really out of ideas for what’s causing this, so I’m out of things to test… I think I need help from Fly staff, because I don’t see any way I can identify the cause of this problem from my end.
Additional Context
My image is built using the Nix function pkgs.dockerTools.streamLayeredImage provided by nixpkgs instead of a Dockerfile. In fact, the entire project is made using Nix. Nix stacks my server on top of Busybox:
{
pkgs,
name,
version,
server ? pkgs.callPackage ./server.nix { inherit name version; }
}: let
_name = "${name}-docker-image";
tag = version;
# update base image using variables from:
# xdg-open https://hub.docker.com/_/busybox/tags
# nix-shell -p nix-prefetch-docker
# nix-prefetch-docker --quiet --image-name busybox --image-tag stable --image-digest sha256:_
baseImage = pkgs.dockerTools.pullImage {
imageName = "busybox";
imageDigest = "sha256:7c3c3cea5d4d6133d6a694d23382f6a7b32652f23855abdba3eb039ca5995447";
sha256 = "0k9ypllg4lmwd1a370z8n3awf5fpvlwwq355hmrfjwlmvqarjmjr";
finalImageName = "busybox";
finalImageTag = "stable";
os = "linux";
arch = "amd64";
};
in {
name = _name;
inherit version tag;
stream = pkgs.dockerTools.streamLayeredImage {
name = _name;
inherit tag;
fromImage = baseImage;
contents = [
server
];
config = {
Entrypoint = [ "${pkgs.lib.getExe server}" ];
Cmd = [];
ExposedPorts = {
"5555/tcp" = {};
"992/tcp" = {};
"443/tcp" = {};
};
};
};
}
I know this should be supported by Fly because I’ve used this tool before, and I’ve seen evidence on these forums of others doing the same.
You can recreate my issue by following the README at this public GitHub repo to create your own copy of my app: GitHub - mboyea/fly-vpn: An SSTP VPN hosted by Fly.io
Or, just inspect parts of the code to try and identify why my image isn’t working with Fly.
