So this is very helpful @rubys, thank you! Now I can poke around the image on a fly machine. My tests gave very interesting results.
> 15:32 fly-nix-test git:(main) podman login -u x -p "$FLY_API_TOKEN" -v registry.fly.io
Used: /run/user/1000/containers/auth.json
Login Succeeded!
> 15:32 fly-nix-test git:(main) podman pull registry.fly.io/fly-nix-test:0.0.0
Trying to pull registry.fly.io/fly-nix-test:0.0.0...
Getting image source signatures
Copying blob e07f43e582c2 skipped: already exists
Copying blob 87197915c18d skipped: already exists
Copying blob 3375c174bfc7 skipped: already exists
Copying blob 10e45524ca1c skipped: already exists
Copying blob cb52c0b7d468 skipped: already exists
Copying blob 1280ab61288a skipped: already exists
Copying blob aa1508391eab skipped: already exists
Copying blob beb7e054375b skipped: already exists
Copying config 6c27eae71e done |
Writing manifest to image destination
6c27eae71e27e390aba3befa760076dd2bea7d8c68fc58a806d4062b4a257218
> 15:32 fly-nix-test git:(main) podman container run --tty --interactive registry.fly.io/fly-nix-test:0.0.0
Hello, World!
^C%
> 15:33 fly-nix-test git:(main) flyctl console --image registry.fly.io/fly-nix-test:0.0.0
Searching for image 'registry.fly.io/fly-nix-test:0.0.0' remotely...
image found: img_rj5yv1jrnmx7vdwq
Image: registry.fly.io/fly-nix-test:0.0.0
Image size: 15 MB
Created an ephemeral machine 185e712b3ed968 to run the console.
WARN The running flyctl agent (v0.3.49) is older than the current flyctl (v0.3.56).
WARN The out-of-date agent will be shut down along with existing wireguard connections. The new agent will start automatically as needed.
Connecting to fdaa:2:439f:a7b:69:4e9d:4886:2... complete
/ # ls
bin dev etc home lib lib64 nix proc root run sys tmp usr var
/ # ls nix/store
2d5spnl8j5r4n1s4bj1zmra7mwx0f1n8-xgcc-13.3.0-libgcc 85m2k33jlw6s5qk654ikb46ygym5fa36-fly-nix-test-server-0.0.0 qwjjm4j652ck9izaid7bz63s4hd5bnha-libidn2-2.3.7
6pqgj71r0850b0cd95yxx0d52zax016i-libunistring-1.2 p6k7xp1lsfmbdd731mlglrdj2d66mr82-bash-5.2p37 wn7v2vhyyyi6clcyn0s9ixvl7d4d87ic-glibc-2.40-36
/ # exit
Waiting for ephemeral machine 185e712b3ed968 to be destroyed ... done.
> 15:34 fly-nix-test git:(main) flyctl console --image registry.fly.io/fly-nix-test:0.0.0 -C bash
Searching for image 'registry.fly.io/fly-nix-test:0.0.0' remotely...
image found: img_rj5yv1jrnmx7vdwq
Image: registry.fly.io/fly-nix-test:0.0.0
Image size: 15 MB
Created an ephemeral machine 080e74ea3d5228 to run the console.
Connecting to fdaa:2:439f:a7b:161:312e:4012:2... complete
exec: "bash": executable file not found in $PATH
Waiting for ephemeral machine 080e74ea3d5228 to be destroyed ... done.
Error: ssh shell: wait: remote command exited without exit status or exit signal
> 15:34 fly-nix-test git:(main) flyctl console --image registry.fly.io/fly-nix-test:0.0.0 -C /nix/store/p6k7xp1lsfmbdd731mlglrdj2d66mr82-bash-5.2p37/bin/bash
Searching for image 'registry.fly.io/fly-nix-test:0.0.0' remotely...
image found: img_rj5yv1jrnmx7vdwq
Image: registry.fly.io/fly-nix-test:0.0.0
Image size: 15 MB
Created an ephemeral machine d8dd957f7d5448 to run the console.
Connecting to fdaa:2:439f:a7b:69:d745:145:2... complete
bash-5.2# sleep 1
bash: sleep: command not found
bash-5.2# echo $PATH
/no-such-path
bash-5.2# exit
exit
Waiting for ephemeral machine d8dd957f7d5448 to be destroyed ... done.
> 15:38 fly-nix-test git:(main) podman container run --tty --interactive --entrypoint /nix/store/p6k7xp1lsfmbdd731mlglrdj2d66mr82-bash-5.2p37/bin/bash registry.fly.io/fly-nix-test:0.0.0
bash-5.2# sleep 1
bash-5.2# exit
exit
> 15:39 fly-nix-test git:(main) podman container run --tty --interactive --entrypoint bash registry.fly.io/fly-nix-test:0.0.0
Error: crun: executable file `bash` not found in $PATH: No such file or directory: OCI runtime attempted to invoke a command that was not found
> 15:42 fly-nix-test git:(main) podman container run --tty --interactive --entrypoint /nix/store/p6k7xp1lsfmbdd731mlglrdj2d66mr82-bash-5.2p37/bin/bash registry.fly.io/fly-nix-test:0.0.0
bash-5.2# echo $PATH
/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
bash-5.2# exit
exit
So the interesting things to note here:
- @ 15:33 & 15:34 I was able to make a Fly machine load into Bash where I could test things and poke around the image. Most notably,
$PATHis defined as/no-such-pathandsleepis not found, just like the server behaves when deployed by my application. - @ 15:38 & 15:42 I was able to use the same image with the same bash exe as the Fly machine, but in here
$PATHandsleepare defined in the expected way.
Something must be wrong with the way Fly interprets my image, possibly to do with containerd? I will see what I can figure out tomorrow. Thank you both for your help today.