fly deploy failing with "No such file or directory" - docker image is good

I’ve been developing a Phoenix application on fly, using essentially the stock Dockerfile and fly.toml, for a number of a weeks, but in the last day or two the deployments have become rather flaky.

During the deploy the cli reports that:

[1/1] Waiting for REDACTED [app] to have state: started

But after 2 minutes it will timeout.
Looking at the monitoring page gives slightly more info:

lhr [info] ERROR Error: failed to spawn command: /app/bin/server: No such file or directory (os error 2)

lhr [info] does `/app/bin/server` exist and is it executable? 

This looks like my problem but I know that the executable is most certainly present in the docker image. If I build and run it locally and even if I download the docker image from the fly registry and run it locally it works fine.

The above error message sometimes just occurs once before the deploy is cancelled, but the most recent attempt seems to be attempting to launch the app repeatedly without success.

I can’t figure out anything that I’m doing wrong so I feel like this is a Fly issue.

Other info:
flyctl version: Happening on both v0.1.51 and v0.1.53
Apps v2

EDIT: flyctl also is sitting at 100% cpu usage and ignores SIGTERM when waiting for the deploy to happen, which doesn’t feel like correct behaviour.

I also have this issue, and this was all i could find on it

I think in the end this was caused by the fly VM this was running on getting into some wonky state that it couldn’t recover from and which persisted to new VMs on each deploy somehow. I think shutting down and deleting all machines (and honestly I might have gone as far as to delete the project) eventually made the deploy work.