Machine stuck in replacing state

jswanner · January 13, 2024, 12:59am

Same boat here

flyer1 · January 13, 2024, 1:27am

My production is down, bad timing for deploying.

Any region thats working? (tried ewr and sjc so far)

This is the log

could not find a good candidate within 90 attempts at load balancing. last error: no known healthy instances found for route tcp/443. (hint: is your app shut down? is there an ongoing deployment with a volume or are you using the ‘immediate’ strategy? have your app’s instances all reached their hard limit?)

hypeed · January 13, 2024, 1:40am

The number of production-level issues that have been happening with Fly recently is very concerning. Is anyone else considering switching to a more stable provider?

jswanner · January 13, 2024, 1:41am

They have now acknowledged the issue on their status page: Fly.io Status - Public API Errors and Timeouts

atridad · January 13, 2024, 2:27am

What was the issue? I’m curious. Definitely a bit frustrating to have so many issues lately…

jswanner · January 13, 2024, 2:37am

Issue was resolved, and I was able to deploy

vplagov · January 28, 2024, 9:24pm

Although the issue is resolved, I’m getting exactly the same error in the arn region with the following errors in GitHub Actions logs:

✖ [1/2] Machine 9185770ef79308 [app] update failed: failed to update VM 9185770ef79308: request returned non-2xx status, 504
Error: failed to update VM 9185770ef79308: request returned non-2xx status, 504 (Request ID: 01HN8Z31SPBY9VMZAZ16RRP0ED-ord)

Is there any solution on how to fix it? The machine is stuck in the Replacing status for ~5+ hours.

arijan · February 14, 2024, 8:30am

Same is happening to me, doesn’t seem like there’s anything that I can do

Edit: I just cloned it and --force destroyed the previous one, it was up again in a minute.

flwi · August 15, 2024, 6:26pm

**Edit: ** never mind. The app works now. Problems seems to be solved.

Hi!

I found this thread and I’m encountering the same error.

First I had some issues pushing the new image to the registry (it . Once that succeeded the machine is stuck in “replacing” state.

Tried redeploying the newest revision, but it’s still stuck.
Got this error after running fly deploy -i registry.fly.io/flwi-spacetraders-rust-leaderboard:deployment-01J5BGR4PGHCEDP455K1B8W81B

-------
 ==> Verifying app config
Validating .../fly.toml
✓ Configuration is valid
--> Verified app config
WARN LEADERBOARD_DATABASE_URL may be a potentially sensitive environment variable. Consider setting it as a secret, and removing it from the [env] section: https://fly.io/docs/apps/secrets/

==> Building image
Searching for image 'registry.fly.io/flwi-spacetraders-rust-leaderboard:deployment-01J5BGR4PGHCEDP455K1B8W81B' remotely...
image found: img_19gm46q996qkvx0j

Watch your deployment at https://fly.io/apps/flwi-spacetraders-rust-leaderboard/monitoring

-------
Updating existing machines in 'flwi-spacetraders-rust-leaderboard' with rolling strategy

-------
 ✖ Machine 56830157a169d8 [app] update failed: failed to update VM 56830157a169d8: request returned non-2xx status, 504
-------
Error: failed to update VM 56830157a169d8: request returned non-2xx status, 504 (Request ID: 01J5BM1V8YNWZ17KJPAFDNS5AC-fra) (Trace ID: 899be6ba2d78572d727be2736cc3b0d9)

Weird thing is that the app is still running in the background (it ticks every 5min and produces some log messages which I can see in the live logs.)

It’d be great if someone from the admin team could help me out.

doodoom · February 21, 2025, 5:25am

I always get stuck. Everytime I need to deploy, I delete it and relaunch it. Is this only happening for region nrt??