Today we tried to do some deploys and first we see:
Error: failed to update VM 48edd90c773558: request returned non-2xx status, 504 (Request ID: 01HCAEWXRMQMGFCHDFVSFBMFGC-dfw)
In an attempt to solve it I ran fly m destroy on that machine. Redeploying and it still doesn’t work, but it just times out after a while:
Pulling container image registry.fly.io/......
I destroyed the builder (I saw someone on the forum suggest this) and the release command succeeded. Moving on, the next process group runs into 504 timeout again.
What do I do?
Thanks.
PS: After the release command finished and it created the first VM for the first process group with 256MB, way too small for our app. This is because scale information is stored on the machine level. Destroying a machine and running a deploy again, and you lose all scale information of the process group. This was explained here: Memory settings get lost after scaling to zero. I think it’s a terrible model, but that’s a sidenote. I think it should be stored on some “process” entity that is independent of how many VMs there are in that process group.
Now it replaced the first two process groups, but the most important one (the one that serves the web requests) went offline because it got stuck in “replacing”. So production is now offline. Would love some guidance. I’ve also emailed support.
This is still an issue, even on different apps in different regions. I see 504 errors even after destroying the machine & running deploy. In the deploy command I see a 504 error and in the logs I just see “Pulling container image registry.fly.io/abc” where it just seems to be stuck
I killed the builder and it proceeded through the first 2 process groups. Then this for the last process group:
2023-10-10T13:35:44Z runner[e28601da740598] ams [info]Pulling container image registry.fly.io/dentalcens:deployment-01HCCWRFW88Y6JG5T3FDHXN2BM
2023-10-10T13:36:11Z proxy[e28601da740598] ams [error]machine is in a non-startable state: created
2023-10-10T13:36:11Z proxy[e28601da740598] ams [error]machine is in a non-startable state: created
2023-10-10T13:36:11Z proxy[e28601da740598] ams [error]machine is in a non-startable state: created
2023-10-10T13:36:12Z proxy[e28601da740598] ams [error]machine is in a non-startable state: created
2023-10-10T13:36:15Z proxy[e28601da740598] ams [error]machine is in a non-startable state: created
2023-10-10T13:36:17Z proxy[e28601da740598] ams [error]machine is in a non-startable state: created
2023-10-10T13:36:19Z proxy[e28601da740598] ams [error]machine is in a non-startable state: created
2023-10-10T13:36:21Z proxy[e28601da740598] ams [error]machine is in a non-startable state: created
2023-10-10T13:36:23Z proxy[e28601da740598] ams [error]machine is in a non-startable state: created
2023-10-10T13:36:25Z proxy[e28601da740598] ams [error]machine is in a non-startable state: created
2023-10-10T13:36:27Z proxy[e28601da740598] ams [error]machine is in a non-startable state: created
2023-10-10T13:36:29Z proxy[e28601da740598] ams [error]machine is in a non-startable state: created
2023-10-10T13:36:31Z proxy[e28601da740598] ams [error]machine is in a non-startable state: created
2023-10-10T13:36:33Z proxy[e28601da740598] ams [error]machine is in a non-startable state: created
2023-10-10T13:36:36Z proxy[e28601da740598] ams [error]machine is in a non-startable state: created
2023-10-10T13:36:38Z proxy[e28601da740598] ams [error]machine is in a non-startable state: created
2023-10-10T13:36:40Z proxy[e28601da740598] ams [error]machine is in a non-startable state: created
