Once flyctl v0.2.51 is released (your local client should update automatically or run flyctl version update)
then destroy builder app fly apps list and fly app destroy BUILDERNAME
finally try to deploy an app with
FLY_REMOTE_BUILDER_REGION=iad fly deploy
Hi guys, the ams builder problem has drifted into a network problem with the S3 bucket used as registry backend. The difference with pushing layers from other regions is that the bucket has S3 Transfer Acceleration enabled, so Amsterdam will go into AWS from a different entrypoint than others.
In the meantime we ~have destroyed~ are destroying all ams builders and instructed our system to recreate them in iad on your next application deploy.
cheers.
Are the machines automatically scaled down because of the current problem? I reconfigured my toml to prevent stoppage.
I also have the same issues. App’s not working, tried scaling down and up again, machine restart is not working (stuck).
App Name: superstash-client
Fly Region: waw
same for me - i didnt get my app pushed and it gets stuck
App Name: bpbr-bot
Fly Region: cdg
i updated my cli and client and destroyed fly-builder several times - no chance 
Make sure you have 0.2.51, not 0.2.50, try to install using this
curl -L https://fly.io/install.sh | sh -s 0.2.51
Then as Dangra suggested, first destroy and then deploy with env var set
FLY_REMOTE_BUILDER_REGION=iad fly deploy
For me it helped. Just make sure that yiu use correct flyctl, because I first install using brew, then curl, and I had to point to the specific path to use 0.2.51
I don’t have a way to redeploy the site at the moment. Looks like Fly is having some issues so I expect them to fix it themselves.
I’m wondering if I should move away from using Fly altogether as those this happen too often.
I had the same message for my ams app. Scaling down and up worked for me. Maybe consider scaling down in waw and up in fra for example until they got it sorted? In my case, it was very simple because I don’t use volumes on fly.
Thinking the same, it doesn’t seem to be reliable
I can’t scale it down to 0. Even if I do this, the broken machine stays there and my app is marked as “Suspended”.
I run 2 machines on my app now, both in fra but the app still won’t start:
I tried distroing the broken machine but got 408 status:
Hi @dangra,
Thanks for sharing the insights, I am facing this issue with my application, app region is SIN, also I have tried removing the builder, but it still stuck at preparing step.
For people having problems with deploys that hang pushing docker image layers to the registry the way to unblock it is to destroy the builder app and start a new deploy forcing the builder region to iad
# be sure you are running flyctl v0.2.51 or greater
❯ flyctl version
flyctl v0.2.51 linux/arm64 Commit: 3cb555f3be736f5e50be72c20f8dfd268f8cf238 BuildDate: 2024-05-08T20:14:10Z
# find the builder app name and destroy it
❯ fly apps list |grep builder
fly-builder-winter-tree-7706 personal suspended
❯ fly apps destroy fly-builder-winter-tree-7706
...
# finally start a deploy forcing the builder region
❯ FLY_REMOTE_BUILDER_REGION=iad fly deploy
...
For others having issues with machines that can’t be started, I will check in a moment but seems to be a different issue.
Hi @talhajunaid , your app is in SIN but your builder is in AMS afaict. Please destroy it and follow above procedure.
Hi @d3im0s, your builder is in AMS too, please follow this instructions to recreate it in another region Machines won't start! - #72 by dangra
Sorry about the question but,
where do you write FLY_REMOTE_BUILDER_REGION=iad?
Writing it directly in the shell command before fly deploy says that’s it is not a recognized operation
For Linux and OSX it is all in the same line. Also the “❯” is the prompt, don’t include that.
Long time since I’ve used Windows but others may be able to help if it is your case.
Yes sorry for not specifying, I’m a WIndows user. Thanks a lot.
Thanks @dangra Its working now. Seems like HomeBrew doesn’t have v0.2.51, so I had to uninstall and reinstall it using CURL.
I’ve tried:
fly deploy -c fly.staging.toml --env FLY_REMOTE_BUILDER_REGION=iad
fly deploy -c fly.staging.toml -e FLY_REMOTE_BUILDER_REGION=iad
but the builder machine created is always placed at ams.
In the documentations it says:
-e, --env stringArray Set of environment variables in the form of NAME=VALUE pairs. Can be specified multiple times.
@matteo ChatGPT unreliably informs me that you can use set to set an environment variable in cmdshell, so:
set FLY_REMOTE_BUILDER_REGION=iad
fly deploy
Or in powershell:
$env:FLY_REMOTE_BUILDER_REGION="iad"; fly deploy
Take these with a grain of salt, as I’m not a windows user!