Hi there y’all,
I tried deploying an update of one of my apps is HKG region and it is failing.
Last time I’ve updated this app was yesterday.
status.fly.io doesn’t give any region specific alarms.
I’ve noticed yesterday that the flybuilder app on my account was gone.
But during the deploy today, a new one got created.
I’ve tried deleting it and trying to deploy again, but the error is still happening.
Here is the error I am having:
==> Verifying app config
Validating /Users/***/fly.toml
Platform: machines
✓ Configuration is valid
--> Verified app config
==> Building image
WARN Failed to start remote builder heartbeat: Couldn't allocate volume, not enough compute capacity left in hkg
Error: failed to fetch an image or build from source: error connecting to docker: Couldn't allocate volume, not enough compute capacity left in hkg
Clone the original machine in hkg to a new one in sin
Stopped and deleted the machine in hkg
Delete the builder app (I tried doing it from the dashboard and from fly cmd line)
Changed fly.toml to use region sin
Still, the error persist.
When I run fly apps list, the builder shows in status pending.
But no matter what I’ve tried, I am still getting error related to the hkg region.
It is strange that the status page still shows 100% deployments…
Is there any other official channel I can check if there is issues going on?
Sounds like the issue is the remote builder, rather than with your app. Yep, you can bypass that by building locally using your local Docker.
The problem is the builders tend to be made close to you, rather than close to the app. I don’t think you can control where the builder is made . So if your closest region is hkg, that is where the builder will be made. Purely out of interest if you request https://debug.fly.dev/ in a browser, check the “fly-region” header to see where Fly returned the response from. That’s likely the closest region.
Thanks for your reply @greg .
I managed to get the --local-only to work. I was not happy about it, but I had to deploy a new release, so…
Is it the builder or the region?
It seems to me is the region, which downtime should have been reflected in the status page…
IMO hkg is “kinda of the closest”, but what matters is what fly deploy determine.
hkg is still unavailable (and still nothing in the status page).
Anyway, I hope hkg gets back on its feet like it was since yesterday. Or that one day I have the option to change this behavior, for the days my closes region goes offline…
Otherwise I will have move to another country to be able to deploy a new release to my fly apps!
I don’t know if it’s the builder or region. One for Fly. As you say, you’d assume the status page would reflect any issue with the region. Yet you say you destroyed the builder app and it happened again.
At some point they may add a flag for the region the builder should be created in. Until such time I wonder if you could use a VPN to make it create the builder in a different region No idea. Moving country would be a bit drastic .
Hey there, yes the builder app is scheduled where we believe you are latency-closest wherever you run fly deploy from.
We’ve been working on making these better schedule for capacity balancing and probably partially broke this yesterday - sorry about that!
We just made some more updates that I think might help your situation. Could you delete the builder app and try a deploy again? Definitely let us know how it goes.
@greg : I do believe that VPN would have gotten me around.
I will try that next time.
But my very first experience with Fly was problematic except because of that.
I’ve created my account and got blocked a few minutes later labeled as “high risk of fraud” or something like that… And could not move forward…
There was probably the first time I created a post here…
People, companies, sites will negatively label you, block you, if you use it… And you will have to bear the burden to figure it out and have them corrected… I speak entirely out of my own experience!
So… --local-only seems to me the best first step.
And I managed to get that working to release a new version of my app.
I certainly see this as some pain points and rough edges as Fly scale and grows.
This was the very first time I’ve upgraded one my Fly apps to a performance1 machine and DB, so again, the first time I am doing something and trying to use more, I hit some hurdles…
But to be honest: this app with performance1 machine/db worked perfectly for the whole last month. fly launch was just perfect!
Anyway… hope next time, when deployment fails due to technical issue in a region (hkg or else), that will be reflected on Status page, and will raise an alert to Fly Ops Team.
@jphenow : it is working now. Thanks for replying and looking into the issue.
A few comments:
I find it extremely unlikely that I am “latency-closer” to hkg compared with sin.
But I’ve noticed that since fly launch automatically selected it.
I will steer away for hkg for the foreseable future.
Have you identified why this hasn’t been flagged in the Status page?
By my accounts, deployments via hkg region was down for at about 1/2 day.
I started having issue yesterday morning, and yesterday night, after @greg’s message I tested again and it was still down.
:hugs-to-the-fly-ops-team: scaling the infra with live customer relying. But an acknowledgment or some info in the Status page would have helped me a ton. Which I also know that has pros & cons (IMO other people will just slum dunk on it… but not me).
It was also trick for me to know if the Fly team was aware of that or not…
Anyway, it is working now, and next time I will have more tools to minimize the impact on my dev/deploy process.
@greg : moving to another country is definitively not a viable option!
But I deeply appreciate your suggestion.
@jphenow : one other details that I don’t think I’ve mentioned is this: fly apps list was showing the builder in pending state and that helped me.
It was clear that while the app was in Pending I’d have to find another way, which put me in the --local-only.
I’ve managed to move the machines from hkg to sin easily with fly machine clone and -r parameter (I think I forgot to move the DB), so if I’d have the option to do that in fly deploy to designate the region where the builder app goes, despite the internal logic about latency-closeness, I would have found a solution faster.
I’ve looked for that option in the Fly docs.
I’ve also tried to get info about which region the builder app was. And neither in the dashboard nor via the cli I was able to get it…
Just some extra feedback. I hope it helps.
These things would have helped me to get to a solution faster.
I hope one day something like this gets priorized for situations where the latency-closest region is unhealthy.
) and fix a unix socket sym link issue (between fly and docker) on my linux desktop.
. So if your closest region is hkg, that is where the builder will be made. Purely out of interest if you request 
.