Fly can't connect to remote builder in automated environment anymore (/cleaning up automated connections)

1

I’ve been running an automated Fly.io deployment setup with GitHub workflows, and it’s been working as expected. This is how a production deployment log usually looks like, running flyctl deploy --now --remote-only to deploy a new app version for every new commit on the production branch (and some other stuff due to this other bug, but that’s not relevant). Since the day before yesterday, this has changed, and I’m encountering a variety of errors before/after connecting to the remote builder:

  • “no space left on device” log
  • “remote builder app unavailable” log
  • “context deadline exceeded” (this happens when trying to deploy from my local machine to the remote builder)

I went through a couple of other reports on these error messages and ran

$ fly ping -o lsglab

Which didn’t output anything (unlike my personal builder). After running fly wireguard list, (I think) I found the culprit: my personal builder only has one interactive- connection to my own computer, but the organization one that is causing problems has 91. I have tried removing a couple of them individually, one-by-one using fly wireguard remove, and now I’m getting a “remote builder app unavailable”/“builder did not start in time” error too.
How can I get this builder to work again? Is there a command to clear all WireGuard peers? (Would that help?) Should I and How can I stop persisting all these automated GitHub runner environments as WireGuard peers?

2

Hey there!
It looks like your builder is at full capacity, the easiest way to fix this issue is to destroy the builder, a new one will be created with a fresh volume upon the next deployment.
fly apps destroy fly-builder-wild-tree-6339

3

Thank you, that fixed it! Is there a way I can prevent this from happening in the future? This is an automated environment, so regular manual intervention isn’t ideal… :slight_smile: