Fly app servers down!

40

We are up and running again!

This is what worked for us:

1. Destroy the running machines one by one

Open your dashboard to get the machine numbers:

image
image1116×316 33.9 KB

Then run

 $ fly m destroy --force 18579eda435778
machine 18579eda435778 was found and is currently in replacing state, attempting to destroy...
18579eda435778 has been destroyed
$ fly m destroy --force 9185955c103158
machine 9185955c103158 was found and is currently in starting state, attempting to destroy...
9185955c103158 has been destroyed

2. Restart machines

$ fly scale count 1 -c prod.fly.toml 
App 'lyn-pay' is going to be scaled according to this plan:
  +1 machines for group 'app' on region 'ams' of size 'shared-cpu-1x'
? Scale app lyn-pay? Yes
Executing scale plan
  Created e7843d1df492e8 group:app region:ams size:shared-cpu-1x

3. Redeploy

$ fly deploy

Note: if your builder machine is stuck like this:

WARN Remote builder did not start in time. Check remote builder logs with `flyctl logs -a fly-builder-falling-thunder-5451`

WARN Remote builder did not start in time. Check remote builder logs with `flyctl logs -a fly-builder-falling-thunder-5451`

Error: failed to fetch an image or build from source: error connecting to docker: remote builder app unavailable

Then do as @xwiles suggested:
Destroy the builder and deploy again

$ fly destroy fly-builder-falling-thunder-5451
Command "destroy" is deprecated, use `fly apps destroy` instead
Destroying an app is not reversible.
? Destroy app fly-builder-falling-thunder-5451? Yes
Destroyed app fly-builder-falling-thunder-5451
$ fly deploy

I hope this works for you as well :pray:
What a nervewrecking experience for all of us!
Grateful to have a community to solve this with.

2 Likes