All my apps' machines disappeared!

1

I just noticed that all my machines from all my apps except one vanished this morning.

When I run fly machine list it will either tell my that there are no machines for the app, or it will give me an error:

core [add_fly_io]× » fly machine list --app eduardo-backend-dev
No machines are available on this app eduardo-backend-dev

core [add_fly_io]× » fly machine list --app eduardo-db-dev
No machines are available on this app eduardo-db-dev

core [add_fly_io]× » fly machine list --app eduardo-frontend-dev
No machines are available on this app eduardo-frontend-dev

core [add_fly_io]× » fly machine list --app jeferson-frontend-dev
Error: failed to list VMs: app not found (Request ID: 01K0CHNZ7N6ZQ3WG915QMSDPRV-gru)

core [add_fly_io]× » fly machine list --app jeferson-backend-dev
Error: failed to list VMs: app not found (Request ID: 01K0CHPBJV2ETABBPQHB63HWM6-gru)

core [add_fly_io]× » fly machine list --app jeferson-db-dev
Error: failed to list VMs: app not found (Request ID: 01K0CHPNQ26HT4JJBPPFMZNRJC-gru)

Any idea of what is happening?

2

I don’t think we have had a report like this before here. What region are you operating in? What does the web console say; are the apps gone from there too? Are you sure you did not delete them?

3

It is this incident: Fly.io Status - Elevated "App Not Found" errors

It fixed for me after they did the fix push 20 minutes ago

4

Super. Did you apps carry on running during the incident?

5

They were all configured to auto-stop, but I couldn’t make them start going to their url

6

Gotcha. I think any that were already running would’ve been OK, but yes, if the app API was faulty, I should think start-on-request would break.

7

Actually, that part seems to still be broken, even if I start the machine, I still can’t access it from the url

8

My app in London seems to be fine currently.

It may be worth shelling into the app to see that the listener is OK. Also, see if you can curl from another machine to the affected machine; you should be able to reach your app via IPv6 on the private network.

9

I can ssh into it just fine, it is just accessing it from the url that is not working. It doesn’t trigger the machine to start, and, if i start it manually, ti doesn’t connect to it.
The db seems broken too, it fails to start.

I will wait until the current issued status is considered fixed and then retry

10

Just reporting that all my machines are back working just fine :smiley:

1 Like