Machines won't start!

1

All our machines are gone and now new ones wont start

Can someone help us as our whole app is down?

Error: could not start machine 3d8d700dc31e48: failed to start VM 3d8d700dc31e48: failed_precondition: unable to start machine from current state: ‘created’ (Request ID: 01HXC0APZG26WPGTP8J49637MC-lhr)

2

We are having the same issue here. All our infrastructure is down.

Can someone tell when this is gonna be fixed please ?

Eliot

3

Hey! We have an active incident with machine starts. We have identified the root case, and are pushing out a fix.

4

Any update with this? Unable to launch any instances.
Is there any way we can launch instances? I tried scaling to 0 and back up as well.

5

We also can’t seem to push new deployments through the ctl (it hangs during upload of new images), and the live logs show failed to change machine state: machine getting replaced, refusing to start

6

Ridiculous. They took more than 3 hours to fix it while everything was down. I’ve been on AWS before and was thinking to shift to fly, but after this experience I really cannot trust them. Moving to Render instead.

7

Does it work for you? I still can’t push new deployment

8

Machines in one of our apps still aren’t starting. The service has been down for hours and customers are complaining.

Surprised about the status page…

image
image1550×746 25.2 KB

9

Hi guys, I am looking at the issue with the failing to start machines. It helps me if you share your app names or a machine id to check

10

I redeployed the app and the machines are passing health checks but still appear stopped when doing fly machines list and are not responding to HTTP requests.

image
image1388×464 96.6 KB

11

The app name is wavekit-audio-encoder-production.

It seems I can start machines with fly machines start but still not responding to HTTP requests.

12

Ok things seem to be working now!

13

I am seeing your machines starting and self gracefully shutting down a minute later

image
image1160×426 27 KB

14

hey @dominiks, do you mind sharing your fly deploy output or the error you get when deploying?

15

Yeah this is normal. I implemented scale to zero before Fly offered it and haven’t migrated yet.

16

It basically stops when trying to push

image
image844×251 7.97 KB

WARN failed to finish build in graphql: Post "https://api.fly.io/graphql": context canceled
17

Hi @aychtang, could you share the app name or machine id ? I see your machines failing to start because of a missing file .env.local but can’t find a machine in “replacing” state.

18

Same here,

image
image1410×536 73.6 KB

Registry seems fucked too

19

The push refers to repository [registry-iad.fly.io/customer-credits]
ff10b430d065: Waiting
a0195938cfde: Waiting
ac8cfc402b6d: Waiting
f47424876570: Waiting
889fe9bc0aaa: Waiting
e0ce87e958a0: Waiting
f0fc506f7982: Waiting
46cf0e48c979: Waiting
c3e35b2afb23: Waiting
b7fcac299347: Waiting

Just tried a different registry, same thing: registry-iad.fly.io

20

I’ve enabled LOG_LEVEL=debug

The push refers to repository [registry-iad.fly.io/customer-credits]
ff10b430d065: Waiting
DEBUG Sending remote builder heartbeat pulse to http://fdaa:0:b44e:a7b:41:b74d:a294:2:8080/flyio/v1/extendDeadline...

DEBUG Sending remote builder heartbeat pulse to http://fdaa:0:b44e:a7b:41:b74d:a294:2:8080/flyio/v1/extendDeadline...

DEBUG Sending remote builder heartbeat pulse to http://fdaa:0:b44e:a7b:41:b74d:a294:2:8080/flyio/v1/extendDeadline...

DEBUG Sending remote builder heartbeat pulse to http://fdaa:0:b44e:a7b:41:b74d:a294:2:8080/flyio/v1/extendDeadline...

DEBUG Sending remote builder heartbeat pulse to http://fdaa:0:b44e:a7b:41:b74d:a294:2:8080/flyio/v1/extendDeadline...
ff10b430d065: Waiting
a0195938cfde: Waiting
ac8cfc402b6d: Waiting
f47424876570: Waiting
889fe9bc0aaa: Waiting
e0ce87e958a0: Waiting
f0fc506f7982: Waiting
46cf0e48c979: Waiting
c3e35b2afb23: Waiting
b7fcac299347: Waiting
ec4a38999118: Waiting
DEBUG Sending remote builder heartbeat pulse to http://fdaa:0:b44e:a7b:41:b74d:a294:2:8080/flyio/v1/extendDeadline...