Fly is down?

I am unable to login to the dashboard an I am unable to use fly ctl cli commands:

This has caused complete downtime of our application because two fly services cannot talk to each other:

request to http://top1.nearest.of.stackable-backend-prod.internal:8080/api/stack failed, reason: getaddrinfo ENOTFOUND top1.nearest.of.stackable-backend-prod.internal

Yes, got a similar issue with fly returning a 503 error on every flyctl command

1 Like

Can confirm. Login/CLI all appear offline. I am still able to reach DNS and my servers are responding to webhook traffic.

1 Like
1 Like

Thanks @jerome and we appreciate you and the team keeping us up to date. :hugops:

The issue is marked as resolved now but I’m still seeing network issues.

Issues with the API specifically or private networking? I created a new incident for the private networking issues.

1 Like

Both I think - flyctl ssh console was hanging, deploy kept failing and my app was unable to connect to the postgres db. However it all started working again about a minute ago.

That would make sense.

A lot hinges on private networking working correctly, including the ssh functionality, remote builders, connecting to pg clusters (using the .internal address).

We’ve possibly fixed this, we’re monitoring the situation.

thanks

1 Like

This should all be fixed now. We have identified the cause and are implementing a permanent fix for this particular scenario.