I am unable to login to the dashboard an I am unable to use fly ctl cli commands:
This has caused complete downtime of our application because two fly services cannot talk to each other:
request to http://top1.nearest.of.stackable-backend-prod.internal:8080/api/stack failed, reason: getaddrinfo ENOTFOUND top1.nearest.of.stackable-backend-prod.internal
Yes, got a similar issue with fly returning a 503 error on every flyctl command
Can confirm. Login/CLI all appear offline. I am still able to reach DNS and my servers are responding to webhook traffic.
Thanks @jerome and we appreciate you and the team keeping us up to date. :hugops:
The issue is marked as resolved now but I’m still seeing network issues.
Issues with the API specifically or private networking? I created a new incident for the private networking issues.
Both I think - flyctl ssh console was hanging, deploy kept failing and my app was unable to connect to the postgres db. However it all started working again about a minute ago.
That would make sense.
A lot hinges on private networking working correctly, including the ssh functionality, remote builders, connecting to pg clusters (using the .internal address).
We’ve possibly fixed this, we’re monitoring the situation.
thanks
This should all be fixed now. We have identified the cause and are implementing a permanent fix for this particular scenario.