Deploys failing LHR, postgres proxy failures, intermittent db connection issues

1

Hi there,

Long time user and fan of fly.io.

For info tho having a lot of issues today.

I deployed twice this morning (UTC) but now (1830UTC) deploys are failing:

output: WARN Failed to start remote builder heartbeat: failed building options: failed probing "personal": context deadline exceeded

Error failed to fetch an image or build from source: error connecting to docker: failed building options: failed probing "personal": context deadline exceeded

I also cannot proxy my postgres instance to localhost like i have been able to prior to today. Error outputs there are:

Opening a wireguard tunnel to personal 🌎 Error tunnel unavailable for organization personal: failed probing "personal": context deadline exceeded

Additionally i have postgres ERRORs and SENTINELs in logs reporting issues. These are coincident with intermittent connection drop outs to the postgres instance.

All issues seem to relate to problems probing my personal parts lol. :laughing:

Can you help?

Thanks

Ben

1 Like
2

Hey,

just want to say that I see many issues in FRA too. Database connections are failing periodically since a few hours. App is basically up and down…

3

Database connection issues are something different! What are you seeing in your logs? Have you run fly checks list -a <db-name>?

4

I just scaled everything down and up again. Seems to be stable at the moment. I will post if this changes again. Sorry I should have done it earlier, but I thought because of the heavy IO incident there is no point in doing it.

5

Ah! I don’t believe that was affecting the host your DB is on. It had a very small blast radius.

If connections were failing, though, something was broken. It sounds like restarting fixed it, but I’m guessing it’ll happen again if you didn’t figure out exactly what was up.

6

FIXED, SOLUTION

Both the deploy and the postgres local proxy were affected by a context deadline exceeded error. This seemed to be an issue with the wireguard magic that happens.

I ran fly doctor and examined its output. Following the doc’s advice i ran fly wireguard reset and now both the deployment and proxy are fixed! Joy!!