Recurring Postgres flex connection timeouts from one Machine while another works

Questions / Help
,
1

I’m running Windmill with two worker Machines and a single-node Postgres Flex database in iad. One of my workers repeatedly loses database connectivity even though the other keeps working.
It happens to me almost daily. Today it lasted 3 hours. The worker shut down after failed database heartbeats, then kept failing on startup with:

Connecting to database: pool timed out while waiting for an open connection

It restarted automatically and repeatedly, eventually with 15-minute retry gaps, and then after about 3 hours recovered.
During that time:

  • The other worker remained operational.
  • Postgres stayed healthy, processing transactions and accepting some new sessions.
  • Sampled database connections were around 19–31 against max_connections=75, with no application-role limits.
  • No corresponding Postgres rejection errors were found in the logs.
    Both workers connect through .flycast:5432. After recovery, opening 25 additional concurrent connections from the affected worker succeeded. Similar failures have occurred on that worker at least five times previously.

This sort of resembles a reported Postgres Flex networking blackout ( Intermittent 10–60 s 6PN blackouts between two Machines in gru (app ↔ Postgres Flex) ), though ours is in iad.

Has anyone seen this pattern? Could it be host-specific networking, and what would be the most useful diagnostic to capture during the next incident?

5

Could you share your app name or one of the machine IDs that’s having this problem?

6

The affected Machine was 683460efd50118 in iad.
Its latest failure window was September 24, approximately 12:11–15:13 UTC. The other worker, 891e141b7d4698, remained working.

7

More on the same on the same Machine, 683460efd50118:

September 27, 10:55:47–13:16:39 UTC.

September 28 (today), 05:44:45–09:50:15 UTC

Same database pool timeouts and automatic restart loop; the other worker remained running.

It seems to be the same machine each time. Should I try to remove the machine and create a new one on a different host?

8

Thanks for sharing these data points @travisp. Over the last week and a half, we’ve been seeing intermittent packet loss from a small subset of hosts in iad with one of our providers. We asked them to look into this.

Incidentally, your unaffected Machine is on a host with another provider. While our upstream investigates the packet loss, I think it’s worth trying to move 683460efd50118 to a different provider.

You can attempt this by cloning and then destroying 683460efd50118 (fly machine clone <id> / fly machine destroy <id>), although this doesn’t guarantee your new Machine lands on another provider’s hardware. We can also coordinate a manual migration with our team - just let us know your preferred timeframe for this.