Since about 13:55 UTC on Sep 23, two Postgres machines in yyz can’t be reached over the private network. It looks like both are on the same host. Both pass their health checks and accept connections locally, but TCP to them fails from every other machine in my org, and they can’t make outbound connections either.
Affected machines (both show 6PN host segment 144):
- ridesnridesnrides-db, machine 9e78497df4e983, fdaa:0:d07b:a7b:144:7359:e9b1:2
- waitsfieldpta-db, machine 9e78497ec66683, fdaa:0:d07b:a7b:144:8a52:cc30:2
What I’ve tested (TCP connect with a 5s timeout):
- App machine on the same host (9080e520c5e987, yyz) → ridesnridesnrides-db.internal:5432: timeout
- Fresh clone on a different host (80e30da36d2318, yyz, …:1d3:…) → both DBs on 5432: timeout
- waitsfieldpta app (e7846e1bd02683, ord) → waitsfieldpta-db.internal:5432: timeout
- From inside ridesnridesnrides-db → its own 6PN IP on 5432/5433: OK
- From inside ridesnridesnrides-db → waitsfieldpta-db.internal:5432 and to the app machine’s port 22: timeout
- .internal DNS still resolves to the correct addresses, and fly ssh console into every machine still works.
Restarting the DB and app machines didn’t help, and there’s nothing on the status page. Could someone check that host’s 6PN / WireGuard mesh? Both apps are down in production because of this.