One-way 6PN routing failure in AMS: Rails machine cannot reach Postgres machine

1

Two machines in ams on the same 6PN (fdaa:0:b2b4) have asymmetric routing. The Postgres machine can reach the Rails machine (gets connection refused on port 8080, meaning packets arrive). The Rails machine cannot reach the Postgres machine at all — raw TCP with cat < /dev/tcp/<ip>/<port> hangs on both port 5432 and 5433.

Tested with:

  • Multiple fresh machines
  • Direct IPv6 address (no DNS)
  • A machine cloned to fra region — same result
  • Pure TCP, no application layer involved

Both machines show HostStatus: ok. DNS resolves correctly on both sides.

This started sometime in the past week. No config changes were made.

2

what are the ids of the machines, or their full 6pn addresses?

3
  1. Rails: 178122ddf917d8 — fdaa:0:b2b4:a7b:c8:2fb:4927:2
  2. Postgres: 7849335a359e78 — fdaa:0:b2b4:a7b:4c9:8720:e32b:2
4

@ask1

Are you still experiencing the issue?

I can see the traffic on port 5432 in both directions on the host where the DB machine is running.

Specifically, I see packets from fdaa:0:b2b4:a7b:c8:2fb:4927:2 port 39810 to fdaa:0:b2b4:a7b:4c9:8720:e32b:2 on port 5432 and back.

5

Thanks. It’s up and running again.
What caused it?

6

No idea. When I looked at it the traffic was flowing just fine.

And I don’t see any indication in our metrics that there was a connectivity problem between the nodes.

7

Weird. It was out for hours, but resolved soon after @lillian asked for ids and addresses. So I assumed someone did something.
Anyways. Thanks