IPv6 PMTUD issue

I’ve noticed some issues with connecting to fly.io from an IPv6 network with < 1500 MTU links internally (ie using tunnels). You may well already be aware of this, but thought I’d write this in case not! Connections hang during TLS handshakes, which appears to be down to a path MTU discovery issue. After many retries, connections do eventually establish.

I can see replies from fly arriving at my router larger than my internal MTU (1420) which causes me to send back packet too big replies, but the vast majority of these are ignored. I’m guessing this is down to the network architecture involved, with ECMP between multiple servers on your side - if that’s the case, most of the time the packet too big response won’t end up going to the server which is answering the original connection.

RFC7690 describes this in more detail, but in short packet too big messages either need to be delivered to all ECMP members, or the correct server that sent the large packet (ie if you can do the hashing based on the packet inside the icmp message). An alternative option is to just avoid sending >1280b packets at all. Cloudflare also have a nice write up of their approach to this.

Of course I could be completely wrong in this guess, and it might be down to simple ICMP rate limiting or something else completely! Happy to provide any information useful for debugging.

1 Like

Yes, this is something we’ve been chasing down for weeks and hope to fix soon. Your detailed notes are very helpful, so thank you!

1 Like

@tds we’ll try the quick fix for now (setting 1280 MTU on the default ipv6 route)- could you repeat your testing and see if it’s working any better for you with this change?

2 Likes

Yep, just tested again - that works great! I confirm I’m seeing inbound packet sizes limited to 1280 now.

Thanks for sorting this so quickly - I’ve contacted various providers about issues such as this, often with varied responses - very impressed with how quickly this was fixed :slight_smile:

1 Like

See more workarounds here: https://blog.cloudflare.com/path-mtu-discovery-in-practice/