What happens
When I connect to some Fly VM via ssh (via flyctl
as well as directly via ssh
and active Wireguard tunnel), this connection blocks/freezes as soon as some larger stream of data shall be transferred. (E.g. the output of dmesg
freezes after a few lines.) This happens reproducible on any VM I tried as well as on different local computers (Mac as well as Windows 10) But only when using some IPv6/IPv4 Dual Stack configuration:
- Cable: Vodafone cable over DS-Lite (IPv6 address only)
- Mobile: Deutsche Telekom/congstar LTE (IPv6 & IPv4 addresses)
Analysis
I suspect some fragmentation problem to be the root cause of the freeze. As DS-Lite does tunnel IPv4 packets over IPv6, the MTU of 1420 for the Wireguard link seems to lead to some MSS that is too large for the transport link and thus some fragmentation might occur at some place. Same for Mobile, also I can’t tell you anything about the transport network on the carrier side before the packets leave their CG-NAT.
Solution
When choosing some MTU of 1280 for the Wireguard link (MTU = 1280 in [Interface] configuration) everything works fine and no more freeze on the ssh session occurs. Also when tunneling my IPv4 traffic over some VPN tunnel with MTU 1280 everything works fine.
Unfortunately this approach does not work for flyctl ssh console
as here I can’t modify the MTU as I’ve found no parameter for this. Some other solution for this problem might be to have some IPv6 endpoint for the Wireguard client. (At least fra2.gateway.6pn.dev
is IPv4 only.)
Does anybody made similar observations?
Oliver
P.S: Also flyctl wireguard websockets enable
solves the issue with hanging ssh session.