Does `fly deploy` and `fly postgres connect` work on IPv4-only clients?

I am just assessing whether fly.io is good for our purposes but I got hit by a roadblock. Apparently neither fly deploy nor fly postgres connect work if you only have IPv4 enabled / available. While I hoped this won’t be a problem in 2023 - it looks like apparently although the UK’s biggest home broadband provider still doesn’t provide IPv6 in 2023… https://test-ipv6.com/ gives me a 0/10 score.

Again back to the issue: both of the commands above try to connect to IPv6 hosts which will always time out / fail, as IPv6 access is simply not available at all, example:

fly postgres connect
Connecting to fdaa:3:c1c1:a7a:af75:f531:5542:2... complete
Error: error connecting to SSH server: connect tcp [fdaa:3:c1c1:a7b:be65:e623:2455:2]:22: operation timed out

Is there any way of forcing these commands to run on IPv4, or I am out of luck as these won’t work without having access to IPv6?

(I managed to work around the issue by creating GitHub actions that ran the commands for me, but that’s far from ideal, especially during assessment)

1 Like

Hmm, that’s strange! It should be working via wireguard tunnelling, which means your ISP shouldn’t need to support IPV6. Mine doesn’t, and I also get 0/10 on that site, but I am able to fly postgres connect.

Can you run fly doctor and see if there are any issues?

WireGuard also doesn’t work for me on networks where IPv6 is not enabled. I have tried on both OSX and Windows boxes without any luck. I have also tried it tethering from my mobile internet to see if my home router might be the problem, but also no luck.

On my EC2 where I could use the deployment commands WireGuard also worked without any issues

That sounds like something different, but I’ll see if I can figure something out. It should ‘just work’ – most of us who work here also don’t have ipv6 supported by our internet providers. So, I wanna see if there’s anything else that can causes issues.

Did fly doctor report anything?

Testing authentication token... PASSED
Testing flyctl agent... PASSED
Testing local Docker instance... PASSED
Pinging WireGuard gateway (give us a sec)... FAILED
(Error: ping gateway: no response from gateway received)

Logs from the agent:

2023/12/05 00:16:47.121431 #4 connected ...
2023/12/05 00:16:47.121597 #4 <- (   18) "establish personal"
wg connect fdaa:3:c1c1::3 lhr1.gateway.6pn.dev:51820 fdaa:3:c1c1:a7b:dc6:0:a:100 fdaa:3:c1c1::
2023/12/05 00:16:47.529888 srv returning port: 52032
2023/12/05 00:16:47.529970 srv (re-)connecting to wss://lhr1.gateway.6pn.dev:443/
2023/12/05 00:16:47.591244 #4 dropped.
2023/12/05 00:16:47.594891 #5 connected ...
2023/12/05 00:16:47.594981 #6 connected ...
2023/12/05 00:16:47.595031 #5 <- (   14) "ping6 personal"
2023/12/05 00:16:47.595086 #6 <- (   18) "establish personal"
2023/12/05 00:16:47.721107 #6 dropped.
2023/12/05 00:16:50.745525 #5 dropped.

I can definitely ping lhr1.gateway.6pn.dev and also connect to it via telnet so it’s accessible

I did not send all of the agent logs last time here they are with some bits redacted:

2023/12/05 00:24:30.362361 #a connected ...
2023/12/05 00:24:30.362608 #a <- (   18) "establish personal"
2023/12/05 00:24:30.555127 #a -> (  753) "\xef\x02ok {\"WireGuardState\":{\"org\":\"personal\",\"name\":\"X\",\"region\":\"lhr\",\"localPrivateKey\":\"X\",\"localpublic\":\"X\",\"dns\":\"\",\"peer\":{\"peerip\":\"fdaa:3:c1c1:a7b:dc6:0:a:102\",\"endpointip\":\"lhr1.gateway.6pn.dev\",\"pubkey\":\"X\"}},\"TunnelConfig\":{\"LocalPrivateKey\":\"[redacted]\",\"LocalNetwork\":\"fdaa:3:c1c1:a7b:dc6:0:a:100/120\",\"RemotePublicKey\":\"X\",\"RemoteNetwork\":\"fdaa:3:c1c1::/48\",\"Endpoint\":\"lhr1.gateway.6pn.dev:51820\",\"DNS\":\"fdaa:3:c1c1::3\",\"KeepAlive\":0,\"MTU\":0,\"LogLevel\":0}}\n"
2023/12/05 00:24:30.555250 #a dropped.
2023/12/05 00:24:30.555607 #b connected ...
2023/12/05 00:24:30.555682 #b <- (   14) "ping6 personal"
2023/12/05 00:24:30.555739 #c connected ...
2023/12/05 00:24:30.555809 #c <- (   18) "establish personal"
2023/12/05 00:24:30.679711 #c -> (  753) "\xef\x02ok {\"WireGuardState\":{\"org\":\"personal\",\"name\":\"X\",\"region\":\"lhr\",\"localPrivateKey\":\"X\",\"localpublic\":\"X\",\"dns\":\"\",\"peer\":{\"peerip\":\"fdaa:3:c1c1:a7b:dc6:0:a:102\",\"endpointip\":\"lhr1.gateway.6pn.dev\",\"pubkey\":\"X\"}},\"TunnelConfig\":{\"LocalPrivateKey\":\"X\",\"LocalNetwork\":\"fdaa:3:c1c1:a7b:dc6:0:a:100/120\",\"RemotePublicKey\":\"X\",\"RemoteNetwork\":\"fdaa:3:c1c1::/48\",\"Endpoint\":\"lhr1.gateway.6pn.dev:51820\",\"DNS\":\"fdaa:3:c1c1::3\",\"KeepAlive\":0,\"MTU\":0,\"LogLevel\":0}}\n"
2023/12/05 00:24:30.679841 #c dropped.
2023/12/05 00:24:33.705282 #b dropped.

Managed to get this working, and yes, it was not IPv6 related!

So what I did was I connected to the EC2 instance where it was working and obtained the wire_guard_state value from ~/.fly/config.yaml, then copied that verbatim into the fly/config.yaml on my computer. All fly commands work now.

Do note that the EC2 instance has connected to the ams region not lhr, so maybe the problem is that something’s wrong with the lhr wireguard service? If so it’s odd that it doesn’t crop up on the status reports though

1 Like

Interesting! I’m hitting exactly the same conditions against LHR, also seeing “establish personal” seemingly work, but “ping6 personal” get dropped without reply.

DEBUG: (fly-ssh) 2023/12/05 23:15:43 peer(eCP0…ZyFk) - Sending handshake initiation
2023/12/05 23:15:46.427845 #5 dropped.

At first I thought it was related to Tailscale conflicting with it somehow, so I uninstalled that completely and restarted. Gone through lots of agent restarting and doctor’ing.

I created a new wireguard configuration to LHR, and imported it into the macos wireguard client. Which returned the same log messages…

2023-12-05 23:42:33.911 [NET] peer(eCP0…ZyFk) - Sending handshake initiation
2023-12-05 23:42:39.001 [NET] peer(eCP0…ZyFk) - Handshake did not complete after 5 seconds, retrying (try 2)
2023-12-05 23:42:39.002 [NET] peer(eCP0…ZyFk) - Sending handshake initiation

And naturally I couldn’t hit the in-network dns server while the tunnel was “active”.

 dig TXT _apps.internal +short
; <<>> DiG 9.10.6 <<>> TXT _apps.internal +short
;; global options: +cmd
;; connection timed out; no servers could be reached

I also tried with an external client to emulate what @sztupy did, but with GCP cloud shell and I got the same symptoms.

$ /home/ben/.fly/bin/flyctl doctor
Testing authentication token... PASSED
Testing flyctl agent... PASSED
Testing local Docker instance... PASSED
Pinging WireGuard gateway (give us a sec)... FAILED
(Error: ping gateway: no response from gateway received)

This is all with the “latest” 0.1.131 client.

Which region did you run the GCP one? The config I got for the ams region still works for me today, but I still cannot connect through the lhr one I would get by default if I reset the agent or wireguard settings

i got the same issue, just trying fly.io and it’s work great last week, but starting from sunday, i can’t proxy postgres app to my local… fly doctor always give me the same error

Pinging WireGuard gateway (give us a sec)... FAILED
(Error: ping gateway: no response from gateway received)

We can't establish connectivity with WireGuard for your personal organization.

WireGuard runs on 51820/udp, which your local network may block.

i have tried on my home and office network (IPv4 only), and try to connect via VPS that i know for sure it have IPv6 gateway, still got the same issue

This morning it’s working just fine to lhr… It may have just been the incident I see in the status page (Fly.io Status - Elevated deploy errors) that was making this fail last night and it just coincided with me onboarding. Bad luck.

$ flyctl doctor
Testing authentication token... PASSED
Testing flyctl agent... PASSED
Testing local Docker instance... PASSED
Pinging WireGuard gateway (give us a sec)... PASSED
No app provided; skipping app specific checks

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.