Unable to establish connection with WireGuard

Hi,

When I am running the latest version of Flyctl… I have been using it for some time and had no issues. I am recently facing issues with Wireguard.

When I run flyctl doctor I see the following output:

flyctl doctor

Testing authentication token... PASSED
Testing flyctl agent... PASSED
Testing local Docker instance... Nope
Pinging WireGuard gateway (give us a sec)... FAILED
(Error: ping gateway: no response from gateway received)

We can't establish connectivity with WireGuard for your personal organization.

WireGuard runs on 51820/udp, which your local network may block.

If this is the first time you've ever used 'flyctl' on this machine, you
can try running 'flyctl doctor' again.
flyctl version
flyctl v0.0.306 darwin/amd64 Commit: b5101f2 BuildDate: 2022-03-15T18:17:35Z

I have tried running for a second time and still see the same error. I am not using a VPN / Proxy or any other networking config.

This is preventing me from doing a flyctl deploy as I see the following output:

DEBUG Loaded flyctl config from/Users/sambird/.fly/config.yml
DEBUG determined hostname: "Samuels-MacBook-Pro.local"
DEBUG determined working directory: "/Users/sambird/Dev/sjb-welding/app"
DEBUG determined user home directory: "/Users/sambird"
DEBUG determined config directory: "/Users/sambird/.fly"
DEBUG ensured config directory exists.
DEBUG ensured config directory perms.
DEBUG cache loaded.
DEBUG config initialized.
DEBUG initialized task manager.
DEBUG started querying for new release
DEBUG client initialized.
DEBUG app config loaded from /Users/sambird/Dev/sjb-welding/app/fly.toml
==> Verifying app config
--> Verified app config
==> Building image
DEBUG trying local docker daemon
DEBUG Local docker daemon unavailable
DEBUG trying remote docker daemon
DEBUG Reporting buildDEBUG --> POST https://api.fly.io/graphql {{"query":"mutation($input: StartSourceBuildInput!) { startSourceBuild(input: $input) { sourceBuild { id } } }","variables":{"input":{"appId":"sjbwelding-ltd"}}}
}
DEBUG <-- 200 https://api.fly.io/graphql (5.45s) {"errors":[{"message":"StartSourceBuildInput isn't a defined input type (on $input)","locations":[{"line":1,"column":10}],"path":["mutation"],"extensions":{"code":"variableRequiresValidType","typeName":"StartSourceBuildInput","variableName":"input"}},{"message":"Field 'startSourceBuild' doesn't exist on type 'Mutations'","locations":[{"line":1,"column":44}],"path":["mutation","startSourceBuild"],"extensions":{"code":"undefinedField","typeName":"Mutations","fieldName":"startSourceBuild"}},{"message":"Variable $input is declared by anonymous mutation but not used","locations":[{"line":1,"column":1}],"path":["mutation"],"extensions":{"code":"variableNotUsed","variableName":"input"}}]}
DEBUG Failed storing buildDEBUG Trying 'Buildpacks' strategy
DEBUG no buildpack builder configured, skipping
DEBUG result image:<nil> error:<nil>
DEBUG Trying 'Dockerfile' strategy
DEBUG --> POST https://api.fly.io/graphql {{"query":"mutation($input: EnsureMachineRemoteBuilderInput!) { ensureMachineRemoteBuilder(input: $input) { machine { id state ips { nodes { family kind ip } } }, app { name organization { id slug } } } }","variables":{"input":{"appName":"sjbwelding-ltd","organizationId":null}}}
}
DEBUG <-- 200 https://api.fly.io/graphql (1.07s) {"data":{"ensureMachineRemoteBuilder":{"machine":{"id":"82531f67","state":"starting","ips":{"nodes":[{"family":"v6","kind":"privatenet","ip":"fdaa:0:1c5d:a7b:25db:0:ac37:2"},{"family":"v6","kind":"public","ip":"2604:1380:81:d02::ac37:3"},{"family":"v4","kind":"private","ip":"172.19.3.50"}]}},"app":{"name":"fly-builder-patient-tree-7785","organization":{"id":"2j9m3yRpYo94bH7l7KOeDQoOeaFJxz","slug":"personal"}}}}}
DEBUG checking ip &{Family:v6 Kind:privatenet IP:fdaa:0:1c5d:a7b:25db:0:ac37:2 MaskSize:0}
DEBUG --> POST https://api.fly.io/graphql {{"query":"query ($appName: String!) { app(name: $appName) { id name hostname deployed status version appUrl platformVersion config { definition } organization { id slug } services { description protocol internalPort ports { port handlers } } ipAddresses { nodes { id address type createdAt } } imageDetails { repository version } machines{ nodes { id name config state region createdAt app { name } ips { nodes { family kind ip maskSize } } host { id } } } } }","variables":{"appName":"sjbwelding-ltd"}}
}
Waiting for remote builder fly-builder-patient-tree-7785... ⣽ DEBUG <-- 200 https://api.fly.io/graphql (468.34ms) {"data":{"app":{"id":"sjbwelding-ltd","name":"sjbwelding-ltd","hostname":"sjbwelding-ltd.fly.dev","deployed":false,"status":"pending","version":0,"appUrl":null,"platformVersion":null,"config":{"definition":{"kill_timeout":5,"kill_signal":"SIGINT","processes":[],"experimental":{"allowed_public_ports":[],"auto_rollback":true},"services":[{"processes":["app"],"protocol":"tcp","internal_port":8080,"concurrency":{"soft_limit":20,"hard_limit":25,"type":"connections"},"ports":[{"port":80,"handlers":["http"]},{"port":443,"handlers":["tls","http"]}],"tcp_checks":[{"interval":"15s","timeout":"2s","grace_period":"1s","restart_limit":0}],"http_checks":[],"script_checks":[]}],"env":{}}},"organization":{"id":"2j9m3yRpYo94bH7l7KOeDQoOeaFJxz","slug":"personal"},"services":[{"description":"TCP 80/443 ⇢ 8080","protocol":"TCP","internalPort":8080,"ports":[{"port":80,"handlers":["HTTP"]},{"port":443,"handlers":["TLS","HTTP"]}]}],"ipAddresses":{"nodes":[]},"imageDetails":{"repository":"unknown","version":"unknown"},"machines":{"nodes":[]}}}}
DEBUG --> POST https://api.fly.io/graphql {{"query":"mutation($input: ValidateWireGuardPeersInput!) { validateWireGuardPeers(input: $input) { invalidPeerIps } }","variables":{"input":{"peerIps":["fdaa:0:1c5d:a7b:1596:0:a:302"]}}}
}
Waiting for remote builder fly-builder-patient-tree-7785... ⣻ DEBUG <-- 200 https://api.fly.io/graphql (174.14ms) {"data":{"validateWireGuardPeers":{"invalidPeerIps":[]}}}
DEBUG result image:<nil> error:error connecting to docker: failed building options: failed probing "personal": context deadline exceeded
Error failed to fetch an image or build from source: error connecting to docker: failed building options: failed probing "personal": context deadline exceeded```

I try 'probing' or pinging with below command to the personal but it just hangs.

`fly ping -o personal`

I have restarted agents, recreated wireguard, the lot. 

Please can someone help? Thanks

Hey to help debug further can you try some of of these commands:

fly agent stop
LOG_LEVEL=debug fly agent daemon-start

Hi Rahmat,

Thanks for your quick response and offering to help me.

It just seems to be in a loop that isn’t ending on validating wireguard peers.

Please find below output:

LOG_LEVEL=debug fly agent daemon-start

DEBUG Loaded flyctl config from/Users/sambird/.fly/config.yml
DEBUG determined hostname: "Samuels-MacBook-Pro.local"
DEBUG determined working directory: "/Users/sambird"
DEBUG determined user home directory: "/Users/sambird"
DEBUG determined config directory: "/Users/sambird/.fly"
DEBUG ensured config directory exists.
DEBUG ensured config directory perms.
DEBUG cache loaded.
DEBUG config initialized.
DEBUG initialized task manager.
DEBUG started querying for new release
DEBUG client initialized.
2022/03/17 16:00:26.305004 srv OK 3850
DEBUG --> POST https://api.fly.io/graphql {{"query":"mutation($input: ValidateWireGuardPeersInput!) { validateWireGuardPeers(input: $input) { invalidPeerIps } }","variables":{"input":{"peerIps":["fdaa:0:1c5d:a7b:1596:0:a:302"]}}}
}
DEBUG <-- 200 https://api.fly.io/graphql (5.48s) {"data":{"validateWireGuardPeers":{"invalidPeerIps":[]}}}
2022/03/17 16:02:31.846347 srv validated wireguard peers
DEBUG --> POST https://api.fly.io/graphql {{"query":"mutation($input: ValidateWireGuardPeersInput!) { validateWireGuardPeers(input: $input) { invalidPeerIps } }","variables":{"input":{"peerIps":["fdaa:0:1c5d:a7b:1596:0:a:302"]}}}
}
DEBUG <-- 200 https://api.fly.io/graphql (5.35s) {"data":{"validateWireGuardPeers":{"invalidPeerIps":[]}}}
2022/03/17 16:04:37.262596 srv validated wireguard peers
DEBUG --> POST https://api.fly.io/graphql {{"query":"mutation($input: ValidateWireGuardPeersInput!) { validateWireGuardPeers(input: $input) { invalidPeerIps } }","variables":{"input":{"peerIps":["fdaa:0:1c5d:a7b:1596:0:a:302"]}}}
}
DEBUG <-- 200 https://api.fly.io/graphql (5.31s) {"data":{"validateWireGuardPeers":{"invalidPeerIps":[]}}}
2022/03/17 16:06:42.634601 srv validated wireguard peers
DEBUG --> POST https://api.fly.io/graphql {{"query":"mutation($input: ValidateWireGuardPeersInput!) { validateWireGuardPeers(input: $input) { invalidPeerIps } }","variables":{"input":{"peerIps":["fdaa:0:1c5d:a7b:1596:0:a:302"]}}}
}
DEBUG <-- 200 https://api.fly.io/graphql (5.34s) {"data":{"validateWireGuardPeers":{"invalidPeerIps":[]}}}
2022/03/17 16:08:48.033650 srv validated wireguard peers
DEBUG --> POST https://api.fly.io/graphql {{"query":"mutation($input: ValidateWireGuardPeersInput!) { validateWireGuardPeers(input: $input) { invalidPeerIps } }","variables":{"input":{"peerIps":["fdaa:0:1c5d:a7b:1596:0:a:302"]}}}
}
DEBUG <-- 200 https://api.fly.io/graphql (5.37s) {"data":{"validateWireGuardPeers":{"invalidPeerIps":[]}}}
2022/03/17 16:10:53.470568 srv validated wireguard peers
DEBUG --> POST https://api.fly.io/graphql {{"query":"mutation($input: ValidateWireGuardPeersInput!) { validateWireGuardPeers(input: $input) { invalidPeerIps } }","variables":{"input":{"peerIps":["fdaa:0:1c5d:a7b:1596:0:a:302"]}}}
}
DEBUG <-- 200 https://api.fly.io/graphql (5.33s) {"data":{"validateWireGuardPeers":{"invalidPeerIps":[]}}}
2022/03/17 16:12:58.869809 srv validated wireguard peers
DEBUG --> POST https://api.fly.io/graphql {{"query":"mutation($input: ValidateWireGuardPeersInput!) { validateWireGuardPeers(input: $input) { invalidPeerIps } }","variables":{"input":{"peerIps":["fdaa:0:1c5d:a7b:1596:0:a:302"]}}}
}
DEBUG <-- 200 https://api.fly.io/graphql (5.35s) {"data":{"validateWireGuardPeers":{"invalidPeerIps":[]}}}
2022/03/17 16:15:04.282059 srv validated wireguard peers
DEBUG --> POST https://api.fly.io/graphql {{"query":"mutation($input: ValidateWireGuardPeersInput!) { validateWireGuardPeers(input: $input) { invalidPeerIps } }","variables":{"input":{"peerIps":["fdaa:0:1c5d:a7b:1596:0:a:302"]}}}
}
DEBUG <-- 200 https://api.fly.io/graphql (5.35s) {"data":{"validateWireGuardPeers":{"invalidPeerIps":[]}}}
2022/03/17 16:17:09.699437 srv validated wireguard peers```

Thanks so much for trying that, @sambird! This validateWireGuardPeers loop is actually normal when the agent is running.
We can actually get more relevant logs if we run that same command,

fly agent stop
LOG_LEVEL=debug fly agent daemon-start

and then run the command that is breaking for you at the same time in a separate terminal, can you try that for me, please?

Sure, thanks Zee!

Here’s the output when I run flyctl doctor in another terminal window:

2022/03/17 17:05:41.151429 #d connected ...
2022/03/17 17:05:41.151528 srv config change at: 2022-03-17 17:05:09.213501696 +0000 WET
2022/03/17 17:05:41.151551 #d <- (    4) "ping"
2022/03/17 17:05:41.151596 #d -> (   57) "7\x00ok {\"PID\":3850,\"Version\":\"0.0.306\",\"Background\":false}\n"
2022/03/17 17:05:41.151616 #d dropped.
2022/03/17 17:05:41.151838 #e connected ...
2022/03/17 17:05:41.151877 #e <- (    4) "ping"
2022/03/17 17:05:41.151906 #e -> (   57) "7\x00ok {\"PID\":3850,\"Version\":\"0.0.306\",\"Background\":false}\n"
2022/03/17 17:05:41.151923 #e dropped.
2022/03/17 17:05:41.152238 #f connected ...
2022/03/17 17:05:41.152269 #f <- (    4) "ping"
2022/03/17 17:05:41.152301 #f -> (   57) "7\x00ok {\"PID\":3850,\"Version\":\"0.0.306\",\"Background\":false}\n"
2022/03/17 17:05:41.152334 #f dropped.
2022/03/17 17:05:41.321500 #10 connected ...
2022/03/17 17:05:41.321557 #10 <- (   18) "establish personal"
DEBUG --> POST https://api.fly.io/graphql {{"query":"query($orgType: OrganizationType) { organizations(type: $orgType) { nodes { id slug name type } } }","variables":null}
}
DEBUG <-- 200 https://api.fly.io/graphql (161.82ms) {"data":{"organizations":{"nodes":[{"id":"2j9m3yRpYo94bH7l7KOeDQoOeaFJxz","slug":"personal","name":"Sam Bird","type":"PERSONAL"}]}}}
2022/03/17 17:05:41.483619 #10 -> (  736) "\xde\x02ok {\"WireGuardState\":{\"org\":\"personal\",\"name\":\"interactive-agent-Samuels-MacBook-Pro-sam-birdd-hotmail-co-uk-246\",\"region\":\"cdg\",\"localprivate\":\"NVgM5UpU8X9oG/JL/Gvb+1ccjVlQ4KqLmo4/1R23TXM=\",\"localpublic\":\"3bRXqL32ZBXY0z4vhsxX4abcPLTocXyLKhkhWJUw3s4=\",\"dns\":\"\",\"peer\":{\"peerip\":\"fdaa:0:1c5d:a7b:1596:0:a:302\",\"endpointip\":\"cdg1.gateway.6pn.dev\",\"pubkey\":\"trM7zOMMKsWHT+F6V08e4e5YVe3VVgf6M8zONd7qzwQ=\"}},\"TunnelConfig\":{\"LocalPrivateKey\":\"3bRXqL32ZBXY0z4vhsxX4abcPLTocXyLKhkhWJUw3s4=\",\"LocalNetwork\":\"fdaa:0:1c5d:a7b:1596:0:a:300/120\",\"RemotePublicKey\":\"trM7zOMMKsWHT+F6V08e4e5YVe3VVgf6M8zONd7qzwQ=\",\"RemoteNetwork\":\"fdaa:0:1c5d::/48\",\"Endpoint\":\"cdg1.gateway.6pn.dev:51820\",\"DNS\":\"fdaa:0:1c5d::3\",\"KeepAlive\":0,\"MTU\":0,\"LogLevel\":0}}\n"
2022/03/17 17:05:41.483657 #10 dropped.
2022/03/17 17:05:41.484067 #11 connected ...
2022/03/17 17:05:41.484080 #12 connected ...
2022/03/17 17:05:41.484128 #11 <- (   18) "establish personal"
DEBUG --> POST https://api.fly.io/graphql {{"query":"query($orgType: OrganizationType) { organizations(type: $orgType) { nodes { id slug name type } } }","variables":null}
}
2022/03/17 17:05:41.484223 #12 <- (   14) "ping6 personal"
DEBUG <-- 200 https://api.fly.io/graphql (157.24ms) {"data":{"organizations":{"nodes":[{"id":"2j9m3yRpYo94bH7l7KOeDQoOeaFJxz","slug":"personal","name":"Sam Bird","type":"PERSONAL"}]}}}
2022/03/17 17:05:41.641660 #11 -> (  736) "\xde\x02ok {\"WireGuardState\":{\"org\":\"personal\",\"name\":\"interactive-agent-Samuels-MacBook-Pro-sam-birdd-hotmail-co-uk-246\",\"region\":\"cdg\",\"localprivate\":\"NVgM5UpU8X9oG/JL/Gvb+1ccjVlQ4KqLmo4/1R23TXM=\",\"localpublic\":\"3bRXqL32ZBXY0z4vhsxX4abcPLTocXyLKhkhWJUw3s4=\",\"dns\":\"\",\"peer\":{\"peerip\":\"fdaa:0:1c5d:a7b:1596:0:a:302\",\"endpointip\":\"cdg1.gateway.6pn.dev\",\"pubkey\":\"trM7zOMMKsWHT+F6V08e4e5YVe3VVgf6M8zONd7qzwQ=\"}},\"TunnelConfig\":{\"LocalPrivateKey\":\"3bRXqL32ZBXY0z4vhsxX4abcPLTocXyLKhkhWJUw3s4=\",\"LocalNetwork\":\"fdaa:0:1c5d:a7b:1596:0:a:300/120\",\"RemotePublicKey\":\"trM7zOMMKsWHT+F6V08e4e5YVe3VVgf6M8zONd7qzwQ=\",\"RemoteNetwork\":\"fdaa:0:1c5d::/48\",\"Endpoint\":\"cdg1.gateway.6pn.dev:51820\",\"DNS\":\"fdaa:0:1c5d::3\",\"KeepAlive\":0,\"MTU\":0,\"LogLevel\":0}}\n"
2022/03/17 17:05:41.641714 #11 dropped.
2022/03/17 17:05:44.668321 #12 dropped.```

And when running flyctl deploy on my app:

022/03/17 17:07:50.689903 #13 connected ...
2022/03/17 17:07:50.689971 srv config change at: 2022-03-17 17:07:50.689690004 +0000 WET
2022/03/17 17:07:50.689984 #13 <- (    4) "ping"
2022/03/17 17:07:50.690016 #13 -> (   57) "7\x00ok {\"PID\":3850,\"Version\":\"0.0.306\",\"Background\":false}\n"
2022/03/17 17:07:50.690044 #13 dropped.
2022/03/17 17:07:50.690230 #14 connected ...
2022/03/17 17:07:50.690260 #14 <- (   18) "establish personal"
DEBUG --> POST https://api.fly.io/graphql {{"query":"query($orgType: OrganizationType) { organizations(type: $orgType) { nodes { id slug name type } } }","variables":null}
}
DEBUG <-- 200 https://api.fly.io/graphql (170.5ms) {"data":{"organizations":{"nodes":[{"id":"2j9m3yRpYo94bH7l7KOeDQoOeaFJxz","slug":"personal","name":"Sam Bird","type":"PERSONAL"}]}}}
2022/03/17 17:07:50.861046 #14 -> (  736) "\xde\x02ok {\"WireGuardState\":{\"org\":\"personal\",\"name\":\"interactive-agent-Samuels-MacBook-Pro-sam-birdd-hotmail-co-uk-246\",\"region\":\"cdg\",\"localprivate\":\"NVgM5UpU8X9oG/JL/Gvb+1ccjVlQ4KqLmo4/1R23TXM=\",\"localpublic\":\"3bRXqL32ZBXY0z4vhsxX4abcPLTocXyLKhkhWJUw3s4=\",\"dns\":\"\",\"peer\":{\"peerip\":\"fdaa:0:1c5d:a7b:1596:0:a:302\",\"endpointip\":\"cdg1.gateway.6pn.dev\",\"pubkey\":\"trM7zOMMKsWHT+F6V08e4e5YVe3VVgf6M8zONd7qzwQ=\"}},\"TunnelConfig\":{\"LocalPrivateKey\":\"3bRXqL32ZBXY0z4vhsxX4abcPLTocXyLKhkhWJUw3s4=\",\"LocalNetwork\":\"fdaa:0:1c5d:a7b:1596:0:a:300/120\",\"RemotePublicKey\":\"trM7zOMMKsWHT+F6V08e4e5YVe3VVgf6M8zONd7qzwQ=\",\"RemoteNetwork\":\"fdaa:0:1c5d::/48\",\"Endpoint\":\"cdg1.gateway.6pn.dev:51820\",\"DNS\":\"fdaa:0:1c5d::3\",\"KeepAlive\":0,\"MTU\":0,\"LogLevel\":0}}\n"
2022/03/17 17:07:50.861098 #14 dropped.
2022/03/17 17:07:50.861728 #15 connected ...
2022/03/17 17:07:50.861813 #15 <- (   14) "probe personal"
2022/03/17 17:07:50.861828 srv probing "personal" ...
2022/03/17 17:07:55.862253 #15 -> (   58) "8\x00err failed probing \"personal\": context deadline exceeded"
2022/03/17 17:07:55.862360 #15 dropped.```

Hey, there. I’ve deleted this peer, since the log you pasted included private key material (this isn’t on you; that log shouldn’t happen); you should be able to create another one easily; the simplest thing would be to make sure you’re on the most current flyctl (flyctl version update) and then flyctl wg reset.

Just a quick note that from the original fly doctor output upthread, it looks like you might have a network connectivity issue; your host might not be able to reach our gateways on that UDP port. Are you behind a corporate firewall, or on a VPN already?

2 Likes

Sure thanks for deleting and I should have checked the key info before posting, my bad!

I have disconnected from my company VPN, and this worked historically. Only difference is I’m on holiday using hotel wifi, maybe that’s prohibiting it?

Do you have a netcat command or similar that I can use for testing and the host / port number of the servers I should be pinging?

Thanks and regards,
Sam

The annoying thing about WireGuard is that it goes way, way out of its way not to be “pingable”; it’s quiet unless the handshake succeeds, which in your case it isn’t.

Here’s a thing you can try if you’re adventurous:

This is an (older) release of flyctl that runs WireGuard over WebSockets, using a TCP connection to port 443 on our gateway (you can just try telnettting to that port if you like to make sure you can contact it). We keep debating merging this into our mainline flyctl but this might be the first time someone has been on a network messed up enough to need it. :expressionless:

The log message thing is 100% our fault, not yours!

I’ll give this a go and come back, hoping it would be easier than this… I’ll try turning the wifi on and off again first :joy:

It should be easier than this! If the WebSockets WireGuard branch works for you, I’ll merge it into mainline right away.

Just checking in and/or providing moral support while you burn vacation time debugging WireGuard networking. :slight_smile: If you’ve tried something and it didn’t work, feel free to let me know!

Hi Thomas,

Thanks for your support. I haven’t had chance to check as yet but will do suggestions later today and respond back. Thank you :pray:

Wireguard over websockets support is live in the latest release of flyctl, v0.0.320. To enable it, grab the release and run:

flyctl wireguard websockets enable

Then try deploying again.