Wireguard peering not working

1

Hi folks,

I’m trying to put together a simple nginx proxy running on fly that will proxy to a wireguard peer. I have the peer all set up and it can see and resolve it’s own _peer.internal DNS name.

When I try to proxy to it from nginx, though, I see dockerpod1._peer.internal could not be resolved (3: Host not found). This is what the nginx config looks like:

server {
  listen 80;
  listen [::]:80;

  server_tokens off;

  server_name proxy-test.petekeen.net;

  gzip off;

  location / {
    resolver [fdaa::3]:53;

    proxy_http_version 1.1;

    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header X-Forwarded-Proto $scheme;

    proxy_set_header Host www.petekeen.net;
    proxy_set_header X-NginX-Proxy true;
    proxy_redirect off;
    # Enables WS support
    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection "upgrade";

    set $proxy_backend dockerpod1._peer.internal;
    proxy_pass http://$proxy_backend; 
  }
}

And here’s the full log with boot and one request:

2020-12-29T20:31:38.715Z 46986b20 ord [info] Starting instance
2020-12-29T20:31:38.751Z 46986b20 ord [info] Configuring virtual machine
2020-12-29T20:31:38.752Z 46986b20 ord [info] Pulling container image
2020-12-29T20:31:40.094Z 46986b20 ord [info] Unpacking image
2020-12-29T20:31:40.494Z 46986b20 ord [info] Preparing kernel init
2020-12-29T20:31:41.053Z 46986b20 ord [info] Configuring firecracker
2020-12-29T20:31:41.086Z 46986b20 ord [info] Starting virtual machine
2020-12-29T20:31:41.181Z 46986b20 ord [info] Virtual machine started successfully
2020-12-29T20:31:41.259Z 46986b20 ord [info] Starting init (commit: 3b4ae0c)...
2020-12-29T20:31:41.268Z 46986b20 ord [info] Running: `/docker-entrypoint.sh nginx -g daemon off;` as root
2020-12-29T20:31:41.280Z 46986b20 ord [info] /docker-entrypoint.sh: /docker-entrypoint.d/ is not empty, will attempt to perform configuration
2020-12-29T20:31:41.281Z 46986b20 ord [info] /docker-entrypoint.sh: Looking for shell scripts in /docker-entrypoint.d/
2020-12-29T20:31:41.284Z 46986b20 ord [info] /docker-entrypoint.sh: Launching /docker-entrypoint.d/10-listen-on-ipv6-by-default.sh
2020-12-29T20:31:41.288Z 46986b20 ord [info] 10-listen-on-ipv6-by-default.sh: info: Getting the checksum of /etc/nginx/conf.d/default.conf
2020-12-29T20:31:41.316Z 46986b20 ord [info] 10-listen-on-ipv6-by-default.sh: info: Enabled listen on IPv6 in /etc/nginx/conf.d/default.conf
2020-12-29T20:31:41.318Z 46986b20 ord [info] /docker-entrypoint.sh: Launching /docker-entrypoint.d/20-envsubst-on-templates.sh
2020-12-29T20:31:41.321Z 46986b20 ord [info] /docker-entrypoint.sh: Configuration complete; ready for start up
2020-12-29T20:31:59.655Z 9b38598e ord [info] 93.187.218.225 - - [29/Dec/2020:20:31:59 +0000] "GET / HTTP/1.1" 200 612 "-" "curl/7.54.0" "2601:401:100:ed50:e4c2:e66:28d3:e7c7, 2a09:8280:1:8968:d249:6201:cd71:c56b"
2020-12-29T20:32:01.734Z 9b38598e ord [info] Shutting down virtual machine
2020-12-29T20:32:01.790Z 9b38598e ord [info] Program exited with code: 0
2020-12-29T20:32:12.860Z 46986b20 ord [info] 2020/12/29 20:32:12 [error] 525#525: *4 dockerpod1._peer.internal could not be resolved (3: Host not found), client: 93.187.218.219, server: proxy-test.petekeen.net, request: "GET / HTTP/1.1", host: "proxy-test.petekeen.net"
2020-12-29T20:32:12.865Z 46986b20 ord [info] 93.187.218.219 - - [29/Dec/2020:20:32:12 +0000] "GET / HTTP/1.1" 502 150 "-" "curl/7.54.0" "2601:401:100:ed50:e4c2:e66:28d3:e7c7, 2a09:8280:1:8968:d249:6201:cd71:c56b"

I’m not really sure what’s going on. Anyone have any guesses or things to try? The v4 ip in the client line is somewhat suspicious.

2

That’s super strange. nginx resolution confuses the heck out of me, maybe try setting resolver [fdaa::3]:53 ipv6=on and see if it helps?

Your peer IP won’t actually change, so you could always just set $proxy_backend "[fdaa::...]" and see if that works? Your VM is definitely resolving properly so I’m pretty sure there’s just some nginx voodoo we’re missing.

3

Thanks! The resolver setting didn’t help. Setting the backed to an explicit IP of course worked, but that’s not very satisfactory. That said I’m unblocked for now so I’ll keep going and then circle back to this at some point.

4

I ran into the same issue with my project[1]. I could never get Nginx’s resolver directive to work. However, removing it and changing the container’s resolver instead by editing fly.toml worked for me:

[experimental]
  private_network = "true"

[1] LukeLambert/fly-dev-tunnel on Github

2 Likes
5

It might just be nginx failing to resolve an AAAA only record. I’ll do some testing later, you might get a kick out of the DNS shenanigans we’ve done in the CDN example we’re working on: https://github.com/fly-examples/nginx-cluster

1 Like
6

I’m also going through this right now, wondering if anyone found a solution?

In my case, I followed these steps:

  • flyct wireguard create → generated a peer called foo (in the atl region, if it matters)
  • also created a second peer config and downloaded it on my Mac. Connecting with it using the WireGuard app
  • installed wireguard/etc in a host on hetzner
  • from within the hetzner host, I can ping6 both my macbook, the host itself (foo._peer.internal) and .internal
  • from my Mac, I can also ping6 all parties - foo._peer.internal, .internal
  • from within the app running in Fly, I cannot ping or resolve either foo._peer.internal or my Mac
  • the app has the same private_networking flag set to true (also tried setting it to “true”, with quotes). Also tried without the experimental section. Doesn’t seem to make a difference.

If it adds any context, my fly container is based on a Docker image (not the default buildpacks)

7

This sounds like the app might be in a different organization than your peers. You can check this by running:

fly wireguard list

And then:

fly ips private

The first three chunks of the IP should be the same for the wireguard peer and the app’s private IPs.