Can I use fly + WireGuard to expose an internal web interface?

tinypilot · April 4, 2021, 7:43pm

I’m looking for an easy way to expose internal access for an internal web interface. I feel like I’m close with fly.io, but I’m not sure if it’s actually the right tool.

I have an internal service that exposes a web interface. I want to be able to pop up a public URL so I can access the internal server outside of my local network.

I followed the steps in the Private Networking docs to create a wireguard tunnel and add my internal server as a peer. Then, I created a Docker image for an nginx reverse proxy that forwarded HTTP requests to the peer’s hostname, but when I I deploy it, loading it in the browser just says “connection reset.”

I tried referring to the internal server as both my-servers-hostname and my-servers-hostname.internal, but neither worked. If I deploy my nginx server to fly, I can ping it from the internal server that’s peered over WireGuard, but I’m not sure if the fly.io VM is seeing the peer.

Here are my fly, Docker, and nginx config files

Is what I’m doing possible with fly? Or am I trying to force a square peg in a round hole?

kurt · April 4, 2021, 10:01pm

This should work fine, it’s an intended future use case. The wireguard peer hostnames are a bit different though (and not actually documented). Try using <wireguard-peer>._peer.internal and see if you have better luck? I just updated the docs you linked with that info.

tinypilot · April 4, 2021, 10:22pm

Hmm, it doesn’t seem to be resolving that one either. I’m not sure if it’s a fly issue or a problem with my nginx config. I see this in the nginx logs:

2021-04-04T22:20:00.552Z 7e931bdc ewr [info] 2021/04/04 22:20:00 [error] 535#535: *7 tinypilot._peer.internal could not be resolved (110: Operation timed out), client: [redacted], server: , request: "GET / HTTP/1.1", host: "nginx-scratch.fly.dev", referrer: "https://fly.io/"
2021-04-04T22:20:00.554Z 7e931bdc ewr [info] [redacted] - - [04/Apr/2021:22:20:00 +0000] "GET / HTTP/1.1" 502 559 "https://fly.io/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/89.0.4389.114 Safari/537.36"

But I can see the peer if I list wireguard nodes:

$ flyctl wireguard list
Automatically selected personal organization: Michael Lynch
+-----------+--------+---------------------------+
|   NAME    | REGION |          PEER IP          |
+-----------+--------+---------------------------+
| tinypilot | iad    | [redacted] |
+-----------+--------+---------------------------+

kurt · April 4, 2021, 10:27pm

Oh! This might actually be nginx using a custom resolver that’s a problem. Will you try setting resolver [fdaa::3] in your nginx config?

tinypilot · April 5, 2021, 2:24am

Oh, interesting. I tried that, but I get the same result:

2021-04-05T02:22:12.349Z 7717e59d ewr [info] 2021/04/05 02:22:12 [error] 542#542: *12 tinypilot._peer.internal could not be resolved (3: Host not found), client: [redacted], server: , request: "GET /favicon.ico HTTP/1.1", host: "nginx-scratch2.fly.dev", referrer: "https://nginx-scratch2.fly.dev/"

Interestingly, I can ping the internal host from my nginx instance on fly.io, but nginx can’t seem to resolve it for some reason:

2021-04-05T02:19:47.762Z 7717e59d ewr [info] + ping -c 3 tinypilot._peer.internal
2021-04-05T02:19:48.015Z 7717e59d ewr [info] PING tinypilot._peer.internal(fdaa:[redacted])) 56 data bytes
2021-04-05T02:19:48.016Z 7717e59d ewr [info] 64 bytes from fdaa:[redacted]): icmp_seq=1 ttl=62 time=22.6 ms
2021-04-05T02:19:48.914Z 7717e59d ewr [info] 64 bytes from fdaa:[redacted]): icmp_seq=2 ttl=62 time=22.4 ms
2021-04-05T02:19:49.917Z 7717e59d ewr [info] --- tinypilot._peer.internal ping statistics ---
2021-04-05T02:19:49.917Z 7717e59d ewr [info] 64 bytes from fdaa:[redacted]): icmp_seq=3 ttl=62 time=23.7 ms

I tried HAProxy instead (no experience with it, so I’m maybe getting it wrong), but I get a similar result:

2021-04-05T02:16:26.703Z a5277e8b ewr [info] [WARNING] 094/021626 (513) : Server http_back/tinypilot is DOWN, reason: Layer4 connection problem, info: "Connection refused", check duration: 23ms. 0 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.

Oddly, if I just run nc -l 80 on the target machine, I don’t see any attempt at connection from HAProxy, so I don’t know where the failure is.

kurt · April 5, 2021, 1:38pm

Can you curl it? nginx might be lying, that “connection refused” error from HAProxy makes me wonder if port 80 isn’t responding on the peer IPv6 address.

Depending on your netcat version, you may need to nc -l -6 80 to make it listen on IPv6.

tinypilot · April 5, 2021, 1:07pm

Can you curl it? nginx might be lying, that “connection refused” error from HAProxy makes me wonder if port 80 isn’t responding on the peer IPv6 address.

Oh, that was it. Thanks! The peer backend uses nginx, and I didn’t realize that nginx doesn’t listen on IPv6 by default. I updated it to accept IPv6 and used HAProxy as a TCP proxy, and now it’s working:

Latency is super low, so the performance is fantastic. I’ve tried other solutions for this, and they add a ton of lag between client mouse movement and mouse movement on the remote machine (it’s a KVM over IP solution).

Do you have recommendations for doing this securely? I want the end-user to have the convenience of visiting the URL in a browser without configuring their machine for Wireguard, but I don’t want it to be available to any random user who hits the URL.

Some ideas:

Generate long, random app names so that the app subdomains are unguessable (feels a bit hacky, DNS names might be discoverable through other means, and the IP address is still guessable)
Add some authentication at the HAProxy level (not sure how much flexibility is available if it’s just a dumb TCP proxy)
Rely on app-level security (the app has username/password auth, but it’s meant more for keeping honest co-workers honest rather than defending against brute force attacks from the open Internet)

kurt · April 5, 2021, 1:30pm

Ok that’s very cool. So that’s TinyPilot on a raspberry pi behind nginx? Is nginx on the same host?

I would add auth in nginx/haproxy, on either end of wireguard, and configure the proxy to force people to https. Most scans on your IPs won’t know what hostname to send with SNI, so redirecting requests to https will effectively “block” their requests and prevent weird traffic from hitting the KVM software.

Can the kvm port be exposed directly on the wireguard peer IP? If it can, you could just run nginx on Fly and skip one of the layers.

Either way, I would suggest people configure wireguard later on. Exposing this on the public internet is great for convenience, but people seem to like seeing “here’s what to do next to make this more secure”.

tinypilot · April 5, 2021, 1:46pm

So that’s TinyPilot on a raspberry pi behind nginx? Is nginx on the same host?

Yep, correct. nginx is on the same host.

I would add auth in nginx/haproxy, on either end of wireguard, and configure the proxy to force people to https. Most scans on your IPs won’t know what hostname to send with SNI, so redirecting requests to https will effectively “block” their requests and prevent weird traffic from hitting the KVM software.

Okay, interesting. I’ll keep messing around with this.

Can the kvm port be exposed directly on the wireguard peer IP? If it can, you could just run nginx on Fly and skip one of the layers.

Oh, that’s a good point.

I theoretically could, but that’s a bit more difficult of a setup. Chaining proxies right now is nice because it essentially lets the user take their existing system and layer on an extra proxy without changing any local configuration (except for peering with WireGuard).

Moving the proxy to fly.io means that the internal backends currently behind nginx would have to listen to the wireguard interface when they’re currently configured just to listen on localhost. Plus, my nginx config isn’t super complicated, but it’s complicated enough that I’d prefer to avoid having to maintain a separate local version and a fly.io version.