Incoming! Private Networks and WireGuard VPNs

Is this expected to work with bare minimal containers created from scratch image?
No /etc/resolv.conf or /etc/hosts exist so I get error when trying to bind to fly-local-6pn:
failed to lookup address information: Name or service not known

fly-local-6pn is populated by our loader in /etc/hosts, but if you don’t have a resolv.conf, Unix tools by default won’t be able to use the .internal names we use; whatever thing you use to configure DNS, it should point to fdaa::3.

Some pretty-standard Unix software won’t bind to hostnames, only addresses, which is annoying; we tend to use tiny shell scripts to resolve the address with grep and pass it to the server, or sed it into the config. Which, I admit, is not “bare minimal”.

Is it described somewhere what the loader expects to be present in the container like /etc/resolv.conf, /etc/hosts/, grep, sed or anything else. Based on that setup a minimal image could then be created.

No, we should do a better job of documenting this (it’s in a bit of flux, is part of the reason we don’t, and it’s subject to change in the future). But: you shouldn’t need anything special to enable 6PN networking! You don’t even need the /etc/hosts entry. Our init will set up a 6PN address (and routing) for you regardless of how your container is configured.

So ultimately, the root of configuration you’d need to work with this is:

  • The fdaa: address on eth0 — whatever tooling you use to retrieve IP addresses (iproute2, or your own netlink code)

  • The nameserver fdaa::3 — the same for every host — configured into your application.

You shouldn’t need to do anything to get the /etc/hosts file we use, by the way: regardless of your container layout, we’re going to splat our own addresses into it.

What about to expose the address as FLY_LOCAL_6PN env var?

Is it already done or going to happen? In case of containers based on scratch image there’s no /etc/hosts or awk nor grep by default so no easy way to bind to 6PN.

I believe we already do expose the address in the environment, but need to check what that is.

As for /etc/hosts — we do that now! The general sequence of events is:

  1. A container image is built, either on your side and pushed to our container repository, or on ours.

  2. When an instance is deployed, on demand, we convert the layers of the container image to a rootfs for Firecracker.

  3. We then inject things into that rootfs — /etc/hosts and /etc/resolv.conf included.

  4. We start the VM.

  5. The VM runs our init, which sets up interfaces.

  6. init calls your entrypoint.

(Steps 3 and 5 might have been swapped lately, because Jerome is tinkering with how we build rootfs’s to make VMs boot faster, but the effect is the same).

I await Jerome’s corrections. :slight_smile:

That was a swift reply, thanks:)
I can see a few cryptic messages in my boot log, e.g.

Program exited with signal: SIGINT (2)
Starting init (commit: 1a7da15)...
error getting user 'root' by name => ENOENT: No such file or directory

Regarding env, this is what I see from the VM:

cgroup_enable: "memory",
FLY_VM_MEMORY_MB: "512",
FLY_PUBLIC_IP: "fdaa:0:...",
FLY_APP_NAME: "...",
FLY_ALLOC_ID: "...",
FLY_REGION: "fra",
FLY_REDIS_CACHE_URL: "..."
PATH: "/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
TERM: "linux"

OK, I found the missing piece: /etc/nsswitch.conf. Having it set up for the distroless image, I can now bind to fly-local-6pn. Now my server starts but the health check is failing.
I have the default services.tcp_checks setting in my config. Anything I should adjust here? Thanks!

A service bound to fly-local-6pn can’t be health-checked (it’s not exposed to our public routing fabric), so if you have nothing but the private service, comment out the TCP health check from your config and you should be set. Most of my private service apps don’t have TCP health checks at all.

(You can of course bind something health-checkable to 0.0.0.0).

It works, thanks. I wondered if this is the preferred way.
Now I’m trying to reach the running 6pn service via wireguard but it seems name resolution doesn’t work. There’s no public nor private net access If I turn on the tunnel I created via fly wireguard create. dig timeouts as well. When I tried yesterday it worked though.

Weird! Try again, it should be working?

Yep, it works now! Thanks!

Excellent. If you’re in a mood to share: what’cha building?

Do you plan to offer Private Network VPN (wireguard) in some European regions like ams, fra and cdg?

Currently, when trying to deploy one on those regions I get this error:

Error add peer failed: no gateways selected region

But choosing iad works, so I think it’s just that it’s not available for every region.

There is a gateway in London. You can see all the regions + gateways with flyctl platform regions.

We’ll have gateway in all regions over the next few months, too.

Is it possible to make all the traffic from my workstation go through the vpn?

Nope, you’ll only be able to reach addresses on your private network through the WireGuard gateways we generate; we did that not because we have strong opinions about VPNs but because it provides a simple strict security rule for locking down traffic into our network.

Ok I’m not a big network nerd(though I would love 2) but I like to try weird stuff. I want to deploy an HAProxy app on @flydotio as a LoadBalancer to my theoretical homelab nomad cluster. Given this development with wireguard 6PN(as you like to call it​:grinning_face_with_smiling_eyes:) how would you suggest I proceed. My theoretical cluster runs consul/HAProxy (fabio) for internal service discovery. How do I get traffic from the internet facing flyio app to my blog at “consul.service.blog”? Does this whole premise even have a leg to stand on or should I choose another career(:grinning_face_with_smiling_eyes:imposter syndrome kicking in)

Yes this works! I have a similar setup (though not with nomad). It’s going to take some tweaking, though.

The big problem you’re going to have is getting the allocations in Nomad exposed on a Fly 6PN IP. Your best bet might be to run HAProxy on your nomad cluster, then a really slim haproxy on Fly that just forwards requests to it.

What you’ll need is a WireGuard gateway configured on your nomad node. flyctl wireguard create will generate this config for you. This is what my config ended up looking like:

kurt@nuc:~$ cat /etc/wireguard/wg0.conf
[Interface]
PrivateKey = <private-key>
Address = fdaa:0:5f4:a7b:bea:0:a:2/120
DNS = fdaa:0:5f4::3

[Peer]
PublicKey = RKAYPljEJiuaELNDdQIEJmQienT9+LRISfIHwH45HAw=
AllowedIPs = fdaa:0:5f4::/48
Endpoint = ord1.gateway.6pn.dev:51820
PersistentKeepalive = 5

Start that interface with sudo wg-quick up wg0.

The Address in there is what you’ll need to forward requests to. You can configure your Fly haproxy to connect to it with something like this:

backend blog-backend
  balance leastconn
  server nomad nomad._peer.internal:80 check inter 1000
1 Like

This confirms what I had found during my research this afternoon. I was just reading this: IPv6 WireGuard Peering · Fly, Load Balancing with HAProxy Service Discovery Integration | Consul - HashiCorp Learn. And it appears that haproxy is my best bet. The only difference with my own theory is I was planning to use fly’s internal DNS as the resolver since every time you add a wireguard peer the DNS server picks it up. But this suggestion does seem better. Thanks. What’s your setup like anyway? k3s?