Incoming! Private Networks and WireGuard VPNs

1

Hello, Flyfriends!

We’ve got a flurry of new fleatures arriving at Fly and I’m taking a minute to talk about two of them.

The first is Private Networking. Enable it in your fly.toml:

[experimental]
  private_network=true

We’ll add your application to a private IPv6 network (we call them 6PNs — or at least I do, and I’m trying to make it a thing) for your organization, and link instances of that application to DNS so you can find them.

Every app running on Fly now has an additional IP address — fly-local-6pn — that has connectivity only within its private network. If you want to run a private service, like Serf or NATS or Pgbouncer, bind it to fly-local-6pn and only apps in your organization will be able to reach it.

We do some rudimentary DNS. With 6PN enabled, an app called fearsome-bagel-43 can do a TXT query of regions.fearsome-bagel-43.internal to discover what regions it’s deployed on:

$ dig -t TXT fearsome-bagel-43.internal
... 
;; ANSWER SECTION:
regions.fearsome-bagel-43.internal. 1500 IN TXT	"ord,nrt"

Since it’s deployed on nrt,ord, you can use nrt.fearsome-bagel-43.internal as a DNS name and it should just work. global.fearsome-bagel-43.internal selects from all regions.

To make this work, we inject the nameserver fdaa::3 into your resolv.conf (that’s what the [experimental] option I mentioned triggers). But you can just talk to fdaa::3 directly if you want, or use dnsmasq, or Golang’s DNS libraries, or whatever. Importantly: fdaa::3 recurses, to 8.8.8.8, for non-.internal names, so it works as your app’s primary nameserver as well.

You should be able to run any protocol you like between your applications on your 6PN — again, the 6PN is private to your organization and spans all the apps in that organization — and we’ll be posting examples of what we’ve been doing with it internally.

The other feature I want to talk about is WireGuard Peering. You can now link your private network to other networks using WireGuard.

To do that, update flyctl, and then run flyctl wireguard create. Select the organization to peer to, and the region you want WireGuard to connect to (note: right now, the only region you can use is “dev”, but that’ll change). Give the peering link a name. We’ll generate a WireGuard configuration for you, which will work with the macOS WireGuard.app, Windows WireGuard, and Linux wg-quick.

Your WireGuard peering connection has the same connectivity apps in your organization do, and we give them a local DNS server that sees the same names your apps do. You can see the IPv6 address of the local nameserver in the WireGuard config we generate for you.

You can use WireGuard peering to manage and talk to your apps directly from your laptop, and to bridge it to third-party services.

I’m a little biased but I’m psyched about how useful WireGuard peering turns out to be. Because it’s WireGuard and not OpenVPN or IPSEC, it pretty much just always works. I open my laptop, I close my laptop, I go make a sandwich, whatever, I’m always connected to the apps in my organization, as if they were just on my local IPv6 network. I hope you all find interesting things to do with it and interesting bugs for me to hunt down.

4 Likes
2

This is amazing. Do you have any writeups about how the Wireguard setup happens between my laptop and the Fly services? How does the key exchange happen, is there a bastion, does it connect to each app instance directly or is there a gateway of some sort, etc. Would love to read about that.

3

Awesome stuff! Will there be support for peering networks between two or more Fly.io apps?

E.g. Have an API as one app and a database as another app, peer them together, then have the API app communicate with the database app all inside the private network without needing to touch the public Internet (ignoring the fact that the wire guard networks use public backhaul).

4

Also, regarding cost, how will bandwidth in the private network be billed?

Will data that stays in a single region be billed differently?

I can see quite a few potential advantages of using the private network if there is a reduced cost to bandwidth used inside a region vs between regions.

5

There will be! You can see a good chunk of it in the flyctl source code. Wireguard is pretty simple, which is part of why we love it.

Right now, all apps in an organization are in the same private network. So you can create an app called my-database and then connect to it from my-frontend by pointing at global.my-database.internal.

We’re going to add support for multiple networks later on. This will let you isolate groups of apps within the same organization.

It’s billed the same as public bandwidth right now. Once we get bandwidth accounting tweaked properly, we’ll see what we can do to make internal bandwidth cheaper.

6

I’m working on a blog post about how we built out the private networking stuff that’ll go into detail, but in the immediacy, I can say:

(a) flyctl generates the WireGuard Curve25519 keys locally (I did it the other way around and Jason Donenfeld found the code and yelled at me about it) and sends the pubkey to our API, and

(b) we terminate WireGuard connections in dedicated WireGuard “gateway” servers, and use BPF programs and WireGuard’s cryptokey routing to lock each connection to its organization private network — the only thing a WireGuard peer can talk to are other 6PN addresses in its own organization.

Routing traffic through a WireGuard peering connection should be just as performant as routing any traffic through Fly, if not a little faster, since nothing but our private DNS ever leaves the kernel on its way to your VM.

7

Is this expected to work with bare minimal containers created from scratch image?
No /etc/resolv.conf or /etc/hosts exist so I get error when trying to bind to fly-local-6pn:
failed to lookup address information: Name or service not known

8

fly-local-6pn is populated by our loader in /etc/hosts, but if you don’t have a resolv.conf, Unix tools by default won’t be able to use the .internal names we use; whatever thing you use to configure DNS, it should point to fdaa::3.

Some pretty-standard Unix software won’t bind to hostnames, only addresses, which is annoying; we tend to use tiny shell scripts to resolve the address with grep and pass it to the server, or sed it into the config. Which, I admit, is not “bare minimal”.

9

Is it described somewhere what the loader expects to be present in the container like /etc/resolv.conf, /etc/hosts/, grep, sed or anything else. Based on that setup a minimal image could then be created.

10

No, we should do a better job of documenting this (it’s in a bit of flux, is part of the reason we don’t, and it’s subject to change in the future). But: you shouldn’t need anything special to enable 6PN networking! You don’t even need the /etc/hosts entry. Our init will set up a 6PN address (and routing) for you regardless of how your container is configured.

So ultimately, the root of configuration you’d need to work with this is:

  • The fdaa: address on eth0 — whatever tooling you use to retrieve IP addresses (iproute2, or your own netlink code)

  • The nameserver fdaa::3 — the same for every host — configured into your application.

You shouldn’t need to do anything to get the /etc/hosts file we use, by the way: regardless of your container layout, we’re going to splat our own addresses into it.

11

What about to expose the address as FLY_LOCAL_6PN env var?

Is it already done or going to happen? In case of containers based on scratch image there’s no /etc/hosts or awk nor grep by default so no easy way to bind to 6PN.

1 Like
12

I believe we already do expose the address in the environment, but need to check what that is.

As for /etc/hosts — we do that now! The general sequence of events is:

  1. A container image is built, either on your side and pushed to our container repository, or on ours.

  2. When an instance is deployed, on demand, we convert the layers of the container image to a rootfs for Firecracker.

  3. We then inject things into that rootfs — /etc/hosts and /etc/resolv.conf included.

  4. We start the VM.

  5. The VM runs our init, which sets up interfaces.

  6. init calls your entrypoint.

(Steps 3 and 5 might have been swapped lately, because Jerome is tinkering with how we build rootfs’s to make VMs boot faster, but the effect is the same).

I await Jerome’s corrections. :slight_smile:

1 Like
13

That was a swift reply, thanks:)
I can see a few cryptic messages in my boot log, e.g.

Program exited with signal: SIGINT (2)
Starting init (commit: 1a7da15)...
error getting user 'root' by name => ENOENT: No such file or directory

Regarding env, this is what I see from the VM:

cgroup_enable: "memory",
FLY_VM_MEMORY_MB: "512",
FLY_PUBLIC_IP: "fdaa:0:...",
FLY_APP_NAME: "...",
FLY_ALLOC_ID: "...",
FLY_REGION: "fra",
FLY_REDIS_CACHE_URL: "..."
PATH: "/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
TERM: "linux"
14

OK, I found the missing piece: /etc/nsswitch.conf. Having it set up for the distroless image, I can now bind to fly-local-6pn. Now my server starts but the health check is failing.
I have the default services.tcp_checks setting in my config. Anything I should adjust here? Thanks!

1 Like
15

A service bound to fly-local-6pn can’t be health-checked (it’s not exposed to our public routing fabric), so if you have nothing but the private service, comment out the TCP health check from your config and you should be set. Most of my private service apps don’t have TCP health checks at all.

(You can of course bind something health-checkable to 0.0.0.0).

16

It works, thanks. I wondered if this is the preferred way.
Now I’m trying to reach the running 6pn service via wireguard but it seems name resolution doesn’t work. There’s no public nor private net access If I turn on the tunnel I created via fly wireguard create. dig timeouts as well. When I tried yesterday it worked though.

17

Weird! Try again, it should be working?

18

Yep, it works now! Thanks!

19

Excellent. If you’re in a mood to share: what’cha building?

20

Do you plan to offer Private Network VPN (wireguard) in some European regions like ams, fra and cdg?

Currently, when trying to deploy one on those regions I get this error:

Error add peer failed: no gateways selected region

But choosing iad works, so I think it’s just that it’s not available for every region.