WireGuard Token API

1

We’ve been talking to people about how to wire up K8s clusters, so they can use Fly to front-end their applications by just hosting haproxy on us and bouncing requests to their own network.

There’s a bunch of different ways to do that, and we’re exploring all of them. But one simple approach you can take is to simply have every one of your K8s nodes bring up an independent connection to Fly, for instance via a DaemonSet.

So we’ve tried to make that easy to do with a new API for building WireGuard connections.

The thing we think you don’t want to do is to load a full-fledged Fly access token into every one of your nodes (ick!). So we’ve created a limited-purpose WireGuard token that can only be used to add new WireGuard connections to an organization (or to rekey them). You mint a token (or several, however you want to carve it up; when you delete a token, you delete every associated WireGuard connection), and hand it out to your Pod or whatever. The token case be used — via a REST interface you can drive from curl — to create or update a named WireGuard connection.

It looks like this:

curl -v 
    -H "Authorization: Bearer ${FLY_WIREGUARD_TOKEN}" 
    -H "Content-Type: application/json" 
    --request POST 
    --data '{"name": "node-1", "group": "k8s", "pubkey": "<c25519>", "region": "dev"}' https://fly.io/api/v3/wire_guard_peers

The PUT to v3/wire_guard_peers/:name will accept “pubkey” to re-key a connection (in other words: if you’re replacing a Pod or a container under the same name, your new instance can reclaim the connection.)

You may have noticed the group in the JSON: it can be whatever, and we’ll populate <groupname>._g.internal with the peers under it in our DNS.

<c25519> needs to be a valid Curve25519 public key, a WireGuard key. You can generate them with wg's tooling, or with flyctl.

flyctl can drive this API, with flyctl token create, and then flyctl token start to create a new connection and flyctl token update to rekey. start and update expect the token in the env var FLY_WIREGUARD_TOKEN.

This is a super specific and corner-casey thing that might not be relevant for most of you, but it might make some system design tasks simpler for some of you. Thoughts welcome!

2 Likes
2

How about a sidecar container that can be started alongside the main container. It would configure the wireguard connection etc., similar to how service meshes do it. If multiple pods use the same group name, they’d get the same internal domain name so that HA proxy can round robin all pods. Would it be possible to use a custom network name for this? From a security and simplicity point of view, it would be nice if you could create a token for a specific internal group DN and then all sidecars can use a simple flyctl CMD to connect. All they could do with this token is connect to this one DN and join the group, nothing else.

3

The other day I wad discussing a similar setup with Kurt. And the basic idea was to run a system job on every new node and have it create a wireguard connection to which HAProxy will push all incoming connections. Does this development mean I could automate the creation of a wg connection for new nodes?

4

Yes, that’s the whole intent behind the API, so if it doesn’t work well for that let me know and I’ll take it back to the drawing board.

The big thing we wanted to do was make it so that cluster nodes could bring up their own WireGuard connections without having to have a Fly API token, which seems like an untenable risk.

5

I see. I only started going deep into wireguard networking recently so I’m wondering how one would achieve this. Would the wireguard client be container/task running alongside the cluster’s reverse proxy(HAProxy)?

6

I think the idea Kurt had was that there’d be a pod in each cluster node that’d be responsible for keeping a WireGuard connection for the node alive, but you could do it more than one way. (Also, tagging @kurt).

1 Like
7

Yes exactly! In Nomad I’d make a system job using raw_exec, and then make sure your services are listening on all IPv6 addresses. Someday we’ll have a demo of this. Or, you know, someone from the community will! :smiley:

1 Like