6PN addressing clarification

1

I really like the private networking offering. I was reading the blog posts going into the details, and I was intrigued by how Fly uses IPv6 addresses for metadata to break down region, network, instance.

From the breakdown of the address metadata, it sounds like all Fly VMs are in a shared ipv6 ULA space? I sort of expected there to be an overlay network, so each user lives in their own ULA. Isn’t it riskier to do this without an overlay since bugs in routing/firewall could leak guest packets? Maybe some BPF magic that helps here?

Seems like ipv6 enables you to do some cool routing/firewall tricks :slight_smile:

Hope I’m not asking too much about the magic!

2

@thomas can give you more details here, but:

Our ipv6 is basically an overlay network, just a very simple one. Each organization gets it’s own network ipv6 prefix. This is one of my wireguard peer IPs:

fdaa:0:863:a7b:bea:0:a:2/120

The fdaa:0:863:: is my prefix, we only route traffic from other fdaa:0:863:: IPs. fdaa:0:864:: is an entirely different, isolated network.

The actual network this runs across is a wireguard mesh. The “tricks” are all related to routing IPv6. We have a different IP space we use for this, it just so happens to share components.

:a7b:bea:

This uniquely identifies a physical host in our infra (kind of like a router, I guess). We route all of fdaa:a7b:bea:: across wireguard to the same host – it’s one of the AllowedIPs in our core wireguard config. We do this by rewriting the 6pn address:

fdaa:0:863:a7b:bea:0:a:2 -> fdaa:a7b:bea:0:863:0:a:2

Then do the reverse when it hits the host router.

It’s amazingly simple, which is good! There’s very little that can go wrong with our specific logic. We don’t route customer 6pn networks at all, and the rewritten version is always different than 6pn prefixes. The most “brittle” part is restricting what gets routed to which VM, but it’s (again) pretty simple because we can just match prefixes.

2 Likes
3

At the level we’re doing this at, I think the bug where we accidentally put instances in the wrong ULA and thus expose organizations to each other is about as likely as the bug where we forget that fdaa:xxxx:: can’t talk to fdaa:yyyy:: for any value of x and y. Importantly: the same tiny BPF program that enforces this rule is also logically required in order to route packets, since our WireGuard mesh uses a different addressing format than our hosts.

3 Likes
4

Thanks for the answers! This in particular helped it “click” for me:

Importantly: the same tiny BPF program that enforces this rule is also logically required in order to route packets, since our WireGuard mesh uses a different addressing format than our hosts.

And that different addressing format is the transform Kurt described, or another layer altogether?

We don’t route customer 6pn networks at all

Hmm, so sharing my mental model aloud – I send some traffic between two fly instances. Packets go out my interface to the other end of wg interface → BPF transform destination address to correct host (??) → reverse it at dest → pass through wg at destination? I guess the “we don’t route customer networks” angle is that wireguard handles all the routing? Fly does some minimal transforms but preserves the addresses otherwise?

anyway appreciate the clarification! happy to mark this “solved”

1 Like
5

That’s right, Wireguard handles all the routing, and only routes the per machine prefixes. If a packet hits a host to one of the 6pn addresses somehow, it just gets dropped.