(Unfortunately) We're NAT'ing Fly Machines' IPv6 Addresses

Fresh Produce
1

This is sort of a long post so a TL;DR: Machines no longer have public IPv6 addresses, but still with IPv6 connectivity. Apps’ public IPv6 addresses are unchanged, only individual Machines. You are very unlikely to be affected by this change, unless your Machines need to connect to something external that supports both IPv4 and IPv6, and you specifically need to connect over IPv6. In that case, you’ll need to configure something like /etc/gai.conf manually.

Fly Machines have had a public IPv6 addresses assigned to them from early on. Eagle-eyed users might have noticed that Machines created recently have been lacking that public IPv6 addresses, and instead with only a private 6PN.

That’s right, we’re using IPv6 NAT now. For outbound internet traffic from Machines, we’re having hosts running Machines SNAT the 6PN addresses to the host’s public IPv6. Now, before anyone says “ewww”, hear me out: we’re not doing this due to dubious “security” arguments about how NAT increases security. It, in fact, does not, at least not in our case. Existing public IPv6s on Fly Machines already have strict, stateful firewall policies: you cannot just talk to a random Fly Machine on its public IPv6 unsolicited, even if that address is “public”. This does not change whether we’re using NAT or not. Apps have IPv6 addresses to accept requests from users, those addresses are routed through our load-balancer, fly-proxy, and those are also unaffected by IPv6 status of individual machines constituting apps.

Rather, we’re doing this mostly out of operational simplicity and reliability. With a public, semi-static IPv6 address for each Machine, renumbering (which is occasionally needed) currently requires at least force-updating each Machine. In some cases, routing the IPv6 blocks to our hosts with some providers have proven unreliable. This can happen, for example, if an upstream does not route via static routes but instead rely on neighbor discovery. This has been the source of some IPv6 instability reported both in premium support and on this community forum.

Granted, most of these issues can be either engineered or negotiated away, and yes, IPv6 NAT is just the “lazy” solution here, in some sense. But it also makes a lot of real problems go away, and, in our opinion, does not really change the experience of using Fly Machines. A Fly Machine is also not like a home device or a traditional VPS: they are only constituents of apps and may be created / destroyed / migrated / … any time, and do not really need a permanent, routable address for themselves; only apps as a whole do. The only difference is that in most glibc or musl-based images, IPv4 will now be prioritized by default over IPv6. This may even be helpful in other cases, but if you really want your app to talk over IPv6 only (if the target domain also has an IPv4 address), you’ll now need to configure getaddrinfo’s behavior manually.

What if you do require a predictable, outgoing IPv6 address for Machines? Again, even when we were allocating public IPv6 addresses, those addresses aren’t truly static, and they can change due to a number of reasons. The recommended way of getting a predictable IPv6 (or IPv4) is to use Egress IPs, which even allows all Machines of your app to share one single outgoing IP (up to some limitations) in each region.

2

Hello, your announcement has been shared in the r/IPv6 community on Reddit, which is how I came across it: Reddit - Please wait for verification

Are you using provider-independent address space? This would solve all of your stated problems, but it sounds like you’re not doing this. If you are using PI space, then I don’t understand how you’re encountering the issues that you talk about here, particularly this point:

In some cases, routing the IPv6 blocks to our hosts with some providers have proven unreliable. This can happen, for example, if an upstream does not route via static routes but instead rely on neighbor discovery.

If you’re not using PI space, that’s a bizarre choice for a company of your size and business nature, especially given that you have multiple ISP uplinks in multiple geographic locations. Please consider getting some PI space from one or more relevant RIRs or LIRs.

3

I am going to be pedantic and say yes, we are, since PA space allocated to our upstream providers would not be allowed to be used for us to provide IPs to our customers; although yes, it is our upstreams’ PI space rather than our own.

it is worth noting that using our own IPs vs. upstream providers’ would not be a meaningful change here. (note that “upstream” means a server provider such as Equinix Metal, not a transit provider such as Cogent)
we have tens or hundreds of servers per region, and even if the border routers advertised our own PI space rather than our upstreams’, we would still have to route traffic from the routers to each server, and this is where the issues with e.g. neighbour discovery come in. we don’t want to run BGP on every server!

some additional context:
we do use PI address space (2a09:8280::/29) for ingress as well as static egress, so everything we advertise from AS40509 that you can use with fly ips allocate-X.
right now we don’t BYOIP so our hosts’ IPs come from our upstream providers’ address space; although we’d like to renumber our hosts to use our own IP space in the future.

4

In addition to what’s said above, I’d also like to add that I in fact agree that this solution is not ideal and we’re not selling this as an advantage over others (hence “unfortunately”). In a perfect world we should be running an iGP to handle routing to machines’ public IPv6 addresses. A couple of problems prevent us from doing this right now, ranging from ops and scaling issues (Fly Machines are very ephemeral and can generate a huge routing table with very frequent updates) to particularities with how Fly Machines are designed and how our network is set up (which you can argue is technical debt that we should get rid of).

We do have plans to do away with a lot of these and in fact we have been working hard towards a better network setup for a while. This solution is just what we can do right now to avoid a lot of the reliability issues while these plans are under way (which will take longer, and ironically the NAT change does make things like renumbering Fly Machines onto our own IP space easier in the future), and for what Fly Machines are, there aren’t that many downsides besides that IPv6 should not be NAT’d. On the ingress direction, we have to do “NAT”-y things either way due to load balancing, and the egress direction requires conntrack in our setup with or without NAT. If we were selling a more VPS-y product, we would definitely not have done something like this.

5

Thanks for the detail, it’s much appreciated. I would agree with you calling this PI space. I was previously under the impression that you were administering your own hardware / bare metal, not using other providers for this; and that “provider” in the original post meant transit providers, not hardware providers.

That said, it sounds like you have some unwarranted aversions to iBGP on every host:

we don’t want to run BGP on every server!

Having nodes perform BGP with each other is standard fare for NAT-less Kubernetes clusters, for example, where nodes in the cluster exchange routes with each other, one route for each node, so that packets destined for particular k8s pods can be routed to the correct pod. If you’re familiar with k8s in a NAT context, this use of BGP and routing makes kube-proxy unnecessary.

To be clear, my remarks merely come from a generic, outsider’s standpoint; I don’t know or use your product at all, and thus don’t know what specific trade-offs you might be evaluating, but I have plenty of experience architecting and managing IPv6 networks as a general thing. With that in mind,@PeterCxy, I can see where you’re coming from with your final sentence:

If we were selling a more VPS-y product, we would definitely not have done something like this.

However, it instinctively seems a poor design choice to me (or indicative of existing poor choices that were made — perhaps these are among the things you’re currently working on fixing/adjusting and said that I might consider technical debt) to not necessarily assign a public address to each individual virtualised host. In particular, the following statement strikes me as bizarre:

With a public, semi-static IPv6 address for each Machine, renumbering (which is occasionally needed) currently requires at least force-updating each Machine.

It raises the question of what updates you need to apply after a renumbering event, and why.

Overall, though, it does sound like you guys know what you’re doing, which is very nice to see :slightly_smiling_face: