Since we launched our machine egress IPs feature, we’ve received numerous feedback about how this felt a bit clunky to use. Not only do you need one per machine, they are also permanently bound to a specific machine with no easy way to “transfer” between them: if you ran a bluegreen deployment, for example, all the newly-created machines will not have egress IPs, and all currently assigned egress IPs will be released when the old machines get destroyed during the bluegreen deployment process. We hear you: it is frankly quite weird to have a platform feature that restricts the use of other core features in this way. The original intention was that egress IPs will be mostly used through some kind of proxy app, but even that comes with its own drawbacks.
To solve this, in the past little while, we have been working on a new kind of egress IPs that are scoped to an (app, region) tuple. Yes, I lied a little bit in the title: due to the way egress IPs are usually used (accessing external resources under GeoIP restrictions, for example), and the fact that “Anycast”-ing these IPs will be quite complicated routing-wise (we’ll essentially have to replicate egress flow tables globally), you still need separate egress IPs per region if you have machines in multiple regions. But: they are no longer bound to single machines, and you are free to use bluegreen deployments without fear of losing egress IPs. If you have an app-scoped egress IP assigned in a region, every machine in your app in that region will use that same egress IP (with some caveats applied – see below). They are only released when you explicitly release the IP via a command, not automatically when a machine is destroyed.
This feature is currently beta, meaning that despite we think it should work, it might not! And there will probably be rough edges which we will gradually polish up in the next little while. We can’t do that without you, the user, and this is why we’re shipping it right now so that you can try it out!
How do I use it?
The flyctl version 0.3.227 (released today) contains initial support for managing app-scoped egress IPs. Since these new egress IPs belong to apps rather than machines, they share the same ip subcommand as our regular Anycast IPs (used for ingress).
To allocate one for a specific region:
flyctl ip allocate-egress -r <region>
To view the list of all egress IPs (in fact, all IPs except for the old machine-scoped egress, which uses a different subcommand):
flyctl ip list
This command is now also explicit about ingress vs. egress IPs. Unfortunately, we still cannot “just” use the ingress IPs for egress purposes (for mostly the same reasons why egress IPs are regional, plus the existence of Fly Proxy as a load balancer on all Anycast IPs).
To release app-scoped egress IPs:
flyctl ip release-egress <egress_ip_1> [egress_ip_2] ...
Managing app-scoped egress IPs is not currently supported by our web control panel, and our documentation has not been updated to reflect this new feature yet. We will be working on that in the coming weeks.
What are the caveats?
- These IPs are associated with regions, so if your app spans multiple regions, you will need multiple egress IPs. Only machines located in the same region as an egress IP can make use of that egress IP.
- These IPs aren’t implemented as managed virtual gateways (yet). Rather, we semi-statically assign source port ranges to each machine, propagate state and resolve conflicts via Corrosion. This means that:
- There can be a delay between when a machine is created or when a new egress IP is assigned and when the egress IP is usable.
- When there are conflicts between port assignment among machines (e.g. when creating a large amount of machines at once), internet may be temporarily unavailable for some machines while the conflict is resolved. You can reduce the chance of this happening by allocating more egress IPs.
- There’s a maximum number of machines each egress IP can support. Currently, that number is 64. Do note that if you use bluegreen deployments, your number of machines will double while a deployment is in progress. The solution is however simple: you can allocate more than 1 egress IPs per app per region. Each machine will pick one randomly out of all that are available.
- There’s a maximum number of connections each one of your machines can create to each destination IP. That number is currently 1024 (this does not mean each machine can only create 1024 connections in total! And it does not apply to the ingress direction either). We do not expect this to be a problem for most use cases requiring egress IPs, though: databases normally use ~100 - 200 connections at most, and HTTP endpoints generally come with multiple IP addresses. Combined with connection pooling and/or HTTP/2 multiplexing, most use cases should not require a lot of simultaneous outgoing TCP connections targetting one single IP. If this does become a problem, let us know and we’re happy to look at what we can do about it as the next step.
For most of these caveats, we have built some sanity checks and warnings into flyctl to bring them to your attention during machine creation, deployments, and IP assignment / release.
How does it interact with machine-scoped egress IPs?
They are not intended to be used together. However, if you do assign a machine-scoped egress IP via flyctl m egress-ip allocate, as of now, the machine-scoped egress IP will override the app-scoped one.
We intend to eventually deprecate machine-scoped egress IPs in favor of app-scoped ones once they are stable – having both is going to be very confusing. As a result, this behavior is not guaranteed to continue after app-scoped egress IPs are out of beta.
How much does it cost?
While in beta, app-scoped egress IPs are free. We intend for the beta period to last a month or so. That means we will likely start billing for it around the time of the new year.
We’re still finalizing the pricing details, but an app-scoped egress IPv4 address will likely cost the same as a machine-scoped one, $3.60/mo billed hourly. IPv6 addresses are always assigned together with an IPv4, but they will likely not be billed for independently. If you do release all of your egress IPv4 addresses but not all of the IPv6s, though, we may bill for the price of 1 IPv4 anyway (there will be a warning in flyctl if you did that!).
