Announcement: Shared Anycast IPv4

Fresh Produce
1

TL;DR: We’re not automatically allocating dedicated IPv4 per app on the first deployment anymore. If you’re deploying a new app that only exposes HTTP on default ports (80, 443), your app will get a shared IPv4 automatically, instead of a dedicated IPv4.

With our rise in popularity and the scarcity of IPv4 available to the world, we’ve decided to implement a way to route multiple apps via the same IPs.

By default, your app will now get a dedicated anycast IPv6 and a “shared” anycast IPv4 if it fits your services. The criteria are:

  • HTTP on port 80
  • TLS + HTTP on port 443

Apps that don’t have services fitting these will now only get a dedicated IPv6 allocated. That’s if you’re doing raw TCP proxying or UDP.

Don’t worry, if your app needs a dedicated, public, anycast IPv4, you can still manually allocate one with: fly ips allocate-v4. However, allocating a dedicated IPv4 is now opt-in only, whatever your defined services are.

There is now a new flag on that command (as of flyctl v0.0.439) if you’d like to switch to a shared IPv4:

      --shared          Allocates a shared IPv4

(It only succeeds if you’ve deallocated all dedicated IPv4 beforehand)

Hopefully, this changes is invisible for all of you. If you do find an issue, be sure to let us know.

17 Likes
2

I’m sure there was a good reason to make this change suddenly, but some warning on changes like this ahead of time would be greatly appreciated in the future. My SAAS depended on a dedicated IPv4 being included in a deployment, and overnight I had new customers wondering why their cluster wasn’t working and I’ve lost some business.

The reason why this wasn’t an invisible change to me, I think, is that my configuration doesn’t include the http or https services, because I’m running caddy clusters that provide and terminate TLS themselves.

Is there a way to flag the launch mutation in the graphql API to include a dedicated IPv4, or is there a separate mutation I could call to add it immediately after?

Also, how does the shared IPv4 work? If I point an A record at that, how does it know which app to send the request to?

4 Likes
3

@carter are you using the API for this? You can allocate IPs before the first deploy. fly ips allocate-v4 (or the API equivalent) will prep an app to work like it did before.

Note, though, that dedicated IPv4 addresses added to apps will cost $2/mo starting in Jan.

Shared IP routing works based on the certificate names on each app. When we generate a certificate for one of these apps, we add the hostname to a kv store. Then route new connections based on that.

4

I am using the graphql API, yep. For deploying, I was using the launch mutation from the graphql playground docs, which kind of does everything in one go. Is there a way to tell it to add an IPv4 with that mutation? If not, I might have to figure out splitting it out into multiple steps.

Pricing on the dedicated IPv4 is all good with me. Looks like the shared IP routing won’t work for my use case because I terminate my own certs. It would be aweseome if there was some way to add hostnames to that KV store you mentioned through the API, without requiring the http/https services for my app.

The reason I’m not using those services is that I’ve got more than 120k domains and subdomains routing through my clusters so switching to Fly’s managed certs would be a huge jump in cost just for the certs, to do something my app already does. Though I do wish I could get some of the nice benefits they provide.

Another question, so are my existing dedicated IPv4 addresses at risk of going away or being changed to shared addresses now, or will I just be billed for those in January?

5

Interesting. What about existing apps (apps created before this announcement) with one public IPv4 assigned? I am guessing $2/mo for those, too?

6

Someday, but with lots of heads up. We’ll send a notice and give people at least 90 days to switch to a shared IP before we start billing. Shipping shared IPs abruptly bought us a bunch of time (we have a lot of IPs, but we need them for other things!) so we can start reclaiming IPs less disruptively.

1 Like
7

A policy of auto reclaiming IPs from apps that have been dead for > 30/60/90 days might also be a decent way to reclaim some, though just charging for them will probably do a decent job of solving that too.

2 Likes
8

$2/mo is super reasonable.

For context, AWS and GCP charge $18/mo for raw TCP/UDP over Anycast IPv4s (+ a markup on egress, routing rules etc). The flip side is, AWS and GCP offer unicast IPv4s at no-cost (or, rather the cost is rolled into compute)… Still, way better than Cloudflare who charge $1/GB egress for TCP/UDP (Spectrum) over Anycast!

1 Like
9

@carter I just saw your questions!

The officially supported way to programmatically launch apps on Fly is with the Machines API. You still need graphql for allocating IPs and volumes (for now), but the best way to do what you’re after is:

  1. Create app with machines api
  2. Allocate IPs with graphql
  3. Create machines for app with services

We probably won’t make shared IP routing more powerful. Shared IPs exist to let people try the service and run side projects without spending money.

For your current dedicated IPs, we’ll send a notice saying “we’re gonna start billing for these on X date”. Then we’ll send a few more. Then we’ll start billing for them. We won’t remove IPs from existing apps production apps.

10

Awesome, thanks for the reply!

Right now I’m using the graphql API to work only at the apps level and it’s been working pretty well. Is that something I should plan to change sooner than later? It sounds like machines will be the only officially supported API.

For billing, I have absolutely no problem with paying for IPs, in fact I’m glad that they’re not free because that seemed like people were always going to abuse that. Though I do appreciate the advanced warning :+1:

My real concern would be if something functionality-wise were to change out from under me without warning and break something. Particularly around the anycast IPs because at the moment Fly is pretty unique with the way you offer those and I couldn’t easily get that somewhere else. If you decided to stop offering that (say, went with shared IPs for all) I suspect a bunch of people would be scrambling pretty hard. Which is why I reacted more strongly when I woke up yesterday to this immediate change.

11

Machines will be the only officially supported API, yes. Right now, it’s lacking some stuff (IPs, volumes) but that should change pretty quickly.

The GraphQL API is very much caveat emptor. We built it to support our CLI. We don’t have any problem with people using it. But its behavior will definitely change. :slight_smile:

1 Like
12

It would be really nice if the free allowance was extended to include 1 dedicated IPv4 address per organization, but I won’t hold my breath.

Off topic (and probably niche), but I would really appreciate some way to use the Machines API without setting up WireGuard VPN or installing flyctl. I really liked how the GraphQL API could be used to create an app from virtually any computer without installing anything (just need curl, or a web browser’s JavaScript console), and it would be a shame to lose that ability.*

I envision a simple public-facing proxy service that forwards requests to the Machines API endpoint on the Fly private network. The requests already include the user’s authentication token, so if the proxy requires authentication (e.g. to connect to the user’s Fly private network) it can simply peek at the request.

(Relatedly, Heroku has a web console that effectively allows a user to SSH into a dyno using just a web browser. Fly’s Code Server launcher sounds really cool, but it requires 1GB RAM which is above the free tier.)

*This approach already has limitations, e.g. remote builders can’t be accessed over the GraphQL API, but I still think it is useful.

2 Likes
13

I would really appreciate some way to use the Machines API without setting up WireGuard VPN or installing flyctl

I’d also love that. Not that it’s a huge deal, but requiring it is enough friction that I haven’t switched over yet. Having a VPN on prod to use an API just isn’t something I’ve had to do for any other service.

14

api.machines.dev not suit your needs? Fly.io Machines Proxy Timeout - #6 by ignoramous

4 Likes
15

Brilliant! I see it is safe to use, I hope it will be added to the docs soon.

1 Like
16

Good to know about! Thanks for surfacing that for me.

1 Like
17

I have a Golang toy app that accepts connections on a TCP port, very similar to the example below.

No matter what I tried, I couldn’t get it to work until I ran fly ips allocate-v4 and from there it started working immediately.

It’s possible that I was doing something wrong on my end trying to connect to TCP via the IPv6 address with netcat <ipv6 addr> 5000, but it’s also possible that TCP connections over IPv6 might be collateral damage to this change?

2 Likes
18

I can reproduce this issue. Specifically, it appears that non-standard TCP ports don’t work via IPv6 when the app also has a shared IPv4 address allocated.

Converting the shared IPv4 address to a dedicated IPv4 address causes the IPv6 address to work. Releasing the shared IPv4 address also causes the IPv6 address to work.

Steps:

$ mkdir app && cd app
$ flyctl launch --image flyio/hellofly:latest # accept defaults
# add external port 5000
$ cat >> fly.toml <<"EOF"
  [[services.ports]]
    handlers = ["http"]
    port = 5000
EOF
$ flyctl deploy

$ flyctl ips list
VERSION IP                      TYPE            REGION  CREATED AT
v6      2a09:8280:1::3:bf58     public          global  30s ago
v4      66.241.124.247          public (shared)

# test from a shell inside the app's VM
# (for reproducibility, and because the computer I'm currently using doesn't support IPv6)
$ flyctl ssh console
# apk add wget # the image includes busybox wget, but we want GNU wget
# wget -O - -6 http://$FLY_APP_NAME.fly.dev:80 # succeeds
...
# wget -O - -6 http://$FLY_APP_NAME.fly.dev:5000 # fails
--2022-12-30 10:43:42--  http://morning-shape-6643.fly.dev:5000/
Resolving morning-shape-6643.fly.dev (morning-shape-6643.fly.dev)... 2a09:8280:1::3:bf58
Connecting to morning-shape-6643.fly.dev (morning-shape-6643.fly.dev)|2a09:8280:1::3:bf58|:5000... connected.
HTTP request sent, awaiting response... Read error (Connection reset by peer) in headers.
Retrying.

--2022-12-30 10:43:43--  (try: 2)  http://morning-shape-6643.fly.dev:5000/
Connecting to morning-shape-6643.fly.dev (morning-shape-6643.fly.dev)|2a09:8280:1::3:bf58|:5000... connected.
HTTP request sent, awaiting response... Read error (Connection reset by peer) in headers.
Retrying.

--2022-12-30 10:43:45--  (try: 3)  http://morning-shape-6643.fly.dev:5000/
Connecting to morning-shape-6643.fly.dev (morning-shape-6643.fly.dev)|2a09:8280:1::3:bf58|:5000... connected.
HTTP request sent, awaiting response... Read error (Connection reset by peer) in headers.
Retrying.

^C
# exit

Convert the shared IPv4 address to a dedicated IPv4 address (causes IPv6 to work):

$ flyctl ips allocate-v4
VERSION IP              TYPE    REGION  CREATED AT
v4      149.248.221.19  public  global  7s ago

$ flyctl ips list
VERSION IP                      TYPE    REGION  CREATED AT
v4      149.248.221.19          public  global  12s ago
v6      2a09:8280:1::3:bf58     public  global  2m17s ago

$ flyctl ssh console
# wget -O - -6 http://$FLY_APP_NAME.fly.dev:80 # succeeds
...
# wget -O - -6 http://$FLY_APP_NAME.fly.dev:5000 # succeeds
--2022-12-30 10:44:53--  http://morning-shape-6643.fly.dev:5000/
Resolving morning-shape-6643.fly.dev (morning-shape-6643.fly.dev)... 2a09:8280:1::3:bf58
Connecting to morning-shape-6643.fly.dev (morning-shape-6643.fly.dev)|2a09:8280:1::3:bf58|:5000... connected.
HTTP request sent, awaiting response... 200 OK
Length: 96 [text/html]
Saving to: 'STDOUT'
<!DOCTYPE html>
<html lang="en">
<head>
</head>
<body>
<h1>Hello from Fly</h1>

</body>
</html>

2022-12-30 10:44:53 (13.6 MB/s) - written to stdout [96/96]

# exit

Release the (now dedicated) IPv4 address:

$ flyctl ips release 149.248.221.19
Released 149.248.221.19 from morning-shape-6643
$ flyctl ssh console
# wget -O - -6 http://$FLY_APP_NAME.fly.dev:80 # succeeds
...
# wget -O - -6 http://$FLY_APP_NAME.fly.dev:5000 # succeeds
...
# exit

Allocate a shared IPv4 address (causes IPv6 to fail):

$ flyctl ips allocate-v4 --shared
VERSION IP              TYPE    REGION
v4      66.241.124.67   shared  global

$ flyctl ips list
VERSION IP                      TYPE            REGION  CREATED AT
v6      2a09:8280:1::3:bf58     public          global  6m20s ago
v4      66.241.124.67           public (shared)

$ flyctl ssh console
# wget -O - -6 http://$FLY_APP_NAME.fly.dev:80 # succeeds
...
# wget -O - -6 http://$FLY_APP_NAME.fly.dev:5000 # fails
--2022-12-30 10:49:05--  http://morning-shape-6643.fly.dev:5000/
Resolving morning-shape-6643.fly.dev (morning-shape-6643.fly.dev)... 2a09:8280:1::3:bf58
Connecting to morning-shape-6643.fly.dev (morning-shape-6643.fly.dev)|2a09:8280:1::3:bf58|:5000... connected.
HTTP request sent, awaiting response... Read error (Connection reset by peer) in headers.
Retrying.

--2022-12-30 10:49:06--  (try: 2)  http://morning-shape-6643.fly.dev:5000/
Connecting to morning-shape-6643.fly.dev (morning-shape-6643.fly.dev)|2a09:8280:1::3:bf58|:5000... connected.
HTTP request sent, awaiting response... Read error (Connection reset by peer) in headers.
Retrying.

--2022-12-30 10:49:08--  (try: 3)  http://morning-shape-6643.fly.dev:5000/
Connecting to morning-shape-6643.fly.dev (morning-shape-6643.fly.dev)|2a09:8280:1::3:bf58|:5000... connected.
HTTP request sent, awaiting response... Read error (Connection reset by peer) in headers.
Retrying.

^C
# exit

Release the shared IPv4 address (causes IPv6 to work):

$ flyctl ips release 66.241.124.67
Released 66.241.124.67 from morning-shape-6643
$ flyctl ssh console
# wget -O - -6 http://$FLY_APP_NAME.fly.dev:5000 # succeeds
...
# exit
2 Likes
19

Sorry about the weirdness with shared IPs. We’re working on better messaging from our API and outright preventing deployments when they don’t fit the criteria to avoid such confusion.

It has been ready for a few days, but I didn’t want to risk breaking anything during the holidays (since this touches deploys) when we might not have the people to fix it fast enough.

I will roll it out Monday.

1 Like
20

btw. The fly UI shows no IP configured, even if a shared IP v4 is set up and working. Would be helpful of you could show this correctly.

1 Like