Announcement: Shared Anycast IPv4

carter · December 25, 2022, 3:11am

Good to know about! Thanks for surfacing that for me.

Seventhnadir · December 30, 2022, 9:33am

I have a Golang toy app that accepts connections on a TCP port, very similar to the example below.

fly-apps/udp-echo-/blob/main/fly.toml

# fly.toml file generated for sample-udp-tcp-echo on 2021-12-07T16:48:59-06:00

app = "sample-udp-tcp-echo"

kill_signal = "SIGINT"
kill_timeout = 5
processes = []

[env]
  ECHO_PORT = 5000

[experimental]
  allowed_public_ports = []
  auto_rollback = true

[[services]]
  internal_port = 5000
  protocol = "udp"

  [[services.ports]]

This file has been truncated. show original

No matter what I tried, I couldn’t get it to work until I ran fly ips allocate-v4 and from there it started working immediately.

It’s possible that I was doing something wrong on my end trying to connect to TCP via the IPv6 address with netcat <ipv6 addr> 5000, but it’s also possible that TCP connections over IPv6 might be collateral damage to this change?

tom93 · December 30, 2022, 11:21am

I can reproduce this issue. Specifically, it appears that non-standard TCP ports don’t work via IPv6 when the app also has a shared IPv4 address allocated.

Converting the shared IPv4 address to a dedicated IPv4 address causes the IPv6 address to work. Releasing the shared IPv4 address also causes the IPv6 address to work.

Steps:

$ mkdir app && cd app
$ flyctl launch --image flyio/hellofly:latest # accept defaults
# add external port 5000
$ cat >> fly.toml <<"EOF"
  [[services.ports]]
    handlers = ["http"]
    port = 5000
EOF
$ flyctl deploy

$ flyctl ips list
VERSION IP                      TYPE            REGION  CREATED AT
v6      2a09:8280:1::3:bf58     public          global  30s ago
v4      66.241.124.247          public (shared)

# test from a shell inside the app's VM
# (for reproducibility, and because the computer I'm currently using doesn't support IPv6)
$ flyctl ssh console
# apk add wget # the image includes busybox wget, but we want GNU wget
# wget -O - -6 http://$FLY_APP_NAME.fly.dev:80 # succeeds
...
# wget -O - -6 http://$FLY_APP_NAME.fly.dev:5000 # fails
--2022-12-30 10:43:42--  http://morning-shape-6643.fly.dev:5000/
Resolving morning-shape-6643.fly.dev (morning-shape-6643.fly.dev)... 2a09:8280:1::3:bf58
Connecting to morning-shape-6643.fly.dev (morning-shape-6643.fly.dev)|2a09:8280:1::3:bf58|:5000... connected.
HTTP request sent, awaiting response... Read error (Connection reset by peer) in headers.
Retrying.

--2022-12-30 10:43:43--  (try: 2)  http://morning-shape-6643.fly.dev:5000/
Connecting to morning-shape-6643.fly.dev (morning-shape-6643.fly.dev)|2a09:8280:1::3:bf58|:5000... connected.
HTTP request sent, awaiting response... Read error (Connection reset by peer) in headers.
Retrying.

--2022-12-30 10:43:45--  (try: 3)  http://morning-shape-6643.fly.dev:5000/
Connecting to morning-shape-6643.fly.dev (morning-shape-6643.fly.dev)|2a09:8280:1::3:bf58|:5000... connected.
HTTP request sent, awaiting response... Read error (Connection reset by peer) in headers.
Retrying.

^C
# exit

Convert the shared IPv4 address to a dedicated IPv4 address (causes IPv6 to work):

$ flyctl ips allocate-v4
VERSION IP              TYPE    REGION  CREATED AT
v4      149.248.221.19  public  global  7s ago

$ flyctl ips list
VERSION IP                      TYPE    REGION  CREATED AT
v4      149.248.221.19          public  global  12s ago
v6      2a09:8280:1::3:bf58     public  global  2m17s ago

$ flyctl ssh console
# wget -O - -6 http://$FLY_APP_NAME.fly.dev:80 # succeeds
...
# wget -O - -6 http://$FLY_APP_NAME.fly.dev:5000 # succeeds
--2022-12-30 10:44:53--  http://morning-shape-6643.fly.dev:5000/
Resolving morning-shape-6643.fly.dev (morning-shape-6643.fly.dev)... 2a09:8280:1::3:bf58
Connecting to morning-shape-6643.fly.dev (morning-shape-6643.fly.dev)|2a09:8280:1::3:bf58|:5000... connected.
HTTP request sent, awaiting response... 200 OK
Length: 96 [text/html]
Saving to: 'STDOUT'
<!DOCTYPE html>
<html lang="en">
<head>
</head>
<body>
<h1>Hello from Fly</h1>

</body>
</html>

2022-12-30 10:44:53 (13.6 MB/s) - written to stdout [96/96]

# exit

Release the (now dedicated) IPv4 address:

$ flyctl ips release 149.248.221.19
Released 149.248.221.19 from morning-shape-6643
$ flyctl ssh console
# wget -O - -6 http://$FLY_APP_NAME.fly.dev:80 # succeeds
...
# wget -O - -6 http://$FLY_APP_NAME.fly.dev:5000 # succeeds
...
# exit

Allocate a shared IPv4 address (causes IPv6 to fail):

$ flyctl ips allocate-v4 --shared
VERSION IP              TYPE    REGION
v4      66.241.124.67   shared  global

$ flyctl ips list
VERSION IP                      TYPE            REGION  CREATED AT
v6      2a09:8280:1::3:bf58     public          global  6m20s ago
v4      66.241.124.67           public (shared)

$ flyctl ssh console
# wget -O - -6 http://$FLY_APP_NAME.fly.dev:80 # succeeds
...
# wget -O - -6 http://$FLY_APP_NAME.fly.dev:5000 # fails
--2022-12-30 10:49:05--  http://morning-shape-6643.fly.dev:5000/
Resolving morning-shape-6643.fly.dev (morning-shape-6643.fly.dev)... 2a09:8280:1::3:bf58
Connecting to morning-shape-6643.fly.dev (morning-shape-6643.fly.dev)|2a09:8280:1::3:bf58|:5000... connected.
HTTP request sent, awaiting response... Read error (Connection reset by peer) in headers.
Retrying.

--2022-12-30 10:49:06--  (try: 2)  http://morning-shape-6643.fly.dev:5000/
Connecting to morning-shape-6643.fly.dev (morning-shape-6643.fly.dev)|2a09:8280:1::3:bf58|:5000... connected.
HTTP request sent, awaiting response... Read error (Connection reset by peer) in headers.
Retrying.

--2022-12-30 10:49:08--  (try: 3)  http://morning-shape-6643.fly.dev:5000/
Connecting to morning-shape-6643.fly.dev (morning-shape-6643.fly.dev)|2a09:8280:1::3:bf58|:5000... connected.
HTTP request sent, awaiting response... Read error (Connection reset by peer) in headers.
Retrying.

^C
# exit

Release the shared IPv4 address (causes IPv6 to work):

$ flyctl ips release 66.241.124.67
Released 66.241.124.67 from morning-shape-6643
$ flyctl ssh console
# wget -O - -6 http://$FLY_APP_NAME.fly.dev:5000 # succeeds
...
# exit

jerome · December 30, 2022, 4:30pm

Sorry about the weirdness with shared IPs. We’re working on better messaging from our API and outright preventing deployments when they don’t fit the criteria to avoid such confusion.

It has been ready for a few days, but I didn’t want to risk breaking anything during the holidays (since this touches deploys) when we might not have the people to fix it fast enough.

I will roll it out Monday.

Jens · December 30, 2022, 4:34pm

btw. The fly UI shows no IP configured, even if a shared IP v4 is set up and working. Would be helpful of you could show this correctly.

jerome · January 2, 2023, 3:09pm

I have deployed a change that should tell you if you can’t deploy when using shared IPs.

This hopefully will prevent a bunch of confusion with shared IPs.

jerome · January 2, 2023, 6:27pm

The UI should now properly display shared IPv4s.

DazWilkin · January 3, 2023, 7:51pm

I’ve been deploying an app successfully for some months that I suspect is broken by this change and I’m unsure why it’s broken and how to fix it.

kill_signal = "SIGINT"
kill_timeout = 5

[build]
  image = "registry.fly.io/healthcheck-server:[[edited]]"

[experimental]
  exec = [
    "/server",
    "--failure_rate=0.5",
    "--changes_rate=15s",
    "--endpoint=:50051",
    "--services=,grpc.health.v1.Health"
  ]
  private_network = true

[[services]]
  internal_port = 50051
  protocol = "tcp"

  [services.concurrency]
    hard_limit = 25
    soft_limit = 20

  [[services.ports]]
    handlers = ["tls"]
    tls_options = {"alpn" = ["h2"]}
    port = "443"

#[metrics]
#  port = 50051
#  path = "/metrics"

On deployment (using docker.io/flyio/flyctl:v0.0.442):

==> Verifying app config


Configuration errors in /fly.toml:

    ✘ base: Services defined at indexes: 0 require a dedicated IP address. You currently have no dedicated IPs allocated. Please allocate at least one dedicated IP before deploying (`fly ips allocate-v4` and/or `fly ips allocate-v6`). Affected services: 
  [0] tcp/443 => 50051

Error App configuration is not valid

It meets one (!?) of the requirements:

HTTP on port 80
TLS + HTTP on port 443

If I understand correctly, I should allocate an IPv4:

podman run \
[[edited]] \
docker.io/flyio/flyctl:v0.0.442 \
ips allocate-v4 --shared --app=healthcheck-server

And this succeeds (and IPv4 is assigned) but the app remains “pending”.

Questions:

Why isn’t my app config conformant?
Should I ips allocate-v4 --shared before deploying?

Thanks!

jerome · January 3, 2023, 10:38pm

Shared IPv4s don’t work with just TLS on port 443. It needs to be both TLS+HTTP.

This would work:

[[services]]
  internal_port = 50051
  protocol = "tcp"

  [services.concurrency]
    hard_limit = 25
    soft_limit = 20

  [[services.ports]]
    handlers = ["tls", "http"]
    tls_options = {"alpn" = ["h2"]}
    port = "443"

It sounds like you need to do HTTP yourself? If that’s the case, you’ll need to have only dedicated IPs assigned.

DazWilkin · January 3, 2023, 11:00pm

Thanks @jerome

Yes, it’s a gRPC app and it doesn’t work with Fly’s http handler.

tom93 · January 4, 2023, 6:37am

I think shared IPv4 did work with just TLS earlier. Is that possible? I assumed Fly was using SNI to identify the hostname so there was no technical reason for an HTTP handler. It would be really neat if it worked, because it would allow non-HTTP protocols (e.g. gRPC) to be used with shared IPv4s by wrapping them in TLS.

Thanks. Some feedback: The error message when running fly ips allocate-v4 --shared on an app with non-standard ports doesn’t make sense:

Services defined at indexes: 0 require a dedicated IP address. You currently have a shared IPv4 assigned to your app (fly ips list). Release the shared IP (fly ips release <shared ip>) and/or allocate only dedicated IPs before deploying (fly ips allocate-v4 and/or fly ips allocate-v6). Affected services:
[0] tcp/80,443,5000 => 8080

This message would make sense in the context of fly deploy, but in the context of ips allocate-v4 --shared it is plain wrong: the app doesn’t currently have a shared IPv4. I’m not sure what a better message would be. Maybe suggest to the user to allocate a dedicated IP by dropping the --shared option. It would also help if the error message explained how to remove the non-standard port (editing fly.toml wasn’t enough, I also had to run fly deploy; not sure if there is a simpler way).

andrwe · January 7, 2023, 4:33pm

I allocated a dedicated ipv4 and then released it. I now have none. Is there a way go back to using a shared one?

ignoramous · January 7, 2023, 4:42pm

Yes… you probably missed this:

andrwe · January 7, 2023, 4:46pm

Yep, I just figured that out by guessing at the --shared flag and came back to share (no pun intended). Sorry for not reading more carefully—I’m having a very frustrating morning. And thank you for the response.

donnamunroe · January 30, 2023, 8:06am

Have you already started charging dedicated IP? I didn’t know until I saw this post. Will I be charged?

jerome · January 30, 2023, 2:01pm

We haven’t started charging for them and we’ll give plenty of warning before this happens.

bruno-scholarcy · February 22, 2023, 8:45pm

Maybe this is a dumb question, but how can I allocate a cert and a domain name to a shared ipv4 address? Is there going to be a CNAME setup or similar? I don’t need a dedicated one for the testing environment, but Stripe doesn’t seem to support ipv6 only domains yet

jerome · February 22, 2023, 8:46pm

Everything works the same for domains / certificates w/ shared IPv4. Best to assign a CNAME if you’re using a subdomain, and a A record is you’re using an apex domain.

bruno-scholarcy · February 22, 2023, 8:58pm

OK I added a CNAME to appNAME.fly.dev and that seems to work now. Perhaps the UI needs to be updated? By default on new apps it’ll only prompt you to create an AAAA record, regardless of whether it’s a apex domain or subdomain. Before this change, I used to create both a A and AAAA record and everything worked fine.

pier · March 14, 2023, 1:42pm

Just to be 100% certain…

A shared IPv4 will not change, right?

I can still point an A DNS record to it?

I’m migrating an app from v1 to v2 and wouldn’t want to mess that up.