Any way to attach a public IPv4 address to a specific machine?

1

Hi,

I am considering using Fly for a new project we’re building, an IoT platform. I really like the ease with which I can deploy apps and make an Elixir cluster.

However, I was wondering if it is possible to attach an IPv4 address to a specific server?

Essentially, I want a device to connect to a particular server over TCP. The device gets the right server’s IP address by either DNS or another method.

I think this is achievable over IPv6 using Fly as individual VMs have public IPv6. However I noticed the public IPv6 changes across deploys, but from the forums I understand that Fly Machines have the same public IPv6 across deploys. However, even my own home internet doesn’t support IPv6 yet, and just yesterday there was a post on Hacker News about how unusable IPv6-only is till today. So I would like to be able to access machines on IPv4 too for the devices and networks that do not have IPv6 capability.

I see additional IPv4s are available for $2/month, but as far as I understand from the docs, they’re attached only to the application, which the Fly-Proxy directs traffic to using Anycast.

I have read through the docs and the forums, and my understanding so far is that an Application gets an IPv4 and an IPv6, and each VM gets a public IPv6 address (on which public traffic can be sent, but I haven’t been able to make it work yet, I’ll try further).

The main goal is reduce the amount of traffic being forwarded from one VM to another because it can amplify really quickly. I thought of another solution, which is to deploy multiple applications in different regions with the same codebase. This way, at least there won’t be too much cross-region TCP traffic flowing through our VMs.

If this doesn’t work today, are you considering allowing attaching IPv4 addresses to specific VMs for extra cost?

What about some programatic way to control what Fly-Proxy does? For example, if a device connects to an IPv4 of the Application, but sends a specifically crafted TCP packet containing some identifier or the public IPv6 address, then the Fly-Proxy could forward it to the right machine, instead of the closest one? Something like one of the HTTP headers which directs traffic to specific machines, but for TCP. It doesn’t even have to be TLS, and perhaps the TLS handshake can start after that (so that the possibility of the VM handling it remains, instead of terminating on the edge)?

Thanks and sorry for the long post!

2

IP address affinity and port affinity are not yet supported. May be Thomas et al are already working on it (going by their recent twitter posts)!

For pseudo IP affinity, you can consider assigning regional IPs if the app only ever has a single Machine VM per region: Multi-region application where users / sessions are pinned to one region? - #5 by ignoramous

3

Hey,

Thanks for the prompt reply!

I think I could build the full architecture I described above for IPv6, and go for a region-level version of that for IPv4. And whenever there’s IPv4-per-machine support, it’ll be pretty easy to upgrade.

Now I just need to figure out how to send TCP traffic to an IPv6 address found in the $FLY_PUBLIC_IP variable. Is there a guide to do that?

I tried adding it to the experimental.allowed_public_ports section, but I’m pretty sure I’m doing something wrong. The deploy fails. Could it have something to do with TCP health checks, even though they’re for the HTTP(S) ports?

4

You’re right: Port numbers in experimental.allowed_public_ports = [] is how you expose ports on your VM’s public IP addresses, ref: Custom TCP ports - #2 by kurt

What deploy failure do you see?

5

It was an issue in my GenServer code. The application startup got stuck, and hence the deploys failed.

I got it working! I can now route TCP traffic to the $FLY_PUBLIC_IP address.

Just to further my understanding, is this IP attached directly to the VM or is it attached to the Fly-Proxy, which then forwards TCP traffic to this VM?

I know it won’t matter much, apart from minor latency differences I believe.

6

Hi!

IP addresses are at the app level (not machine level). Basically that means they go to the proxy, yes (not to a specific VM)

Fly proxy will load balance requests across machines in an app (based on what port is used in the request).

Furthermore, my understanding is that the proxy will only load balance requests to machines listening on the port used in the request if they match the “external” port defined by the services configuration. That was a mouthful!

7

Okay so I do get the reasons for that architecture, and they’re quite well-suited to web applications of most kinds.

However, is there a plan to implement an option to add IPv4s on-demand to each VM? And any timeline on that?

8

Not currently (to my knowledge)! A few options you have are:

  1. use regular apps over machines
  2. create an app per VM if you need them to have a dedicated IP
  3. setup a proxy to accept requests and route them to the appropriate machine VM (each machine gets an internal network host name based on the machine ID)
9

What’s your use case for wanting an IP per VM in your case?

10

I am using VMs for now. I haven’t understood Machines well enough to try them out just yet.

For the second point, I’m thinking of using the Regional IPs feature Fly supports.

The third point is an issue. It’ll exponentially increase the amount of traffic flowing between VMs, when instead a device could directly connect to the right VM first.

11

Yeah, you’d essentially need a two-tiered architecture, with one app with a set of VMs that are frontends, and the other apps that are backends, where frontends are L4/L7 load balancers / request routers (HAProxy, for example) to your services hosted on the backends.

This is wasteful since there’s no advantage doing so on a platform like Fly.

1 Like
12

True! I’ll experiment with this architecture on Fly and see how far I can take it without IPv4 per VM. If its good enough for now, I’ll stick to Fly, because the few things it makes simpler like internal networking and deploys, I would rather have for the start.

Thanks for all your help @ignoramous and @fideloper-fly!

1 Like
13

Two-tiered architecture, with one app with a set of VMs that are frontends, and the other apps that are backends, where frontends are L4/L7 load balancers / request routers (HAProxy, for example) to your services hosted on the backends.

This is not so wasteful as it may seem. Actually, this is the right approach for any API service and maybe even WWW sites. The reason is because if you want to achieve a long-term compatibility of your HTTP endpoints then it becomes extremally frustrating to try to do that at the app level.

With a two-tiered architecture, long-term compatibility becomes realistically approachable as you can always proxy the URL segments to the right app(s) or put redirects when needed.

For instance, we do it using Fly and it works very decently. Anycast nature of fly routing helps immensely.

As for the routers themselves, they are just Fly apps with HAProxy/Nginx/Caddy (choose one to your liking).

2 Likes