Hoping to get some clarification on how dedicated IPv4s are billed. I see that they are currently $2/mo, but how does that work if I release the IP halfway through the month? Is it prorated? Daily? Hourly? Or will I be billed for peak usage (i.e. if I allocate 20 IPs, release them, then reallocate 20 more, then release all of them 5 minutes later, I’ll get billed $40 that month?).
I have a use case that involves allocating dedicated IPs for short durations somewhat frequently, but I don’t need them long term.
This recently was brought up on a previous post too
And the honest answer is that we haven’t really started doing it yet but it’s on our short radar of things to do so feedback like you need for IPs for a short period use case is actually very valuable for decisions we are going to make.
When we do actually build this billing we’ll make sure everyone gets a fair heads up though like we did for Upstash Redis, stay tuned to Fresh Produce for more news.
Just to clarify then - it sounds like you haven’t yet decided exactly how the billing would work (i.e. peak usage vs. hourly/daily)? I’m less concerned about the final dollar amount (as long as it’s reasonable, which everything mentioned so far is). I’m more concerned that my usage pattern would rack up enormous bills if you end up charging per call to allocate, for example (i.e. if I never use more than a few IPs, but I’m constantly allocating / releasing them).
And in case it’s helpful - the main reason I need dedicated IPs for short durations is because I have need to use WebRTC, which needs a dedicated IP and UDP support. Since Fly’s dedicated IPv6 doesn’t support UDP traffic, I’ll need the dedicated IPv4s, but my app should be able to allocate/release them only when needed to hopefully minimize my costs.
That is correct, we are not set on a billing scheme.
Thanks for sharing more on your use case. Can I suggest something then? I’d advise you to create a simple router app with a pool of dedicated IPs (you might only need one, maybe) and use Fly Replay.
The router app is as simple as responding with the replay header with the correct machine/app you want to point to. That’s what we do for our Web Terminal, we route one IP to one of thousands of machines based on who you are.
Appreciate the suggestion! Unfortunately, my use case is for stateful WebRTC sessions, which use UDP streams as transport, not HTTP requests, so I can’t use the fly-replay header.
I considered doing something analogous - setting up some kind of UDP router app that got a single dedicated IP and proxied the data to different Machines using application-managed port ranges to differentiate and direct traffic.
But unfortunately that just pushes the problem another layer down - now the router app needs to communicate with the machines running the WebRTC application over UDP, which means I need dedicated IPv4 addresses for each … so we’re back to square one.
But perhaps I’m misunderstanding - I’ll happily take any further suggestions!
Fly-Replay is not really a proxy as you’d do with something like nginx, it’s just a way of one “proxy” app tell Fly Proxy to reroute the request to another machine as if it was always that case. Your router app doesn’t need to communicate with your other machines.
Ah interesting. Sorry if I’m being a bit dense, but I still don’t see how that would work with my WebRTC use case (or maybe you were just clarifying how it works as an FYI?).
WebRTC uses UDP to send media data, and my understanding is that in order to accept incoming UDP traffic, a Fly Machine needs to allocate a dedicated IPv4 address (because the dedicated IPv6 and shared IPv4 addresses don’t support UDP traffic).
Since the Fly-Replay header is an HTTP header, I don’t see how it would help here, since there’s no HTTP request that my app would be responding to - it’s trying to respond to UDP traffic.
I think I understand better now and Fly-Replay might not work here, you’re correct.
If I may suggest another possible solution that could help prevent a ton of IP usage, would it be possible for your use case to have one app have multiple UDP listener ports and run on the same dedicated IP? You could create a pool of machines on the same app and reuse them as needed.
Would UDP traffic from the same client get routed to the same machine? WebRTC is stateful, so I need traffic from a particular user to be routed to the same machine for the length of a session. I wasn’t sure how the load balancer works for UDP traffic. I couldn’t find any documentation for that.
If the load balancer maintains session stickiness that would be ideal (would love to know how it calculates a session “key”). If not, it still might work if I can influence the load balancer’s routing in any way (i.e. if a particular port on the load balancer will always route to a consistent machine, I could probably make that work).
But if it just round-robins packets to the machines, I don’t think that would work unfortunately.
Oh interesting - so Fly’s load balancer detects the port that each machine binds to, and forwards incoming traffic on that port to the machine that has bound it?
Can a machine accept UDP traffic over multiple ports? I left that bit out for simplicity, but WebRTC usually works over a range of ports.
It sounds like this might work quite well actually. Out of curiosity - what happens if two machines in an app bind to the same UDP ports? Is traffic randomly / round-robin distributed? (It doesn’t impact my use case, I can ensure the port bindings are non-overlapping easily enough, just curious).
Yes, it should route to the correct port though I must say I haven’t experimented with UDP on machines so I’m not sure I can help with the other questions though
Hey @lubien - sorry to resurrect this thread, but I’m trying to test this approach out and running into problems, maybe you can see an obvious mistake I’m missing?
I’ve tried following this tutorial as closely as I can, using both a Machines app (fly apps create --machines) and a Nomad app. I cloned the repo, allocated an IPv4 address, then:
$ fly m run . -c fly.toml
But when I try to ping it with any traffic (TCP or UDP), I get no response (I’ve tried the app’s fly.dev subdomain, as well as it’s direct public IP):
echo hello | nc fly-udp-echo.fly.dev 5000
# Nothing, was expecting an echo back
Thanks @jphenow - I actually just followed the linked tutorial exactly, including cloning the Github repo. The only code change was to change the app name in the fly.toml file to my own app name.
One quick update - if I add some logging to the example repo to log when a TCP connection or UDP packet comes in, I see it fire for the TCP connection (though I don’t see the echo back happen from my netcat command). I don’t see any logging fire for the UDP connection.
Hm, I was able to get it working for one of my previous deploys. Identical code deployed for both apps, both list an IPv4 allocated, and both have the same port mappings.
How long does the IP allocation take? Is it allocated once the CLI command returns? What I’m seeing could be explained if the IP allocation takes several minutes to take full effect.