Throttling and DDoS protection for fly app

1

I have a backend REST service deployed on Fly. How do you recommend going about implementing API request throttling (using leaky bucket, for example) for each customer at the load balancer level so the boxes don’t get overwhelmed? Do I need to run my own nginx reverse proxy? We had an event today where one user spammed the service with a ton of requests, which generated a ton of logs from the proxy (which I presume is the Fly.io internal reverse proxy) from the iad region:

2023-03-20T23:56:36.902 proxy[a11f988a] sjc [warn] Instance reached connections hard limit of 25

2023-03-20T23:56:36.874 proxy[a11f988a] iad [warn] Failed to proxy HTTP request (error: no known healthy instances found for route tcp/443. (hint: is your app shutdown? is there an ongoing deployment with a volume or using the 'immediate' strategy? if not, this could be a delayed state issue)). Retrying in 1000 ms (attempt 30)
2023-03-20T23:58:43.615 proxy[302d9500] lhr [warn] Instance reached connections hard limit of 25

2023-03-20T23:58:43.656 proxy[cba9b5df] iad [warn] Failed to proxy HTTP request (error: no known healthy instances found for route tcp/443. (hint: is your app shutdown? is there an ongoing deployment with a volume or using the 'immediate' strategy? if not, this could be a delayed state issue)). Retrying in 1000 ms (attempt 10)

2023-03-20T23:58:43.663 proxy[cba9b5df] iad [warn] Failed to proxy HTTP request (error: no known healthy instances found for route tcp/443. (hint: is your app shutdown? is there an ongoing deployment with a volume or using the 'immediate' strategy? if not, this could be a delayed state issue)). Retrying in 1000 ms (attempt 30)

2023-03-20T23:58:43.667 proxy[cba9b5df] iad [warn] Failed to proxy HTTP request (error: no known healthy instances found for route tcp/443. (hint: is your app shutdown? is there an ongoing deployment with a volume or using the 'immediate' strategy? if not, this could be a delayed state issue)). Retrying in 934 ms (attempt 40)

2023-03-20T23:58:43.672 proxy[cba9b5df] iad [warn] Failed to proxy HTTP request (error: no known healthy instances found for route tcp/443. (hint: is your app shutdown? is there an ongoing deployment with a volume or using the 'immediate' strategy? if not, this could be a delayed state issue)). Retrying in 1000 ms (attempt 10)
2

Hi! We have support for limiting the number of connections to specific instances of an app.

1 Like
3

Two questions:

  1. How can I throttle based on individual users’ tps? I want to allow each user to only be able to hit the API at N requests per second.
  2. I already have the connection limiting settings applied to my app (hard_limit = 25, soft_limit = 20, type = “connections”). Why am I still seeing these proxy errors from fly?
2023-03-20T23:58:43.663 proxy[cba9b5df] iad [warn] Failed to proxy HTTP request (error: no known healthy instances found for route tcp/443. (hint: is your app shutdown? is there an ongoing deployment with a volume or using the 'immediate' strategy? if not, this could be a delayed state issue)). Retrying in 1000 ms (attempt 30)
4

  1. We don’t offer client request ratelimiting right now. You’ll have to do that in your app (or in a nginx proxy, as you mentioned).

  2. Those proxy errors (when hitting hard limit) are a bug. Our proxy filters instances based both on the instance health and the hard limit, and prints that error if either of those don’t return any available instances. We should fix that.

1 Like
5

What is the impact on the Fly user of the bug and those proxy errors? Does that mean even after each host hits the hard limit that requests are still routed to it?

6

It’s just a display bug. It prints a log line when it shouldn’t. It’s confusing (hence you asking about it), but it doesn’t cause the proxy to route to instances it shouldn’t.

7

Got it, thank you! Right now since we’re getting spammed by requests we’re getting 10s of thousands of those lines which is cluttering our logs and makes it hard to debug, and is costing us extra as we’re persisting everything in logtail.

1 Like
8

Is something like an out of box rate-limiter at Fly’s proxy layer something that’s on the roadmap?

9

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.