Something is spamming my server

Hi,

I deployed my site built with Remix to 13 regions including fra. We set the --max-per-region to 1 to make sure all instances are well distributed.

When I check the log and also the metrics today, the numbers look suspicious.

Something keeps requesting our blog page with a really weird URL. We don’t have that weird query param

GET /blog/start-using-opensearch-with-nodejs?a=&b=&e=&e=%22and%2F%2A%2A%2Fextractvalue%281%2Cconcat%28char%28126%29%2Cmd5%281494505220%29%29%29and%22&j=m&j=i&n=r&o=c&t=&version=v3

The fra instance uses up all the CPU while other instances are using only 20% of the CPU

This makes other numbers such as response times pretty high.

It only happens with the fra instance.

I don’t know how we should deal with this issue. Any suggestion? Is there any way that we can block that IP from spamming us?

Thanks

If you decode the query string you’ll get the following:

That’s an attack payload, similar to the examples here: someone is probing your service to determine if your service is vulnerable to an attack. There’s nothing to be worried about: if you expose a service to the internet, people will try and run all sorts of attacks against it. The presence of these requests does not indicate vulnerability. The most common ones (in my experience) involve things like Wordpress, where you’ll see lots of requests to endpoints like /wp-login.php.

You could, conceivably, block the attacker based on their IP address, but it’s important to understand that this is a constant problem you’ll face, from hundreds if not thousands of potential attackers, and it’s not specific to Fly. The most common solution is to use some sort of WAF (Web Application Firewall) which is a system (often a hosted service) that has an actively managed blocklist, and so known attacks are blocked before they can even make it to your server, but it does mean that you’ll need to add something between your service and your users (which can introduce latency and negate a lot of the value of having servers close to your users!).

CloudFlare’s WAF is included in their $20/month plan, but there’s bunch of self-hosted options you could consider – this “Awesome WAF” repository has a bunch of options you could look at.

If you’re seeing an impact on your service’s performance from the vulnerability probing, it’s probably a sign that your application is either vulnerable to a specific type of attack (very unlikely if you’re using Remix because of the limited surface area) or your application has logic that is causing a bunch of work to be done when there’s a request that doesn’t match any expectations. A good first step is to emulate these requests locally, and observe the behaviour of your service: what information is exposed in response to the requests? Are more database queries than you expect being performed?

Personally, I don’t worry at all about these sort of vulnerability probes and do not use a Web Application Firewall unless it’s mandatory (i.e: it can be necessary for SOC 2 compliance) because they’re not particularly effective for custom built applications (i.e: they’re great for protecting against well known Wordpress exploits, but much less helpful in protecting against someone specifically targeting a service you built) and they have a performance penalty. If you’re running a mostly static site (like a blog) then a WAF would be overkill and focusing on the performance issues instead would be a good use of time. I use k6 to test the performance of my services under load, it’s very helpful!

3 Likes

To add to that, Cloudflare offers umetered rate-limiting for free. So, if you do not expect a single client IP to call your APIs more than 20 times per minute, say, you can add that rule to your Cloudflare zone (for that endpoint). Note though, this means you’d have to effectively proxy (orange cloud) traffic through Cloudflare (adding a bit of unnecessary latency) on its way to Fly’s network.

Personally speaking, vulnerabilities or not, I gate things like sign-up forms, log-in forms, email subscriptions behind rate-limiters (or glue up one myself atop FOSS libraries). I don’t (because users over VPNs / Mixnets like Tor have a hard time with it), but you could enable Turnstile (Cloudflare’s take on Captcha) to ward off bots.

1 Like

Are these coming through an HTTPS endpoint?

If you add the force_https option in your fly.toml, they’ll get a redirection and then our proxy will reject them if they can’t provide a SNI (unless you have default_self_signed TLS handler option set to true).

For example:

[[services.ports]]
  handlers = ["http"]
  port = 80
  force_https = true

(sorry to hijack the thread…)

Re: Terminate multiple TLS domains, with both Fly and custom certificates

So, the usecase mentioned in that post (Fly terminating TLS for app-name.fly.dev but letting other SNIs through) is possible if default_self_signed is set to true?

Yes, but they will be served a self-signed certificate. No clients will accept them as “trustable”.

1 Like

Thanks, I already set force_https to true

Here is my fly.toml

app = ""
kill_signal = "SIGINT"
kill_timeout = 5
processes = []

[env]
  PORT = "8080"
  FLY = "true"

[experimental]
  allowed_public_ports = []
  auto_rollback = true

[[services]]
  internal_port = 8080
  processes = ["app"]
  protocol = "tcp"
  script_checks = []

  [services.concurrency]
    hard_limit = 200
    soft_limit = 150
    type = "connections"

  [[services.ports]]
    force_https = true
    handlers = ["http"]
    port = 80

  [[services.ports]]
    handlers = ["tls", "http"]
    port = 443

  [[services.tcp_checks]]
    grace_period = "1s"
    interval = "15s"
    restart_limit = 0
    timeout = "2s"
    
  [[services.http_checks]]
    interval = 10000
    grace_period = "600s"
    method = "get"
    path = "/healthcheck"
    protocol = "http"
    timeout = 5000
    tls_skip_verify = false
    [services.http_checks.headers]

If I ignore the issue, I wonder if the attack crashes the region due to hitting the CPU limit?

My site is built with Remix and served by a NodeJS server. I will try to limit the request rate with this package express-rate-limit - npm

Does Fly have some kind of protection out of the box? @jerome

So if force_https isn’t enough, then it’s probably a more targeted attempt at an exploit.

As of now we do not have anything built-in for this. We’re working on a design for a “routing config” that will give people a lot more freedom.

For now your best bet would be to add express middleware or a thin proxy, that can block what you want to block, running inside the same instance.

1 Like