Machine receiving mysterious internal (172.16.x.x) HTTP requests every second

[CostPal]

Someone else is having the same issue ( Machine receiving internal http requests every 3 seconds never stops ) but they deleted their post for some reason.

Seems like this started around 18:00 PST today.

Not sure what these requests are or where they are coming from. They are polluting our logs and keeping our machines alive (we have them configured as suspended).

[info] {“level”:30,“time”:1761264302201,“pid”:671,“hostname”:“[redacted]”,“reqId”:“req-hs”,“req”:{“method”:“GET”,“url”:“/”,“host”:“[redacted]”,“remoteAddress”:“::ffff:172.16.7.2”,“remotePort”:37134},“msg”:“incoming request”}

[mayailurus]

Hm… The health checks use the 172.16.* private address range, or at least they have in the past. Could you log more of the request header (e.g., the user-agent)?

In general, internal traffic (apart from Flycast) won’t keep Machines awake, though. The Grafana metrics (“app concurrency”) should tell you what the Fly Proxy thinks your external traffic really is…

Also, it’s a good idea to post your entire fly.toml in this situation. (Feel free to * out any names that you consider sensitive, but show the full structure.)

Hope this helps a little!

[Bonadio]

Hi, I am the one with the same problem, I created the post but deleted because we thought it was an error in our app.

I have the same configuration as another app that is working fine but this one is receiving this checks every 3 senconds, here is the screen shot

here is the config

[http_service]
internal_port = 8080
force_https = true
auto_stop_machines = “stop”
auto_start_machines = true
min_machines_running = 0
processes = [‘app’]

[[vm]]
memory = “1gb”
cpu_kind = “shared”
cpus = 2

[mayailurus]

If you manually stop the Machine (fly m stop), does it wake back up?

[Bonadio]

@mayailurus

fly m stop

Yes

[CostPal]

Pretty sure it isn’t the health check, because I have mine setup at ‘/health’ URL, but the requests are all to / so they are all 404.

Fly.toml:

app = ‘[redacted]’
primary_region = ‘ord’

[build]

[env]
NODE_ENV = “production”

[http_service]
internal_port = 3000
force_https = true
auto_stop_machines = ‘suspend’
auto_start_machines = true
min_machines_running = 0
processes = [‘app’]

[[http_service.checks]]
grace_period = “30s” # Wait 30s after start before checking
interval = “30s” # Check every 30s
method = “GET”
timeout = “10s” # Timeout after 10s
path = “/health”

[http_service.concurrency]
type = “requests”
soft_limit = 2
hard_limit = 3

[[vm]]
memory = ‘2gb’
cpu_kind = ‘performance’
cpus = 1

[Bonadio]

The other thing is that my app is a python Fastapi and it is listening to port 8080 on 0.0.0.0 but every time I deploy I get this message, but the app is running just fine

WARNING The app is not listening on the expected address and will not be reachable by fly-proxy.596c25e78 reached started state

You can fix this by configuring your app to listen on the following addresses:

• 0.0.0.0:8080

Found these processes inside the machine with open listening sockets:

PROCESS | ADDRESSES

-----------------*---------------------------------------

/.fly/hallpass | [fdaa:b:ebda:a7b:50d:b6aa:5226:2]:22

[joehelper]

I’m still seeing the same thing. Out of the blue, a bunch of the exact same requests with 172.16…. GET HTTP/1.1 . All started about 30 minutes ago.

To me it seems like they’re messing around with the health checks and the new log doctor, because I’m seeing the exact same thing in a stg and a prod env, and I didn’t change anything.

[CostPal]

Recorded headers and it’s “user-agent":"Consul Health Check”

Again, it’s attempting to get / route, but my health checks are configured for /health path. Even if I remove the health check section from my fly.toml, I still get these requests.

[mayailurus]

@joehelper

the new log doctor

I’m pretty sure that’s purely in the dashboard UI (not the infrastructure), , but thanks for confirming the request pattern that @CostPal and @Bonadio were seeing!

[CostPal]

Hope someone from Fly can chime in, seems like an issue on their end then.

[joehelper]

Agreed, very curious why there hasn’t been any mention of it yet in their status page.

[mayailurus]

It typically takes a bit of time for an initial investigation. (Which I believe may already be in progress, actually.)

Moreover, they officially only post global incidents on status.flyio.net, not ones that affect just a handful of hosts—although they’ve been relaxing that filter a little lately…

[kaysonargyle]

Glad to know I’m not crazy

[joehelper]

Is it safe to assume those are calls from fly for health checks since they come from 172.16 .. ?

[mayailurus]

That would be the first guess, I think, given the User-Agent that @CostPal logged. (That specific “Consul Health Check” string has been cited a few times in the forum in the past.) But there are some other headers you could look at to get more information:

https://fly.io/docs/networking/services/#http-handler

[bglw]

Hi all, apologies! This was a very eager doctor indeed, and these checks were being made while you (or someone in your org) had the dashboard on on your app.

We have disabled these checks from the system for now.