Suspended machines woken up by PUT from fc_api every few mins

, ,
1

I spot fc_api logs every few mins:

This means, in absence of any requests, the machines gets woken up and remains idle for 5mins/7mins (as there are no requests). Then, suspends for a few mins before being woken up by fc_api.

# 15:36:13
2024-09-26T15:36:13Z app[9e784e671b4283] bom [info]2024-09-26T15:36:13.861268024 [01J8N5T2CJBACAKJ8ZXY6KY7DB:fc_api] The API server received a Put request on "/logger" with body "{\"log_path\":\"logs.fifo\",\"level\":\"info\"}".

...

# 15:50:05
2024-09-26T15:50:05Z app[9e784e671b4283] bom [info]2024-09-26T15:50:05.020806485 [01J8N5T2CJBACAKJ8ZXY6KY7DB:fc_api] The API server received a Put request on "/logger" with body "{\"log_path\":\"logs.fifo\",\"level\":\"info\"}".

...

# 15:57:55
2024-09-26T15:57:55Z app[9e784e671b4283] bom [info]2024-09-26T15:57:55.951970638 [01J8N5T2CJBACAKJ8ZXY6KY7DB:fc_api] The API server received a Put request on "/logger" with body "{\"log_path\":\"logs.fifo\",\"level\":\"info\"}".

Is fc_api a Fly thing? If so, how can I stop it from waking my suspended machines up?

2

I don’t think fc_api is a fly thing (could be wrong) because I’ve never seen that thing in any of my apps. Must be something specific to your app.

EDIT: I just checked my logs and now I’m seeing fc_api… so it must be new.

image
image2757×207 18.7 KB

9/17/2024 was when it was first introduced, at least for me.

1 Like
3

To fly-proxy, my app is TCP-only (ie, does not rely on Fly’s HTTP handlers). So, it is weird to see PUT requests. Since these logs aren’t printed by my code (as far as I looked), it could be a Fly process / service?

4

Something else must be waking up your machine. I don’t think fc_api itself is triggering it.

5

Sorry, I should have shared the full log:

# suspending at 16:18:23
2024-09-26T16:18:23Z proxy[9e784e671b4283] bom [info]App has excess capacity, autosuspending machine 9e784e671b4283. 0 out of 1 machines left running (region=bom, process group=app)
2024-09-26T16:18:24Z app[9e784e671b4283] bom [info]Virtual machine has been suspended
2024-09-26T16:18:25Z health[9e784e671b4283] bom [error]Health check on port 10000 has failed. Your app is not responding properly. Services exposed on ports [853, 10000] will have intermittent failures until the health check passes.
2024-09-26T16:18:28Z health[9e784e671b4283] bom [error]Health check on port 8080 has failed. Your app is not responding properly. Services exposed on ports [443, 8080] will have intermittent failures until the health check passes.
2024-09-26T16:18:29Z health[9e784e671b4283] bom [error]Health check on port 8888 has failed. Your app is not responding properly.

# starting at 16:27:03 (no reqs before this)
2024-09-26T16:27:03Z proxy[9e784e671b4283] bom [info]Starting machine
2024-09-26T16:27:03Z app[9e784e671b4283] bom [info]2024-09-26T16:27:03.699430338 [01J8N5T2CJBACAKJ8ZXY6KY7DB:main] Running Firecracker v1.7.0

# fc_api at 16:27:03
2024-09-26T16:27:03Z app[9e784e671b4283] bom [info]2024-09-26T16:27:03.700047413 [01J8N5T2CJBACAKJ8ZXY6KY7DB:fc_api] The API server received a Put request on "/logger" with body "{\"log_path\":\"logs.fifo\",\"level\":\"info\"}".
2024-09-26T16:27:03Z runner[9e784e671b4283] bom [info]Machine started in 231ms
2024-09-26T16:27:03Z proxy[9e784e671b4283] bom [info]machine started in 235.597918ms
2024-09-26T16:27:03Z proxy[9e784e671b4283] bom [info]machine became reachable in 7.725773ms
2024-09-26T16:27:05Z app[9e784e671b4283] bom [info]ERROR stdout to vsock zero copy err: Broken pipe (os error 32)

# health checks
2024-09-26T16:27:06Z health[9e784e671b4283] bom [info]Health check on port 10000 is now passing.
2024-09-26T16:27:06Z health[9e784e671b4283] bom [info]Health check on port 8080 is now passing.

...

# no reqs after this
6

Looks like your fly.toml has multiple ports? 853/1000/443/8080. One of those is getting pinged and waking up your app.

7

I checked the logs, there were no requests to those ports. The logs on health checks we see are also from Fly sent after the machine is woken up.

8

Maybe it’s a :ghost:

9

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.