Autostop is causing problems for my streaming web API

Python
, , ,
7

That’s how it started, but it turned out in the end there was a bug in the infrastructure, :sweat_smile:…

I just tried a small test app with a super-simple streaming text/plain response—no fancy client-side keepalives, WebSockets, or anything—and it was able to stream for the full 10 minutes. (Auto-stop didn’t kick in until 3 minutes after the server had completely finished.)

primary_region = "ewr"

[env]
  RESPONSE_SECONDS = "600"

[http_service]
  internal_port = 8080
  auto_stop_machines = "suspend"
  auto_start_machines = true
  [http_service.concurrency]
    type = "connections"
    soft_limit = 5
    hard_limit = 10

[[restart]]
  policy = "never"

This writes one line every second. There’s a flush after each one, but that’s the only kind of nuance.

And the App Concurrency panel in Grafana correctly showed 1 the entire 10 minutes.

I was testing over Flycast, but generally there’s the same auto-stop behavior there (since it’s the same Fly Proxy).

If both sides (client and server) are idle for a long time, though, then something else in the public Internet might close the connection.