We run a rendering API (Rust/axum behind http_service, single machine in fra). While benchmarking it we found a sharp latency cliff for large request bodies and spent a day root-causing it. Posting the findings here because the behavior matches what was described in Replay with file stream ( Replay with file stream - #11 by pavel ), and because the workaround we hoped for doesn’t work.
The symptom
POST requests; request body sizes from a fast client (Hetzner, same region as the app), median time-to-first-byte:
| Body size | Response time |
|---|---|
| 9.9MB | 546ms |
| 11.2MB | 3417ms |
| 13.1MB | 3366ms |
A sharp step right at ~10 MB, then a near-constant ~3 s penalty regardless of how much larger the body gets. The upload itself completes in ~90 ms.
Root cause
It’s exactly the proxy behavior described by Fly staff in the thread above: fly-proxy buffers up to 10 MB of the request body for retry/replay purposes, and once the buffer is full it stops reading from the client until the instance produces a response or a hardcoded 3 seconds pass. Our app needs the complete body before it can respond (it renders the whole packet), so every >10 MB request from a client that can outrun the buffer pays the full 3-second pause.
What we tried: 103 Early Hints
The thread says the pause ends when the proxy “gets a response from the instance”. We hoped an interim response would count, since it wouldn’t force us to commit a status code before seeing the body. Minimal Go repro:
func handler(w http.ResponseWriter, r *http.Request, hints bool) {
if hints {
w.Header().Set("Link", "</ready>; rel=preload")
w.WriteHeader(http.StatusEarlyHints) // sent before reading the body
}
n, _ := io.Copy(io.Discard, r.Body)
fmt.Fprintf(w, "read %d bytes\n", n)
}
Deployed as a fresh http_service app in fra, probed with 9 MB and 12 MB random bodies from a Hetzner box:
control 9 MB: ttfb=0.48s
control 12 MB: ttfb=3.23s
hints 9 MB: ttfb=0.46s
hints 12 MB: ttfb=3.23s ← identical; the 103 changed nothing
Server-side logs confirm the body read itself takes ~3.1 s for 12 MB either way, and curl -v shows the client only ever receives the final HTTP/2 200 — the proxy appears to swallow the interim response entirely. So 1xx responses neither reach the client nor count as “a response” for releasing the buffer.
Workarounds we considered
- Respond before reading the body (early 200 + streamed body). Works in principle, but forces committing the status code and result headers before the request has been parsed — for an API that reports errors via status codes and post-parse headers, that’s a contract break on exactly the requests that hit the pause.
- handlers = [“tls”] service (skip the HTTP layer entirely). This does avoid the buffering, but gives up client IP headers (PROXY protocol needed), request-level load balancing, transparent retries during deploys and connection keep-alive seems to break the auto-suspend feature .
- Keep bodies under 10 MB (chunked/multi-request API). Works, but pushes proxy internals into the public API design.
@pavel Since you wrote about the original fix.
My question would be “Can we trigger the same behavior that happens after the 3s”, but instantly, ideally in configuration ?
- We have about 50:50 split of requests <10MB and >10MB, so the 3s pentalty is really annoying, while not really bringing any value, if the proxy just waits 3s and then continues.
- Could the 10 MB / 3 s values become configurable per service — or better, an explicit opt-out of fly-replay / request-body buffering for services that never use fly-replay and don’t want retries? (http_service.http_options.request_buffering = false or similar.)
- We basically want Fly-proxy without fly-replay.
- A documentation note under http_service would save the next person the investigation — the current behavior is only discoverable in forum threads.
Happy to share the full measurement setup (k6 + curl probes, all reproducible) if useful.