Hi all,
I’ve noticed a strange pattern with my Fly app recently. At seemingly random times of the day the app concurrency metric will drop all the way to zero, then nearly exactly 40 minutes later it will come back to normal. (These are long-lived WebSocket connections.)
The strange thing is that the connections are still active during this 40-minute period. My app logs when a connection is established and terminated, so I checked against those and the pattern is clear: nothing happens when the concurrency metric drops, but when it comes back up my app sees every single connection close then come back immediately, resulting in a storm.
So the connections do drop, but nothing actually happens until the tail end.
I annotated the beginning and end of the event here:
And you can see the storm of connections:
I can also confirm that all appears well on the client side until the end.
This has happened several times over the past few days. Here are the most recent instances (UTC):
2026-06-10 14:32 - 15:10
2026-06-10 15:45 - 16:25
2026-06-10 19:18 - 19:58
2026-06-11 12:15 - 12:55
The very first instance of this occurred on 2026-06-03 between 22:24 and 23:04 UTC and has become more frequent since then. I can’t correlate it with anything I’ve done on my end or other conditions in the app or services it depends on.
To me it seems like something is going wrong with Fly Proxy. It’s very suspicious that each event is pretty much exactly 40 minutes (give or take a few mins due to metrics lag).
Hi TJ,
This is normal and happens when the Fly proxy is redeployed. A new proxy instance comes online to serve new requests, but the old instance remains around with the connections it was already handling, so they have a chance to finish and close down.
This works great for short-ish connections like the typical HTTP request but is more of a tricky thing for longer-lived ones (e.g. connections to Postgres, Redis, and of course, websockets).
What we recommend here is having your app recycle connections unconditionally after about 10-15 minutes, so when the old proxy instance eventually shuts down, it doesn’t kill any connections it was still handling.
Note that for websockets, the Fly proxy will correctly send 1001 GOAWAY frames with a reason of “restarting” to both server and client, so they can handle this gracefully: e.g. if the client sees this, it could disconnect and immediately reconnect, and the server should be aware that the client might come back and keep enough state to continue handling the task it was doing.
Depending on what your WS connections are doing, if you can find a moment where they’re basically idle and you can then stop/reconnect, you should not have any problems with this.
Thanks, that makes sense to me.
A few questions:
- Is that 40-minute period the duration where the old Fly Proxy is still running, waiting for my app’s connections to drain then times out?
- Is there a mechanism for my app to detect when this happens so that it can start draining connections slowly rather than Fly Proxy killing them all at once?
I would also appreciate if the old instance of Fly Proxy kept reporting metrics until it actually dies.
My app seems to handle the reconnection storm like a champ, so that part isn’t really a concern. It’s mostly about the metrics being accurate.
Hi TJ:
- It is, but we ask that folks stick to a 10-15 minute lifetime - the sooner the old proxy releases all its connections, the sooner it can die and release more resources.
- Not really - part of the point here is that other than the disconnects (which any app should handle - it’s the Internet, baby, connection stability is not guaranteed - and on websockets we do give you a nice GOAWAY so you know when things are shutting down) this should be transparent to applications.
Which metrics are you having trouble with, other than concurrency?
we ask that folks stick to a 10-15 minute lifetime
Sure. But even so, there would still be up to a 10-15 minute period where the app concurrency metric is incorrect.
which any app should handle - it’s the Internet, baby, connection stability is not guaranteed
Definitely 
I wrote both the client and server with resilience in mind so they both actually handle this pretty gracefully.
Which metrics are you having trouble with, other than concurrency?
Just concurrency as far as I can tell. It’s not a super big deal—just a nice-to-have—since my app reports its own connections-related metrics and I can use those instead.
So, mystery solved, but I would still be happy to see Fly Proxy instances reporting concurrency up until it actually dies so I can rely on the metric to be accurate.
Thanks!
Hm, I checked and there are additional metrics affected by this: the fly_edge_data_in/out metrics report nothing for the connections still on the old Fly Proxy instance:
Again, not a big deal since I don’t personally use it very often, but maybe it affects billing.
Hey @tjhorner,
I’ve spoken to our proxy team and they’ve told me that this is on the roadmap but there’s no date available.
For billing it seems like we’d be ever so slightly under-billing for these metrics whilst the proxy deploy happens, so right now its in your favor but this’ll be fixed.
Thanks for letting us know!
