ORD edge latency spike?

Around July 6, 2026, 1:30 AM UTC (or 18:30 in my screenshot) I saw big spikes in latency to my app that I can’t explain based on request volume. Any pointers on debugging this kind of thing?

Fly metrics say the HTTP response times spike to > 5 seconds. This is true when looking at specific instances, or just the ord edge where my requests are mainly going.

Machine memory, CPU, and metrics for managed Postgres look fine to me.

I was reading Fly machines VERY SLOW - All (10+) apps taking 1-3s per patch/page load. - #6 by liamkillingback and wondering if it would be something similar?

I did manage to get this fly request ID 01KWTKQ47KC8EM1M1NKJX21H12-ord for a slow request (albeit roughly an hour afterwards) with the 'flyio-debug: doit header. My network doesn’t provide IPv6 support but the IPv4 mtr looked fine to me.

My database metrics do show an increase in query and queue time right around when the latency spikes, but it doesn’t seem commensurate with the drastic increases in latency. I can’t think of anything on the application side that would’ve caused this, as the latency is consistent across a bunch of tables.

I’m on the basic shared-2x CPU 1 GB memory MPG plan. Is it possible that this is a noisy neighbour? The metrics in the fly.io dashboard all look fine to me though…

Thanks for all the diagnostic info! Super helpful. It looks like there was some packet loss between two of our providers in SJC, where this was routed, that matches the timeline. That has since been addressed so you shouldn’t see this ongoing.

To cover the others bases: app metrics look fine, the ORD edges handled that request quickly, and none of the machines involved were on busy hosts.

amazing thank you! guess I need more practice interpreting the mtr output :^)

If you were running this from your connection then your interpretation was correct; mtr only gets you as far as the ORD edge, which was indeed healthy :slight_smile:

This was loss between machines in SJC, which would need an mtr running on one of your app machines (toward postgres) to spot.