Fascinating detail as ever, thank you for this ongoing effort 
Fast-forwarding six days, to the next Log entries under the rubric of June…
| incid. date |
pub. date |
description |
|---|---|---|
| 06/10 | 06/19 | Two GRU edges OOM |
| 06/11 | 06/21 | ORD transit loss destabilized Managed Postgres clusters |
| 06/11 | 06/21 | Everyone lives on NULL island (Depot, WG placement) |
| 06/12 | 06/21 | Elevated Sprites error rates in SIN |
Apparently, the first was due to an accumulation of pending metrics (waiting for export from those edge nodes).
(And this sunburst overall, 
, was very welcome insight into all four incidents, of course.)
Aside: A recent commit to the official docs repository introduced explicit bounds on metrics, including a limit on the size of your app’s /metrics response, which people had expressed doubts about before.
(Metrics and logs have generally been getting overhauled and having their expectations clarified lately.)
An unusually widespread (by Season Two’s baseline) incident the subsequent Monday:
| incid. date |
pub. date |
description |
|---|---|---|
| 06/15 | 06/25 | Machines API Outage (tokens) |
| 06/17 06/22 |
06/26 | Singapore network outage |
Fly.io themselves briefly lost Machine-control access on that day, and they had to manually bootstrap from a host console (or similar).
Another six days’ hop, to the final full week of June…
| incid. date |
pub. date |
description |
|---|---|---|
| 06/23 | 07/01 | Corrosion OOMs and wedged systemd |
| 06/25 | 07/02 | Corrosion migration caused elevated CPU and routing failures in BOM and NRT |
(Corrosion is the main distributed metadata outlet, handy for things like assessing started/stopped Machine status all around the world.)
The remainder of the month’s Log concluded with an atypically tall vertical bar:
| incid. date |
pub. date |
description |
|---|---|---|
| 06/28 06/29 06/30 |
07/07 | VictoriaMetrics ingestion delays and backlog |
Fortunately, this was fairly localized, but there was at least one forum report of problems with auto-scaling (which relies on timely metrics).
A small graphical overview of the previous month, now that June 2026 is complete in the Log…
See the earlier March grid for a description of the annotations.
The <details> elements below can be expanded, to get each row’s links.
Week of June 01: SSH, Sprites, 6PN, Consul
Week of June 07: ARN, Arcjet, ORD×3, IAD, secrets, GRU, NA net, Depot, WireGuard, Sprites
incid.
datesources symbol box description 06/08 s.f.n,
forum½▪ ARN capacity 06/09 forum │ Arcjet registration 06/09 s.f.n ▪ ORD egress IPs 06/09 s.f.n ½▪ IAD MPG 06/09 s.f.n ─ secrets 06/10 infra,
s.f.n½▪ GRU Fly Proxy 06/11 s.f.n ½─ North America network 06/11 infra,
s.f.n½▪ ORD MPG 06/11 infra ½│ Depot placement 06/11 infra ½│ WireGuard placement 06/11 s.f.n ½▪ ORD network (subset) 06/12 infra,
s.f.n½▪ SIN Sprites Week of June 14: IAD×2, MPG, Machines API, SIN, FRA, logs
Week of June 21: SIN×2, NRT×3, IAD, Corrosion, NA net, LAX, CDG×2, BOM, Depot, EWR
incid.
datesources symbol box description 06/22 infra,
s.f.n▪ SIN network 06/22 s.f.n ▪ NRT network 06/22 infra,
s.f.n▪ SIN network (again) 06/22 s.f.n ▪ NRT network (again) 06/22 forum ½│ IAD Machine creation slowness 06/23 infra │ state propagation 06/24 s.f.n ½▪ North America network 06/24 s.f.n ½▪ LAX 6PN 06/24 forum │ CDG WireGuard node 06/25 forum ▪ CDG WireGuard node (continued) 06/25 forum,
forum′▪ CDG WireGuard node (again) 06/25 infra,
s.f.n½▪ BOM Fly Proxy & Machines API 06/25 infra,
s.f.n½▪ NRT Fly Proxy & Machines API 06/25 s.f.n ½─ Depot 06/26 s.f.n ▪ EWR IPv6 Week of June 28: metrics×2, SIN, NRT
(In the expanded tables’ second columns, “s.f.n” is status.flyio.net, the real-time status page, which serves as a second-tier source in this context.)
The first third of summer saw a moderate increase in heftier boxes, although not to an alarming extent.
Of the more prominent cases…
A brief but global Machines API outage became a topic of significant comment in the middle of the month, June 15. The widest (graphically) incident of Season Two, thus far.
North American networking glitched on June 11 and on June 24, leading to tremulous service across a wide area (as well as a narrower after-shock in ORD on the earlier date). It’s not clear whether these were re-occurrences of the BGP faults from the opening March installment of Season Two, or some new way of networking not working.
Delays in processing metrics spanned most of June 28–30. Across those three days, there were clogs at certain points in the pipeline, resulting in localized stale statistics and hence bad auto-scaling behavior for some apps (among other things).
Sprites URLs and dashboard went awry on June 2 (actually starting a bit in June 1), possibly related to authentication. This was a rare case of a forum-only incident.
Onward to the first week of July…
| incid. date |
pub. date |
description |
|---|---|---|
| 07/01 | 07/14 | Runaway cleanup job filled Redis (GraphQL, dashboard) |
| 07/02 | 07/14 | Certificate issuance outage |
| 07/03 | 07/14 | Power supply failure in ORD |
That last one was a bit more involved than it sounds at first (!).
Later, after five consecutive quiet days, the registry became overloaded, and a nearly simultaneous builder capacity crunch led to considerable comment in the forum…
| incid. date |
pub. date |
description |
|---|---|---|
| 07/09 | 07/16 | A couple of deployment issues (Depot, registry) |
The registry half was resolved with a tiny droplet of forum lore: “our TAP interfaces, for one reason or another, start to struggle a bit under higher ultilization (> 1Gbps)”. This unintended networking bottleneck has been a kind of folknerd knowledge for a while, but the above, to the best of my recollection, is its first appearance in the official record.
Similarly with the useful plumbing fact that “machines can auto-migrate on start when capacity error happens, but this is explicitly not enabled for machines and volumes as large as builders.”
Moreover, there are details on planned improvements to the Depot builder infrastructure, moving closer to the “authoritative fix” that many well wishers have been hoping for…
Forward one week to a new networking mystery and then a swan song on a familiar pattern…
| incid. date |
pub. date |
description |
|---|---|---|
| 07/14 | 07/21 | SJC worker hosts locked up and rebooted |
| 07/16 | 07/24 | Vault throws one last tantrum on its way out the door (Fly Proxy, GraphQL) |
The second row may have also been the cause of the Machines API glitch mentioned on the real-time status page that same hour.
Small bump to prevent thread auto-close, 
…
I turned off auto-close after mayailurus’s last message 
The subsequent week began with an unusual Sunday empty-query puzzle…
| incid. date |
pub. date |
description |
|---|---|---|
| 07/19 | 08/03 | Missing app logs because we forgot a systemctl restart |
| 07/20 | 08/03 | tkdb primary host outage caused token validation failures |
The second row includes an interesting detail about the operational differences between tkdb and PetSem. (Those two distributed databases are very closely related.)
Aside: The first row perhaps referred in part to a forum thread begun late on the 18th.
A small graphical overview of the previous quarter, now that May through July 2026 are complete in the Log…
See the earlier March grid for a description of the annotations.
The <details> elements below can be expanded, to get the July image’s links.
Week of July 01: SYD, GraphQL, dashboard, NRT, certs, ORD×2
Week of July 05: Depot, registry
Week of July 12: LHR, DFW, GRU, SJC, Fly Proxy, Machines API, YYZ, Grafana
Week of July 19: Grafana, BOM, Sprites, Machines API, MPGv1, DFW, Redis, JNB, NRT, OAuth
incid.
datesources symbol box description 07/19 infra,
forum│ Grafana logs (continued) 07/19 s.f.n ▪ BOM egress v6 IPs 07/20 infra,
forum│ Grafana logs (continued) 07/20 forum │ Sprite OAuth-based connectors 07/20 infra,
s.f.n─ Machines API 07/20 s.f.n ½│ MPGv1 (segue) 07/21 infra,
forum│ Grafana logs (continued) 07/21 forum,
forum′│ Sprite OAuth-based connectors (continued) 07/22 infra,
forum│ Grafana logs (continued) 07/22 forum,
forum′│ Sprite OAuth-based connectors (continued) 07/22 s.f.n ▪ DFW network 07/22 forum ▪ Redis in DFW 07/22 forum ½│ JNB capacity 07/23 forum,
forum′│ Sprite OAuth-based connectors (continued) 07/23 forum ½│ JNB capacity (continued) 07/23 forum ▪ NRT egress IPs 07/24 forum,
forum′│ Sprite OAuth-based connectors (continued) 07/24 forum,
forum′▪ dashboard(?) OAuth connections 07/25 forum,
forum′│ Sprite OAuth-based connectors (continued) Week of July 26: Sprites, Machines API, dashboard×2, CDG, FRA
incid.
datesources symbol box description 07/26 forum,
forum′│ Sprite OAuth-based connectors (continued) 07/27 forum,
forum′│ Sprite OAuth-based connectors (continued) 07/28 forum,
forum′│ Sprite OAuth-based connectors (continued) 07/29 forum,
forum′│ Sprite OAuth-based connectors (continued) 07/29 s.f.n ½▪ Machines API 07/29 s.f.n ½▪ dashboard 07/31 s.f.n ▪ Outbound email via dashboard 07/31 s.f.n ½▪ CDG capacity 07/31 forum ▪ FRA freebie Consul cluster
(In the expanded tables’ second columns, “s.f.n” is status.flyio.net, the real-time status page, which serves as a second-tier source in this context.)
The second third of summer returned to a calmer atmosphere early on but then the winds picked back up in its final weeks.
Of the more prominent things blown over…
The Machines API went down for an hour worldwide on July 20, due to a failure of the tokens service’s primary node.
A rare global Fly Proxy anomaly July 16 was the tune of Vault’s swan song. This only affected uncached SSL certificates—as well as consonant operations of GraphQL and friends.
A brief registry overload coincided with an IAD-specific builder capacity crunch on July 9. The Depot builders continued to be rather sensitive to regional congestion.
A 10-day Sprites connectors glitch was etched into the month’s final weeks, an additional forum-only incident in this corner. (As others have mentioned, it would be nice to know more about what goes awry in Sprites-land.)
[The parallel (for a while) 4-day logs problem did have an Infra Log entry, however.]
Onward to the first incidents of August…
| incid. date |
pub. date |
description |
|---|---|---|
| 08/01 | 08/11 | IAD MPGv2 Patroni etcd lease-expiration storm |
| 08/03 | 08/11 | BGP misconfiguration dropped ingress traffic |
| 08/04 | 08/11 | Certificate issuance delays from Sidekiq overload |
This was much appreciated detail, all around!
Aside: Fly.io mentioned deep in a forum thread recently that they had stealthily formed a group in June(-ish) to keep tabs on reliability. The possibility of the long-awaited automated system statistics appearing as a consequence was hinted at shortly afterward.
(There’s long been a rough consensus among both users and Fly.io staff that there’s a gap in the current incidents / uptime-reporting setup. The 2024 post proposed “detailed, unfiltered metrics” published online, without the need for manual intervention or judgment calls each time. This would really fill in the missing span of the spectrum between handcrafted status page updates and the retrospective Infra Log, 
…)
The weekend started with more distributed database raindrops…
| incid. date |
pub. date |
description |
|---|---|---|
| 08/08 | 08/18 | Global Corrosion lag caused Machines API app-not-found errors |
| 08/10 | 08/19 | Egress IP issues in SJC |
The second incident’s bug, or something with very similar symptoms, still seems to be affecting some people.
Aside: People who are interested in network performance minutiae might also want to read the following recent forum thread…
The phenomenon of wedging due to systemd dependencies made an encore appearance with…
| incid. date |
pub. date |
description |
|---|---|---|
| 08/13 | 08/24 | Stuck nftables reload blocked flyd startup causing API errors |
| 08/14 | 08/25 | Petsem primary crash loop from corrupted volume |
| 08/20 | 08/27 | MPGv1 clusters in ORD-0 failed to restart |
| 08/20 | 08/28 | Some API endpoints briefly rejected flyctl requests |
Also, that fourth one was triggered by a shift to rejecting old-style OAuth tokens. These are the ones that were deprecated back around May 2025, I think.
(The introduced filter briefly balked at certain other combinations, as well.)
Metrics processing backlogs briefly returned, as the final month of summer started to reach its end…
| incid. date |
pub. date |
description |
|---|---|---|
| 08/20 | 08/31 | Development node advertised Fly public DNS anycast IPs and broke it |
| 08/24 | 09/02 | Metrics backlog |
| 08/26 | 09/04 | WireGuard gateways broke after schema migration mismatch |
Fortunately, it didn’t extend into multiple days this time.
The first August in the Second Season of the Infra Log then closed with an interesting behind-the-scenes storage detail—along with its effect on a Sprite’s lifecycle…
| incid. date |
pub. date |
description |
|---|---|---|
| 08/30 | 09/09 | Stuck background job queue blocked Sprite deletions |
| 08/31 | 09/09 | Fly-proxy HTTP/2 routing broke due to missing Consul service rows |
There was another metadata database glitch as well, possibly one of the last(?) of Consul’s remaining duties.
oh, on this topic: we just migrated incident tracking platforms so the next few infra logs will be a bit delayed as we figure out hooking everything up again.
A small graphical overview of the previous month, now that August 2026 is complete in the Log…
See the earlier March grid for a description of the annotations.
The <details> elements below can be expanded, to get each row’s links.
Week of August 01: MPGv2
Week of August 02: BGP, certs, GRU, IAD, Corrosion
Week of August 09: Corrosion, SJC, Machines API, IPv6, secrets
Week of August 16: ARN, YYZ, ORD, tokens, DNS, egress IPs
Week of August 23: egress IPs, LAX, metrics, WireGuard, GRU×2
Week of August 30: Sprites, ORD, Fly Proxy
(In the expanded tables’ second columns, “s.f.n” is status.flyio.net, the real-time status page, which serves as a second-tier source in this context.)
The final third of summer calmed down even more, although again with strong winds in a few places.
Of the more prominent things blown over…
Corrosion, the house subsystem for distributed metadata, was down for around 7 hours, beginning just slightly before midnight (00:00 UTC) August 8. This was Season Two’s first box that rendered both long and wide.
For a few minutes on August 20, tokens went flat globally in relation to flyctl, due to false positives by a new filter.
A glitch with BGP occurred on a continent-wide scale once more, although fortunately only for a short interval of the month’s third day. This time Europe bore the brunt of the outage.