PSA: Postmortem(s) rollup

Fresh Produce
,
47

A small graphical overview of the previous month, now that June 2026 is complete in the Log…

unofficial fly.io incidents grid for june 2026
(see the <details> sections below for clickable links)450×413 12.8 KB

See the earlier March grid for a description of the annotations.

The <details> elements below can be expanded, to get each row’s links.

  • Week of June 01: SSH, Sprites, 6PN, Consul

    incid.
    date    		sourcessymbolbox description
    06/01s.f.n▪SSH & OIDC due to Consul(?)
    06/01forum,
    forum′    		▪Sprite URLs & dashboard
    06/02forum,
    forum′    		│Sprite URLs & dashboard (continued)
    06/04infra,
    s.f.n,
    forum    		│post-migration 6PN translations
    06/04infra,
    s.f.n    		▪Consul freebie cluster due to above 6PN

  • Week of June 07: ARN, Arcjet, ORD×3, IAD, secrets, GRU, NA net, Depot, WireGuard, Sprites

    incid.
    date    		sourcessymbolbox description
    06/08s.f.n,
    forum    		½▪ARN capacity
    06/09forum│Arcjet registration
    06/09s.f.n▪ORD egress IPs
    06/09s.f.n½▪IAD MPG
    06/09s.f.n─secrets
    06/10infra,
    s.f.n    		½▪GRU Fly Proxy
    06/11s.f.n½─North America network
    06/11infra,
    s.f.n    		½▪ORD MPG
    06/11infra½│Depot placement
    06/11infra½│WireGuard placement
    06/11s.f.n½▪ORD network (subset)
    06/12infra,
    s.f.n    		½▪SIN Sprites

  • Week of June 14: IAD×2, MPG, Machines API, SIN, FRA, logs

    incid.
    date    		sourcessymbolbox description
    06/14s.f.n,
    forum    		│IAD network
    06/15s.f.n▪MPG provisioning
    06/15infra,
    s.f.n,
    forum    		─Machines API
    06/17infra,
    s.f.n    		▪SIN network
    06/17forum½│IAD edge node(?)
    06/17forum½▪FRA edge node
    06/19s.f.n▪Historical logs

  • Week of June 21: SIN×2, NRT×3, IAD, Corrosion, NA net, LAX, CDG×2, BOM, Depot, EWR

    incid.
    date    		sourcessymbolbox description
    06/22infra,
    s.f.n    		▪SIN network
    06/22s.f.n▪NRT network
    06/22infra,
    s.f.n    		▪SIN network (again)
    06/22s.f.n▪NRT network (again)
    06/22forum½│IAD Machine creation slowness
    06/23infra│state propagation
    06/24s.f.n½▪North America network
    06/24s.f.n½▪LAX 6PN
    06/24forum│CDG WireGuard node
    06/25forum▪CDG WireGuard node (continued)
    06/25forum,
    forum′    		▪CDG WireGuard node (again)
    06/25infra,
    s.f.n    		½▪BOM Fly Proxy & Machines API
    06/25infra,
    s.f.n    		½▪NRT Fly Proxy & Machines API
    06/25s.f.n½─Depot
    06/26s.f.n▪EWR IPv6

  • Week of June 28: metrics×2, SIN, NRT

    incid.
    date    		sourcessymbolbox description
    06/28infra,
    s.f.n    		½│metrics
    06/29infra,
    s.f.n,
    forum    		½│metrics (again)
    06/30infra,
    s.f.n,
    forum    		½│metrics (continued)
    06/30s.f.n½▪SIN egress IPs
    06/30s.f.n½▪NRT egress IPs

(In the expanded tables’ second columns, “s.f.n” is status.flyio.net, the real-time status page, which serves as a second-tier source in this context.)

The first third of summer saw a moderate increase in heftier boxes, although not to an alarming extent.

Of the more prominent cases…

A brief but global Machines API outage became a topic of significant comment in the middle of the month, June 15. The widest (graphically) incident of Season Two, thus far.


    

North American networking glitched on June 11 and on June 24, leading to tremulous service across a wide area (as well as a narrower after-shock in ORD on the earlier date). It’s not clear whether these were re-occurrences of the BGP faults from the opening March installment of Season Two, or some new way of networking not working.


         

Delays in processing metrics spanned most of June 28–30. Across those three days, there were clogs at certain points in the pipeline, resulting in localized stale statistics and hence bad auto-scaling behavior for some apps (among other things).


    

Sprites URLs and dashboard went awry on June 2 (actually starting a bit in June 1), possibly related to authentication. This was a rare case of a forum-only incident.