PSA: Postmortem(s) rollup

A small graphical overview of the previous quarter, now that May through July 2026 are complete in the Log…

unofficial fly.io incidents grid for may 2026 unofficial fly.io incidents grid for june 2026

See the earlier March grid for a description of the annotations.

The <details> elements below can be expanded, to get the July image’s links.

  • Week of July 01: SYD, GraphQL, dashboard, NRT, certs, ORD×2

    incid.
    date
    sourcessymbolbox description
    07/01forum½│SYD capacity
    07/01infra,
    s.f.n
    ½▪GraphQL
    07/01infra,
    s.f.n
    ½▪dashboard
    07/01s.f.nNRT static egress IPs
    07/02infra,
    s.f.n
    certificates
    07/03infra,
    s.f.n
    ORD power outage
    07/03s.f.nORD network
  • Week of July 05: Depot, registry

    incid.
    date
    sourcessymbolbox description
    07/09infra,
    s.f.n,
    forum
    IAD Depot builders
    07/09infra,
    s.f.n
    ½─registry
  • Week of July 12: LHR, DFW, GRU, SJC, Fly Proxy, Machines API, YYZ, Grafana

    incid.
    date
    sourcessymbolbox description
    07/13forum½▪LHR capacity
    07/13s.f.nDFW
    07/14forum½▪GRU capacity
    07/14infra,
    s.f.n
    SJC
    07/16infra,
    s.f.n
    ½─Fly Proxy due to Vault
    07/16infra,
    s.f.n
    Machines API due to Vault
    07/17s.f.nYYZ Flycast
    07/18infra,
    forum
    Grafana logs
  • Week of July 19: Grafana, BOM, Sprites, Machines API, MPGv1, DFW, Redis, JNB, NRT, OAuth

    incid.
    date
    sourcessymbolbox description
    07/19infra,
    forum
    Grafana logs (continued)
    07/19s.f.nBOM egress v6 IPs
    07/20infra,
    forum
    Grafana logs (continued)
    07/20forumSprite OAuth-based connectors
    07/20infra,
    s.f.n
    Machines API
    07/20s.f.n½│MPGv1 (segue)
    07/21infra,
    forum
    Grafana logs (continued)
    07/21forum,
    forum′
    Sprite OAuth-based connectors (continued)
    07/22infra,
    forum
    Grafana logs (continued)
    07/22forum,
    forum′
    Sprite OAuth-based connectors (continued)
    07/22s.f.nDFW network
    07/22forumRedis in DFW
    07/22forum½│JNB capacity
    07/23forum,
    forum′
    Sprite OAuth-based connectors (continued)
    07/23forum½│JNB capacity (continued)
    07/23forumNRT egress IPs
    07/24forum,
    forum′
    Sprite OAuth-based connectors (continued)
    07/24forum,
    forum′
    dashboard(?) OAuth connections
    07/25forum,
    forum′
    Sprite OAuth-based connectors (continued)
  • Week of July 26: Sprites, Machines API, dashboard×2, CDG, FRA

    incid.
    date
    sourcessymbolbox description
    07/26forum,
    forum′
    Sprite OAuth-based connectors (continued)
    07/27forum,
    forum′
    Sprite OAuth-based connectors (continued)
    07/28forum,
    forum′
    Sprite OAuth-based connectors (continued)
    07/29forum,
    forum′
    Sprite OAuth-based connectors (continued)
    07/29s.f.n½▪Machines API
    07/29s.f.n½▪dashboard
    07/31s.f.nOutbound email via dashboard
    07/31s.f.n½▪CDG capacity
    07/31forumFRA freebie Consul cluster

(In the expanded tables’ second columns, “s.f.n” is status.flyio.net, the real-time status page, which serves as a second-tier source in this context.)

The second third of summer returned to a calmer atmosphere early on but then the winds picked back up in its final weeks.

Of the more prominent things blown over…

The Machines API went down for an hour worldwide on July 20, due to a failure of the tokens service’s primary node.


A rare global Fly Proxy anomaly July 16 was the tune of Vault’s swan song. This only affected uncached SSL certificates—as well as consonant operations of GraphQL and friends.


A brief registry overload coincided with an IAD-specific builder capacity crunch on July 9. The Depot builders continued to be rather sensitive to regional congestion.


                   

A 10-day Sprites connectors glitch was etched into the month’s final weeks, an additional forum-only incident in this corner. (As others have mentioned, it would be nice to know more about what goes awry in Sprites-land.)

[The parallel (for a while) 4-day logs problem did have an Infra Log entry, however.]