# PSA: Postmortem(s) rollup

**URL:** <https://community.fly.io/t/psa-postmortem-s-rollup/27367>\
**Category:** Fresh Produce\
**Tags:** blog, postmortem\
**Created:** [March 15, 2026, 7:26am UTC](https://community.fly.io/t/psa-postmortem-s-rollup/27367 "2026-03-15T07:26:33Z")\
**Posts on this page:** 3\
**Page:** 4

<div class="post-metadata">

**Author:** ![mayailurus](https://sea1.discourse-cdn.com/fly1/user_avatar/community.fly.io/mayailurus/32/17593_2.png) [@mayailurus](https://community.fly.io/u/mayailurus)\
**Post date:** [September 24, 2026, 7:20am UTC](https://community.fly.io/t/psa-postmortem-s-rollup/27367/63 "2026-09-24T07:20:20Z")

</div>

Onward to the first incidents of September…

| incid.  
date | pub.  
date | description |
| --- | --- | --- |
| 09/02 | 09/11 | [Sidekiq periodic scheduler not running after Redis timeout](https://fly.io/infra-log/sidekiq-scheduler-redis/) |
| 09/02 | 09/16 | [Sprites API partial outage from SQLite database evictions](https://fly.io/infra-log/sprites-api-sqlite-evictions/) |

---

<div class="post-metadata">

**Author:** ![mayailurus](https://sea1.discourse-cdn.com/fly1/user_avatar/community.fly.io/mayailurus/32/17593_2.png) [@mayailurus](https://community.fly.io/u/mayailurus)\
**Post date:** [October 6, 2026, 1:12am UTC](https://community.fly.io/t/psa-postmortem-s-rollup/27367/64 "2026-10-06T01:12:55Z")

</div>

Builder outages have gotten rarer lately, but [another set](https://community.fly.io/t/depot-builds-fail-every-time-failed-to-list-workers-authentication-handshake-failed-eof/28636) appeared. This time, it was due to a DNS glitch instead of the previous pattern of capacity shortfalls…

| incid.  
date | pub.  
date | description |
| --- | --- | --- |
| 09/15 | 09/28 | [Stuck DNS broke some Depot builds](https://fly.io/infra-log/stuck-dns-depot-builds/) (SYD, JNB) |
| 09/19 | 10/05 | [Sprite create failures due to network instability](https://fly.io/infra-log/sprite-create-failures/) |
| 09/19 | 10/05 | [WireGuard gateways couldn’t reach recently created or moved Machines via 6PN](https://fly.io/infra-log/wireguard-gateways-6pn/) |
| 09/23 | 09/24\* | [Upstash unavailability in FRA region](https://status.flyio.net/incidents/9855z4sdljbr) |

\*Postmortem published inline at `status.flyio.net`, rather than as a separate Infra Log entry.

The third row may have been the underlying cause of the [series of isolated SSH failures](https://community.fly.io/t/machine-unreachable-from-wireguard-peer-but-healthy-and-reachable-from-other-apps-in-the-same-org/28666) that started on [the 17th.](https://community.fly.io/t/28660)

* * *

Aside: The first Log entry mentioned in its final sentence that major changes to the DNS infrastructure are in the works, which is intriguing…

---

<div class="post-metadata">

**Author:** ![mayailurus](https://sea1.discourse-cdn.com/fly1/user_avatar/community.fly.io/mayailurus/32/17593_2.png) [@mayailurus](https://community.fly.io/u/mayailurus)\
**Post date:** [October 19, 2026, 12:01am UTC](https://community.fly.io/t/psa-postmortem-s-rollup/27367/65 "2026-10-19T00:01:05Z")

</div>

Continuing with a couple more on September 23 (and then into September 24)…

| incid.  
date | pub.  
date | description |
| --- | --- | --- |
| 09/23 | 10/08 | [Sprites API outage during database migration](https://fly.io/infra-log/sprites-api-outage/) |
| 09/23 | 10/08 | [Our gateways took out 6PN for a bit](https://fly.io/infra-log/gateways-broke-6pn/) |
| 09/24 | 10/09 | [Network readiness checks stalled worker hosts during reroute](https://fly.io/infra-log/network-readiness-checks-stalled/) |
| 09/24 | 10/14 | [Broken routing in GRU](https://fly.io/infra-log/broken-routing-gru/) |
| 10/16 | 10/16\* | [Fleet-wide emergency patches](https://community.fly.io/t/gru-machines-and-attached-volumes-remain-unreachable-during-rolling-host-reboots/28843/26) (PwnEscape) |

\*Preview postmortem in the community forum.

The single-day rolling reboot of the entire set of 1000+ physical host machines was the biggest event in quite some time, of course. The link above has an initial explanation of the context and trade-offs, with a promise of further details later.

[Previous page](https://community.fly.io/t/psa-postmortem-s-rollup/27367.md?page=3)
