Onward to the first incidents of September…
| incid. date |
pub. date |
description |
|---|---|---|
| 09/02 | 09/11 | Sidekiq periodic scheduler not running after Redis timeout |
| 09/02 | 09/16 | Sprites API partial outage from SQLite database evictions |
Builder outages have gotten rarer lately, but another set appeared. This time, it was due to a DNS glitch instead of the previous pattern of capacity shortfalls…
| incid. date |
pub. date |
description |
|---|---|---|
| 09/15 | 09/28 | Stuck DNS broke some Depot builds (SYD, JNB) |
| 09/19 | 10/05 | Sprite create failures due to network instability |
| 09/19 | 10/05 | WireGuard gateways couldn’t reach recently created or moved Machines via 6PN |
| 09/23 | 09/24* | Upstash unavailability in FRA region |
*Postmortem published inline at status.flyio.net, rather than as a separate Infra Log entry.
The third row may have been the underlying cause of the series of isolated SSH failures that started on the 17th.
Aside: The first Log entry mentioned in its final sentence that major changes to the DNS infrastructure are in the works, which is intriguing…
Continuing with a couple more on September 23 (and then into September 24)…
| incid. date |
pub. date |
description |
|---|---|---|
| 09/23 | 10/08 | Sprites API outage during database migration |
| 09/23 | 10/08 | Our gateways took out 6PN for a bit |
| 09/24 | 10/09 | Network readiness checks stalled worker hosts during reroute |
| 09/24 | 10/14 | Broken routing in GRU |
| 10/16 | 10/16* | Fleet-wide emergency patches (PwnEscape) |
*Preview postmortem in the community forum.
The single-day rolling reboot of the entire set of 1000+ physical host machines was the biggest event in quite some time, of course. The link above has an initial explanation of the context and trade-offs, with a promise of further details later.