Builder outages have gotten rarer lately, but another set appeared. This time, it was due to a DNS glitch instead of the previous pattern of capacity shortfalls…
| incid. date |
pub. date |
description |
|---|---|---|
| 09/15 | 09/28 | Stuck DNS broke some Depot builds (SYD, JNB) |
| 09/19 | 10/05 | Sprite create failures due to network instability |
| 09/19 | 10/05 | WireGuard gateways couldn’t reach recently created or moved Machines via 6PN |
| 09/23 | 09/24* | Upstash unavailability in FRA region |
*Postmortem published inline at status.flyio.net, rather than as a separate Infra Log entry.
The third row may have been the underlying cause of the series of isolated SSH failures that started on the 17th.
Aside: The first Log entry mentioned in its final sentence that major changes to the DNS infrastructure are in the works, which is intriguing…