# PSA: Postmortem for the Nov 25 Outage

**URL:** <https://community.fly.io/t/psa-postmortem-for-the-nov-25-outage/22933>\
**Category:** General\
**Tags:** distributed, blog, sqlite, postmortem, login-only\
**Created:** [December 5, 2024, 9:00pm UTC](https://community.fly.io/t/psa-postmortem-for-the-nov-25-outage/22933 "2024-12-05T21:00:37Z")\
**Posts on this page:** 1\
**Showing post:** 7

<div class="post-metadata">

**Author:** ![kyleatcausadix](https://sea1.discourse-cdn.com/fly1/user_avatar/community.fly.io/kyleatcausadix/32/18530_2.png) [@kyleatcausadix](https://community.fly.io/u/kyleatcausadix)\
**Post date:** [December 6, 2024, 5:06pm UTC](https://community.fly.io/t/psa-postmortem-for-the-nov-25-outage/22933/7 "2024-12-06T17:06:18Z")

</div>

> If you’re concerned about the ‘uptime’ graphs in the latter two screenshots, I can give some context

Thanks, ~~Jordan~~ Will (sorry!)! That bit of context is helpful and appreciated.

> We want to publish _actual_ detailed availability metrics straight from our platform and hope to eventually! We’re still working on it.

Very much looking forward to that 🤞

> Hope this clarifies what you saw on the page, sorry this caused extra confusion for you during the incident.

Let’s talk about this one some more?

Briefly, let’s establish shared context by recapping some truths:

1. The status page confused many new customers and [several](https://community.fly.io/t/fly-io-site-is-currently-inaccessible/22791/31) [tenured](https://community.fly.io/t/fly-io-site-is-currently-inaccessible/22791/50) ones (including [myself](https://community.fly.io/t/fly-io-site-is-currently-inaccessible/22791/30), if \>= 10 months on the platform is the bar 😉). By my count, at least 7 new accounts with a first post in that topic specifically mention confusion from unclear communication.
2. Regional availability (forget about the uptime showcase) was marked _Operational_ and not _Degraded_ while multiple (many?) regions were failing to service customer traffic. Indeed, `https://fly.io` itself serviced `504`s during the incident. I might misunderstand what that status is supposed to communicate.
3. Deployments was marked _Operational_ and not _Degraded_ while many customers (including myself) were unable to deploy. Given Machines API was _Degraded_, weren’t `bluegreen` deployments, by definition, also at least _Degraded_?
4. Remote Builds was marked _Operational_ and not _Degraded_ while many customers (including myself) were unable to ~~deploy~~ build. Given Machines API was _Degraded_, weren’t apps that needed to `create` new builder machines, by definition, also at least _Degraded_?
5. Fly Machine Image Registry (1 and 2) were marked _Operational_ and not _Degraded_ while customers (including myself) were unable to push new images. I observed 504 timeouts both authenticating and while pushing.
6. While not listed under the Extensions component, [Sentry account](https://fly.io/docs/monitoring/sentry/) creation is initiated via `flyctl ext sentry *`. I was unable to access Sentry during the incident because the authentication flow consistently failed due to timeouts. Extensions was marked _Operational_ and not _Degraded_.

My question is this, @wjordan: given _so much_ of Fly’s platform has a SPOF on both Corrosion and Machines API (and _the_ GraphQL API), is it fair to say that many other components, including those listed above, were misrepresented as _Operational_ during the incident when they were also exhibiting degraded performance and instability?

Thank you for taking the time to hop into this thread. I know many customers, myself included, appreciate your time and consideration while helping us understand how Fly’s incidents have impacted our businesses and how we can better adapt and build resilience to these challenges. After all, every one of Fly’s customers (including myself) are rooting for the platform to succeed.

---

_[View the full topic](https://community.fly.io/t/psa-postmortem-for-the-nov-25-outage/22933)._