All our volume-attached apps went down for 2+ hours last night

We have a total of 5 prod apps deployed on Fly, and 2 of them that both have attached volumes went down 11pm-2:30am EDT last night.

I noticed the incident around broken deployments but wanted to share that we did not make any deployments across any of our apps at the time, they simply went down on their own.

1 Like

There was a single server in iad that crashed and required a full reboot last night, unrelated to the incident around broken deployments. (We’re still investigating the root cause, but signs point to some kind of host-kernel bug.) The concidental timing made our evening extra fun, and also meant we didn’t communicate this separate incident clearly- sorry about that!

1 Like