We have a total of 5 prod apps deployed on Fly, and 2 of them that both have attached volumes went down 11pm-2:30am EDT last night.
I noticed the incident around broken deployments but wanted to share that we did not make any deployments across any of our apps at the time, they simply went down on their own.
There was a single server in
iad that crashed and required a full reboot last night, unrelated to the incident around broken deployments. (We’re still investigating the root cause, but signs point to some kind of host-kernel bug.) The concidental timing made our evening extra fun, and also meant we didn’t communicate this separate incident clearly- sorry about that!