Hi Konstantin, I’m glad we got everything restored, and I just wanted to follow up with a few clarifying points for you and anyone else who reads this about why this happened and how to prevent this from happening again.
This started with a single host server outage. We don’t regard single host failures as a failure of the Fly Platform, which is why we say often in the docs and CLI (and are looking for more places to repeat) that the intended way to use the Fly Platform is to run Apps with two or more Machines. I will quote this again from the docs:
Create and store backups: If you only have a single copy of your data on a single volume, and that drive fails, then the data is lost. Fly.io takes daily snapshots and retains them for 5 days, but the snapshots shouldn’t be your primary backup method.
It looks like the software you’re running is Vaultwarden, and after a quick look I couldn’t tell if Vaultwarden offers a “High Availiblity” configuration or not. If not, it will be difficult to run it as a unified cluster. But even if you have to run this as a single Machine, you should have an automatic procedure to make your own backups.
But the second step towards this problem was our mistake. You didn’t receive any notification that the host server was offline. We should have sent one. The reason for this mistake is that we were in the middle of a release of a new, more automatic method of sending notification emails. Due to this new change in procedure, was a mixup among our staff about what needed to be done to send notification emails, and as a result no emails were sent. If we had sent emails, you could have taken action in time to restore from a Fly.io Volume snapshot, instead of waiting for the server to come back online.
I apologize for this mistake. There should have been notification emails. But I hope you can see there was a specific cause to the failure to send emails, and you can trust that you will receive host failure notification emails in the future. With notification emails and multi-machine deployment or custom backups, you data will be secure.