One of my apps had an outage a couple of weeks ago. i followed hte instructions here: Troubleshoot apps when a host is unavailable · Fly Docs but there are no snapshots available for my volume. I thought fly.io saves snapshots every 24hrs but its not giving me access to any. Also, when I do fly volumes list it lists one volume but after it says * These volumes' hosts could not be reached.
I’m very upset with this service, as I’ve had no way of contacting fly to find out where all the data in my app has gone.
Hi gjfkgkjf,
Thank you truly for taking the time to report what you’re experiencing here, and for your honest feedback about the service.
I hate to be the bearer of bad news, but the host that volume is on is indeed down, and the snapshots for your volume on this host are out of the default 5-day retention window and are therefore probably lost. We’re still working on this host to see what might be possible, but the most likely case is that we won’t be able to bring it back, and that the data is gone for good. Providing more/better platform options and guardrails to alleviate or prevent this kind of situation is something we’re really focused on recently.
There’s a point about volumes that we try to emphasize often (through the CLI, in the docs), but probably not often or well enough- we view Fly Volumes as persistent storage, but not durable long-term storage. From our docs on volumes:
Create and store backups: If you only have a single copy of your data on a single volume, and that drive fails, then the data is lost. Fly.io takes daily snapshots and retains them for 5 days, but the snapshots shouldn’t be your primary backup method.
If this is coming as a surprise to you, we haven’t done our job to set expectations about the platform correctly. There’s a couple things we’ve started doing recently to try to make this aspect of the platform better, to recently to be of use in this case, but:
We’ve recently updated flyctl to make the retention period for volume snapshots configurable per volume: Retain your snapshots from 1 to 60 days, your choice
We’re also rolling out a new system for sending out email notifications in the event of host failures like this one to try to give anyone affected enough time to recover from snapshots or take any other necessary actions: Host Issue notifications
I wish I could have brought you a better outcome here, but I hope this at least clarifies what happened to your snapshots here and how this feature of the platform currently functions.
- Ben
This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.