The server running my postgres db suffered a hardware failure on 1st April. The database is part of an infrequently-used app and I only found out about this yesterday (12th April).
When running the command fly volumes snapshots list vol_6d7xkrkndjyrw2q9 I get the response No snapshots available.... Is this because snapshots are only retained for 5 days? If that’s the case, I guess there’s no way to retrieve a snapshot and my data is lost?
Hi Matt, I hate to be the bearer of bad news, but yes, your data is gone. I just said this in another thread and I’ll say it here, this is actually the first time we’ve had hardware failures like this, and we’re still working out the kinks in how to respond. Yes, we need to send out emails sooner. We also need to emphasize still more that we view Fly Volumes as persistent storage, but not durable long-term storage; as we say in the docs:
Create and store backups: If you only have a single copy of your data on a single volume, and that drive fails, then the data is lost. Fly.io takes daily snapshots and retains them for 5 days, but the snapshots shouldn’t be your primary backup method.
But if that comes as a surprise to you, then we didn’t succeed in setting your expectations about the platform correctly, and that’s something we should be better about.
None of this is helpful for getting your data back though. If that’s possible, it will require thinking outside the box. The database is gone on the Fly.io side, but maybe you had it dumped locally at some point? Or, if you echoed query results to your logs, maybe you could recreate your data from the logs? Try to think of any place that might have seen data cached or echoed, and try to recover it form there. I wish you the best of luck.
Thanks for confirming John. I do have a local dump but it’s quite old and a lot was added since.
I am disappointed to have lost the data because a lot of work went into it. But ultimately it’s my own fault for not keeping on top of backups and relying on Fly.io: it was my data after all.
To echo what was said in the other thread that you referred to above, perhaps fly.io could do more to actively notify users of hardware issues. Because I use the app infrequently, the first I knew of this was when I tried to access the app a couple of days ago and it was unavailable. Had I been notified of the failure within the 5 day snapshot retention period I might have been able to save the data.
That said, it’s ultimately my responsibility to monitor the health of my app in the same way that it’s my responsibility to take care of backups. And that’s especially the case on a free service.
I will just have to chalk this one up to experience. Lesson learned!
This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.