persistent but not resilient?

1

apologies if this has been covered already, but am a relative newcomer to fly.io and search efforts did not bear fruit

I’m looking into fly.io to use for some hobby projects (to run some postgres db clusters) and am curious about how its storage volumes actually work

the docs and marketing materials use the term ‘persistent storage’ for volumes, but I also found this blog post: Persistent Storage and Fast Remote Builds · Fly

which mentions: “for raw Fly volumes, resilience is your problem”

which has me a bit perplexed

I’ll admit I’m more of an app developer than devops/infra person at this stage, but is there any magic to the word “raw” in that quote, or can we basically not rely on any fly persistent volume to be there without providing our own backup infrastructure?

followup note, i see from the suggested topics window to the right of this editor this link: Is there backups for persistent volumes

so I guess this question is then more: is this still the case?

and following on from that, why is this only mentioned in one blog post and a support ticket rather than the actual docs?

2 Likes
2

This is a good question!

It would be useful for us to define what we mean by resilient. We’re talking about data durability and availability. That is, the chance of either losing data or having the disk go offline when you need it.

When you allocate a volume on Fly, you get part of one physical NVMe drive (this is what we mean by raw). If you’re used to services like EBS, our single physical NVMe drive is less resilient than what EBS offers. But probably equivalent to AWS ephemeral storage.

Resiliency is a sliding scale, to make our volumes more resilient, we’d need to put them on arrays of NVMe drives and ensure that you wouldn’t lose data if some number of those drives failed. This still isn’t 100% resilient, though, because it’s not possible to reach 100% resilience. Instead, we’d be targeting numbers like 99.999% durability and an annual failure rate of 0.001%. We aren’t there yet, though. We’re not even to the point where we have these kinds of metrics.

We run 2 node Postgres clusters intentionally. They should have better availability than a single node, and they have data redundancy. When you run HA postgres, you’re getting data storage on two isolated NVMe drives. It’s simpler and cheaper to run high availability databases than it is to run durable disk arrays.

Backups are there now! If you run fly volumes list and then fly volumes snapshots list vol_<id> you’ll see the snapshots we’ve taken. You can use one of those snapshots to restore into a new Postgres cluster with fly pg create --snapshot-id <id>.

We’ve gone back and forth about how to explain all this. Partially because people come with all kinds of different disk expectations. If you use DigitalOcean droplets, you probably don’t care that they’re putting your data on a single disk. If you’re using EBS, you might care about things like data durability.

So the tldr might be: it’s not as scary as we made it sound, it’s also not as good as EBS. You can trust Fly persistent volumes about as much as you trust your local hard drive. And you can trust but verify that we’re taking snapshots for you. :slight_smile:

2 Likes
3

Hi Kurt,

Thanks for the quick and also quite in depth reply, much appreciated!

very cool to hear that the snapshots are there, that does alleviate quite a bit of my worries

for context, my use case is for hobby projects, things where, i want them to ‘just work’ and it may not be the end of the world if they don’t but also ‘complete data loss’ if i don’t check on them for a bit is also unacceptable (or at least really annoying!)

I agree that the different disk expectations are weird, I certainly came with my own that were a bit mismatched.

While I’d love to see these things more fleshed out in the docs in the future, I can at least say I’m impressed at the speed of a quality reply in the community queue here, which is also a good sign.

All the best and happy holidays!

1 Like
4

quick followup if you don’t mind:

I see 2 docs re: postgres:

and

is one more or less recent than the other?

what’s the most current recommended way to setup a simple fly postgres server (which will be connected to from external services)

5

The first doc is what you want, the second is how you’d expand that to multiple regions.

Just run fly pg create and you should have everything you need!

6

Before they built fancy ASICs, it wasn’t uncommon for a hyperscaler cloud I worked at, to run distributed DBs with software RAID0 on JBODs, because resiliency (like Kurt defines it) was handled by replication (usually over Multi-Paxos) while RAID0 provided speed (a job now within the purview of aforementioned ASICs).

Speaking of EBS-esque resiliency (like Kurt defines it)-- something equivalent to RAID1 (software or otherwise) is what EBS replicated storage is. So, if one can set that up on Fly, I don’t see why one can’t get “close enough” to what EBS guarantees (AFRs aside).

1 Like