I understand some of the limitations on volumes, but I’m a bit confused about the region limitation. Certainly that would make an application more secure, avoids some of the complexity of distributed filesystems and cdns, but it seems to significantly reduce the utility value of a volume.
A common and simple use case like uploaded images doesn’t work with multiple regions unless I’m completely not understanding something. It seems like having multiple regions [mount] from fly.toml would require each region to have its own volume except they’d each potentially have different data.
Is the fly.io recommendation for a shared filesystem to spin up storage somewhere like AWS S3?
My understanding is that an app/vm has ephemeral storage, so if you just need a temporary store you can put vms all over the world and use that. They don’t need to maintain state so can be moved/scaled as needed. They are all equal. But volumes are like databases: somewhere needs to have a primary/master copy of the data as there is a state. And so a volume is in one region. So yes, using volumes does trade-off the multi-region flexibility by its nature.
Even with S3, multi-region storage is complicated. At least when I last looked they did have auto-replication (so that would like if Fly offered multi-region volumes) however it’s not instant and so for a period of time a file is in not in both buckets simultaneously. You come back to needing one primary region.
You could run your own S3 on Fly to reduce cost. As @thattommyhall says. With Minio. But again, it’s still one region.
If you need Filesystem semantics, https://www.gluster.org/ is my favourite (it can be simple to recover if something goes wrong, you can see the files on a disk somewhere), Ceph.io — Home is very popular too.
“volume” connotes a block device (even though they mount it for you in the VM), usually only one machine at a time can read these so pinning in one region is reasonable in my opinion