Tigris-backed volumes

Hi all! We have a few small updates to our work on Bottomless S3-backed volumes, which should make it a bit easier for folks to get started using this feature.

:warning: Note—this is still an experimental feature. It’s not officially supported or ready for production use, and backwards compatibility is not guaranteed. If you run into problems, please get in touch with us here on the community forum.

Volumes :heart_hands: Tigris

Since Tigris launched earlier this year, we’ve had quite a bit of interest in using Tigris as the provider for object-storage backed volumes. We’re really excited about this combination too! This setup was possible via fly storage create (or adding a Tigris bucket during fly launch) plus a bit of environment-variable tweaking, but it’s now a bit easier through the fly volumes lsvd setup command directly. When you run fly volumes lsvd setup and select Tigris as the service, it will automatically create a Tigris bucket with all necessary secrets on your app to get going.

We also made a small tweak to the lsvd binary to recognize the BUCKET_NAME environment variable that the Tigris extension sets by default (in addition to FLY_LSVD_S3_BUCKET), to make the integration smoother.

Multiple Machines Support

The initial release of object-storage backed volumes came with this warning:

For now, don’t run more than one LSVD-enabled machine per app! They’ll conflict with each other and corrupt your volume.

This is now supported- each instance can upload its logs to a different prefix in the same bucket (currently lsvd/$FLY_MACHINE_ID by default), so running multiple LSVD-enabled machine instances per app works out of the box.

Note that this prefix change is incompatible with existing deploys of LSVD-enabled machines. If you need backwards compatibility for an existing volume, set the FLY_LSVD_KEY_PREFIX=/ environment variable, or rename/move existing object files to the new prefix.

Fly App support

The initial release required you to add an --lsvd flag to the fly machines run command to create a machine with the required lsvd background daemon running to provide the volume. Now, the daemon automatically starts on any machine with FLY_LSVD_* variables set in its environment, no additional flags required.


Speedrun

These three updates combined make it easier to launch an app with object storage-backed volumes: After creating an app, run fly volumes lsvd setup, select Tigris, enter the volume size and mount path, then deploy like usual. That’s it!

Feedback and additional testing will be tremendous help as we continue iterating on this feature, so please feel free to try it out, kick the tires a bit and let us now how it’s working (or not) for you! We’re actively working on this and hope to have more updates ready to share soon.

9 Likes

Just FYI, the link to the example database using S3 currently throws an error if I try to visit https://ct-log-viewer.fly.dev/.

Does this mean that each instance sees its own separate filesystem? Or are they able to share files? E.g., if one instance writes a file “test.txt”, can the other instances read it?

2 Likes