Scaling file system/volumes on Fly io (GlusterFS?)

Questions / Help
, ,
1

I have an app called Rainfall (https://rainfall.dev) that works as a GUI frontend for a static site generator (Faircamp). The basic architecture is that users upload song files to the app, and they are stored on the filesystem of a Fly volume. When it’s time to create a preview of the site the app will generate, the faircamp binary is run on the uploaded files, which generates a self contained directory structure that contains the user’s site. These files are then served from the app server (not nginx or a static file server) as a preview, because the previews require authorization and are not publicly available.

My basic question is: how do I scale this setup? Currently I have one machine, with one volume. By my calculations, this will satisify about 5000 users, if I scale the volume to the maximum of 500 GB. But that’s a hard limit for this architecture and I would prefer to have options to scale beyond that.

I’m considering trying to set up a GlusterFS cluster inside of a Fly io app, and using Flycast to communicate between the Gluster app and my main app. The main advantage of this is that I would be able to add as many additional volumes as I need to additional cluster members. Additionally, I wouldn’t have to change the architecture of my app because the Gluster storage would be mounted on the app server as a single volume.

I assume I would have to mount a small volume on each server in the cluster to hold the Gluster configuration, as well as additional, larger volumes to hold the user data. One question is, how would I make a Fly volume available to a machine without mounting it, since I will have to create my own filesystem for the Gluster “bricks”? Also does it make sense to use the fly console command to set up Gluster inside of each of the servers, assuming I can save the configuration to the aforementioned configuration volume?

Please let me know if this makes any kind of sense, or if I’m going in the complete wrong direction, thanks!

3

Easiest way would be to not store or serve from the local volumes and instead use object storage (e.g. Tigris).

After the preview is generated, upload all the files to object storage and have your app server fetch from the object store instead of reading from the volume.

This way you don’t have to worry about storage since it’s handled by the object storage provider, and you can even setup things like auto deleting old preview files after 30 days.

4

Thanks for the reply.

I guess I didn’t consider serving the files from object storage, since I assumed that latency will be a problem, since I would be downloading from object storage, then sending the file to the client. Maybe not?

Also I still need to have a filesystem structure for Faircamp to operate on as it constructs the static site. I guess I could upload to object storage and have a worker task that pulls from there and recreates the directory structure on disk before generating the preview and re-uploading that?

5

Latency is in the milliseconds. For a static website that’s perfectly acceptable. We serve all of our static sites directly from Tigris buckets via a Caddy server.

All machines come with temporary storage so unless you need gigabytes of temporary storage while generating the site, you don’t need a volume.

What you can do is create a folder in the temporary directory, put all the necessary files there, generate the static site, upload the resulting files to object storage, then delete the folder.

1 Like
6

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.