Here’s the latest Appkata example, Minio. Specifically setting up a persistent S3 compatible storage node on Fly with it. See the example at appkata-minio.
Discuss it here…
Just found this while researching object storage options. This looks great! Do you have any production-level experience with Minio? I’m curious what kinds of gotchas might pop up with clustering.
We have been running Minio as a S3 “gateway” ever since Remote builds consistently failing with gpc error to our Docker registry.
We’ve had no troubles so far! Didn’t have to modify their base image or anything, just set a few environment variables and secrets.
I think this is pretty different from running in non-gateway mode though. I assume it should work about as well as anything. It should be able to replicate (if setup as such) through the private network?
Nice - so I looked into this. It looks like you can almost setup a very reliable cross-region cluster using a combination of in-region clustering (for HA and aggregated storage) and two-way bucket replication.
The only caveat is that Minio has an unfortunate requirement when specifying peer cluster nodes. The hostnames must have sequential integers in them, as this is the only supported discovery format: https://minio-host-{1...4}/disk.
They suggested adding entries to from service discovery to /etc/hosts at boot time. This is kind of gross, but I suppose it would work fine for testing.
It might also be useful to do something similar with DNS service discovery, and not just for this case. Adding DNS entries for node1.region.appname.internal, node2.region.appname.internal, etc, and passing the regional VM count as an env var to allow building the above.
1 Like
Wow that’s certainly unique. But you can make it work with no discovery changes!
When you mount persistent volumes, the VMs that come up get persistent 6pn addresses. So you could actually make real DNS entries for minio once you figure out what those addresses are. It should work really well, despite being a little manual to get going.
I see - how do you make DNS entries?
Oh, just normal DNS entries for any domain you happen to own. You can actually add domains and manage DNS with fly domains and fly dns-records respectively.
For something like the minio setup, you could just make a zonefile for each volume you create and then run fly dns-records import <domain> zonefile.txt to sync 'em up.
Ah, OK, I thought you meant we could control the internal DNS entries.
For now I think it makes more sense just to use /etc/hosts, as it’s easy to setup at boot time without involving DNS at all. Any reason not to try this?
1 Like
Should work fine!
Oh yeah I guess that’s basically the same thing just with less work. Do you have to restart minio processes when new members of the cluster join?
Yes, you have to restart them. I think it’s designed to be run on large instances with many disks.
I was looking into volumes this morning, realized it was not an s3-type solution, kept exploring and found this article. It’s very interesting, but I can’t help but feel like this is something that fly itself would excel at. I’m looking a gift horse in the mouth because I think giving fly to devs is like giving fire to humans (i.e., its great). But, are yall considering offering a service like s3+cloudfront or Digital Ocean Spaces? It seems like a lot of the pieces are already in place with volumes, regional redis clusters, the static assets + anycast stuff, etc.
1 Like
We’d like to! The problem is, object storage is a very hard service to run well. We’ve thought a lot about building some basic storage into apps for small scale usage, though. Something like “useful scratch space for about 1GB of data” seems nice and constrained.
3 Likes
I am thinking about giving this a go on fly.
the client gets posix file system. It can also use fuse.
the backend is redis and minio…
https://juicefs.com/docs/cloud/replication
The replication works exactly how postresql works on fly. 
golang with a redis cache and minio store.
I don’t know about the “best” way, but it’s pretty dang easy. I noticed the GitHub - fly-apps/appkata-minio: MinIO S3-compatible storage on Fly repo is archived. This only means we haven’t tested it in a while and can’t vouch for it, but after a quick glance I think it’ll work just fine.
Hmm, fly init no longer exists. Happy to try fly create instead but the semantics look slightly different.
SeaweedFS uses minio but is much faster . It’s another option
You can add volumes on the fly , which solves the discovery problem .
Others are doing it according to this issue : Address already in use error when binding ipv6 addresses in 2.95 · Issue #2983 · seaweedfs/seaweedfs · GitHub
1 Like