Global caching object storage on Fly.io in private beta

Fresh Produce
1

At Fly.io, our pet-project-we-won’t-build has always been: S3-compatible, pay-as-you-go, globally caching object storage.

I’m happy to announce that our friends and partners at Tigris made it happen. Their service is running in production on Fly.io hardware today, fully integrated with flyctl. Learn more in the Tigris overview, S3 API Compatibility docs, and pricing.

Sign up for the private beta here. We’ll start rolling out invites within a week! :metal:

Here are a few highlights:

Tigris buckets are truly global - they cache objects around the world, depending on where the objects are needed most, intelligently handling object updates and removals across regions with zero configuration. By default, objects are first stored close to where they’re pushed from. So, even single-region apps can benefit without explicit bucket placement.

Smaller assets are optimised for fast delivery directly from the FoundationDB clusters that run the show.

Replication protects objects against unlikely region-wide failures.

Tigris already covers basics like public buckets, http2 support, and signed object URLs. So, frameworks like Rails ActiveStorage work out of the box.

Note that get half of this setup on big providers, you need to setup storage buckets and a caching CDN separately, each with their own configuration and costs. Here it’s as simple as flyctl storage create. Here’s what provisioning looks like:

f[~]$ fly storage create
? Select Organization: fly-ephemeral (fly-ephemeral)
? Choose a name, use the default, or leave blank to generate one:
Your  project (crimson-violet-6526) is ready.

Set one or more of the following secrets on your target app.
BUCKET_NAME: crimson-violet-6526
AWS_ENDPOINT_URL_S3: https://fly.storage.tigris.dev
AWS_ACCESS_KEY_ID: tid_xxxxxxxxxxxxxxxxxxxxxx
AWS_SECRET_ACCESS_KEY: tsec_xxxxxxxxxxxxxxxxxxxxxxx

The Tigris and Fly.io teams are here to answer your questions. Shoot!

29 Likes
2

This is awesome!

To confirm, is the data stored in fly data centers as well?

Are all fly regions supported for data storage? (not just access, but actual at rest storage) if not, which regions are supported?

Is there a private endpoint available so that data doesn’t travel via the internet? (essentially stays within the organization’s wireguard network)

Also, since the service runs on fly, will there be reduced bandwidth cost? (especially for same-region transfers)

2 Likes
3

Hey @charsleysa, I am Ovais, Co-Founder and CEO of Tigris. These are great questions. Let me answer them for you.

So Tigris is deployed and running inside Fly. When your service communicates with Fly, all the network communication between your app and Tigris stays within Fly’s network and doesn’t travel over the Internet. This, of course, also means the bandwidth costs will be reduced. We haven’t yet put a number on the bandwidth cost, but be assured that they will be much lower.

We are currently deployed in the following regions, and that is where we are storing the data as well:
fra, gru, hkg, iad, jnb, lhr, mad, ord, sjc, syd. The plan is to be deployed and running in all the Fly regions. However, even if we are not deployed in all the regions right now, we have good geographic coverage, so if your app is running in a region not in the above list, we will still be able to serve it from a nearby region.

7 Likes
4

Hi @ovaistariq

Thanks for answering the questions!

I’m not 100% sure what your partnership with fly is like so I just want to clarify on cost.

Fly has egress costs for all transfers out of any VMs. Will those costs continue to apply when transferring to the bucket? I’m just trying to clarify who is charging for what.

Also, will billing be done through fly or will it be separate?

5

We haven’t decided what bandwidth pricing will look like yet, but we’ll likely have a tiered model. This will be worked out as we approach a public launch. That said, we don’t want this service to be more expensive than alternatives. We’ll post more as we get there!

Billing is handled through Fly.io. Like other costs, costs will show up in your Fly.io dashboard and be updated regularly throughout the month.

2 Likes
6

Thanks @joshua-fly!

Just something to keep in mind when looking at the bandwidth pricing, service providers like AWS provide zero cost bandwidth when for transfers between their services and between buckets, including their CloudFront CDN.

Would be awesome if we could spin up a CDN on fly using these buckets and only have egress costs for transfers out of the CDN and not between the buckets and CDN.

2 Likes
7

We would definitely have this in mind as we want to be fair about pricing.

8

Yeah, it’s most likely we’ll follow this pattern. Just need to discuss internally first :slight_smile:

We’re talking about CDNs a lot lately. I’m curious: can you explain what your CDN requirements are?

9

Currently our frontend servers run caddy which proxy to an S3 bucket.

It’s done mainly because there’s no easy secure way of exposing files in an S3 bucket unless you use CloudFront and we want redirect to index when a path is not found to support SPAs, but also we do a few smarts like adding CSP headers and have special routes under the cdn. subdomain that proxy to our API for fetching dynamic content or user uploaded content.

We proxy to our API for user uploaded content mainly for handling permissions so we can keep the frontend servers with as little business logic as possible.

Most of content we serve up from the non-cdn. subdomain is static content from our frontend bucket (so all our SPAs are served from their own subdomain but using the same servers).

We deploy our frontend services to about 8 or so regions around the world to improve load times while our API services are currently in 1 region (but thanks to fly edges our user’s API requests route to the nearest edge and then use fly backhaul to get to our API region).

10

Are you using any caching on the Caddy VMs?

1 Like
11

Yes, though I don’t think it’s optimized yet as we only recently switched from nginx to caddy and we prioritized getting it working vs having perfect caching configuration.

12

So in your case, you wouldn’t be able to make the files public, or use S3 signed URLs? If you can use either of those, you could send users directly to a Tigris bucket.

13

No, probably not. Much of the files are private and can content PII so making them public is out of the question, and using signed S3 urls only works if the file exists ahead of time in the exact size and format needed. We do a lot of resizing and reformatting on the fly as needed and cache the results.

14

@joshua-fly
Had an odd issue:
Got accepted but I think it used the wrong org? I gave the org “Atridad Lahiji” which is what flyctl gave me as the org name. The slug is “personal”. Should I have given “Atridad Lahiji” or “personal”? I ask because I seem to not have access.

Edit:

Realized it should be “atridad-lahiji-96”… also if credits were give to “Atridad-Lahiji” that is not me :slight_smile:

15

Can this be used with the bottomless s3 backed volume? Bottomless S3-backed volumes

Use case: have multiple instances of my app accessing a shared volume, eg an embedded sqlite replica

16

Not sure where to post this, but I’m getting a FLY API 500 error when trying to delete a bucket.

17

Indeed this is something we’re looking into!

18

Thanks - a fix is going out now.

1 Like
19

Just to confirm that we ended up fixing the access to the right org.

20

Oh perfect, I just encountered this one!

I’ve also noticed we can’t have more than one bucket for a project even if I set the name to something different. Basically I wanted to delete my existing bucket and re-make it public.

Screenshot 2024-01-19 at 4.12.24 PM
Screenshot 2024-01-19 at 4.12.24 PM970×58 10.3 KB