Blob storage without blowing transit budget

Questions / Help
#1

I’m working on a project that requires that I download many fairly large file (400mb - 5GB), process them in place on a volume, and then upload them to a blob storage provider like S3, B2, or R2. Downloading files is no problem, but re-uploading the results in outbound traffic.

I would preferably like to do this work on Fly, since the main app lives here, but I’m nervous about incurring the cost of that outbound traffic.

Are there any above board “tricks” that can be used to get data into blob storage from Fly without that outbound cost? Perhaps something with wireguard? Or if it becomes a problem, does this compute need to live in-cloud near the blob storage that is used to avoid this internet traffic?

#2

If you’re going for R2, Cloudflare’s bandwidth alliance partners are a better fit for your compute (like Vultr, Alibaba Cloud, Oracle Cloud, Scaleway). Fly I don’t think is part of that alliance. Also, I believe, you may need to subscribe to Cloudflare’s Business ($200/mo) / Enterprise ($5000/mo?) plan for that.

Fly is a non-starter, unless your usage warrants a custom agreement with Fly (Fly already handles Tailscale VPN’s data-plane traffic (ref) and it is highly unlikely they’re paying a premium on bandwidth).

You may know already that there are providers like Hetzner / OVH / Dreamhost / Scaleway that do sell uncapped bandwidth (or for as low as $1/TB); so not all hope is lost.

1 Like
#3

This is one of those things where it depends entirely on volume.

Our outbound data rates are relatively cheap, so it might be fine to just start doing this naively if you’re starting at relatively low volumes.

At high volume, though, the only way to make this work is to run your own blob storage on Fly. Which will be more expensive.

I think this mostly comes down to how much time you have and whether running things on Fly saves you enough time. There’s a lot we’re not good at.

2 Likes
#4

Cool thank you for the answers! A blob storage API on fly (or an agreement with another provider) could be cool at some point.

#5

Yeah it’s on my list! I’m hoping we can figure this out this year.

2 Likes