I’m working on a project that requires that I download many fairly large file (400mb - 5GB), process them in place on a volume, and then upload them to a blob storage provider like S3, B2, or R2. Downloading files is no problem, but re-uploading the results in outbound traffic.
I would preferably like to do this work on Fly, since the main app lives here, but I’m nervous about incurring the cost of that outbound traffic.
Are there any above board “tricks” that can be used to get data into blob storage from Fly without that outbound cost? Perhaps something with wireguard? Or if it becomes a problem, does this compute need to live in-cloud near the blob storage that is used to avoid this internet traffic?
If you’re going for R2, Cloudflare’s bandwidth alliance partners are a better fit for your compute (like Vultr, Alibaba Cloud, Oracle Cloud, Scaleway). Fly I don’t think is part of that alliance. Also, I believe, you may need to subscribe to Cloudflare’s Business ($200/mo) / Enterprise ($5000/mo?) plan for that.
Fly is a non-starter, unless your usage warrants a custom agreement with Fly (Fly already handles Tailscale VPN’s data-plane traffic (ref) and it is highly unlikely they’re paying a premium on bandwidth).
You may know already that there are providers like Hetzner / OVH / Dreamhost / Scaleway that do sell uncapped bandwidth (or for as low as $1/TB); so not all hope is lost.