About a year ago, Kurt found a secret stash of A100s somewhere and we thought, “hmm, wouldn’t it be great if we could get Fly machines to work with GPUs?” We’ve been working on the idea ever since. Right now we have new A100 40G PCIes and 80G SXMs (read - big bois) coming online every day. They live in
syd with more regions to come.
We wanted working with GPUs to feel a lot like working with Fly machines, so that’s what we built - machines that are attached to GPUs. As an aside, Firecracker doesn’t play well with GPUs so we used Cloud Hypervisor, but the experience of working with these VMs is very similar.
To deploy an app on a GPU machine, you’d run something like this in flyctl (but don’t do it now, it won’t work until your org is GPU-blessed):
fly deploy --vm-gpu-kind a100-pcie-40gb --volume-initial-size 100
Judging by our first batch of GPU users, a variety of things!
AI is, of course, popular - mostly inference with existing open-source models but also with custom models. We’re not set up for training large models, but I could imagine someone using us to fine-tune the last few layers of a model on their own data.
Some users have asked about rendering, and we’re into that. And of course some of you are exploring running your platform on top of Fly GPUs, which we love.
It’s still early, but you can check out some of the demos we’ve put together in our docs.
Right now, the on-demand cost is $2.50/hr per GPU for the A100 40G PCIes and $3.50/hr per GPU for the A100 80G SXMs. There’s no minimums for usage. You decide the CPU, RAM, and storage you need. We have discounted pricing for reserved GPU machines and dedicated hosts.
GPU access requires a paid plan that includes support. We’re not trying to add to the cost - the $29/month Launch plan comes with $29/month of usage. If you think you’ll use $29/month worth of GPUs and accoutrements (CPUs, volumes, etc), then you’re signing up for a free support email and help from our team.
We have been reaching out to folks on the waitlist to learn more about their use cases, and if we think we can help them do what they’re trying to do, we give them GPUs.
We can’t wait to open these for anyone to experiment on, but we’re focused on company-esque entities right now. Like, you have a product with users and someone we can talk to about future plans and capacity requirements. This will change, but it means we can get GPUs out faster so we’re doing it.
What are YOU interested in building with Fly GPUs? What other GPU-enabled shenanigans would you like to see Fly support? Leave us a note.