TL;DR
You can deploy GPU machines now, and we’re letting people in off the waitlist to try them. Sign up here if you want one.
Longer Version
About a year ago, Kurt found a secret stash of A100s somewhere and we thought, “hmm, wouldn’t it be great if we could get Fly machines to work with GPUs?” We’ve been working on the idea ever since. Right now we have new A100 40G PCIes and 80G SXMs (read - big bois) coming online every day. They live in ord, iad, ams, sjc, and syd with more regions to come.
How Do I Use These?
We wanted working with GPUs to feel a lot like working with Fly machines, so that’s what we built - machines that are attached to GPUs. As an aside, Firecracker doesn’t play well with GPUs so we used Cloud Hypervisor, but the experience of working with these VMs is very similar.
To deploy an app on a GPU machine, you’d run something like this in flyctl (but don’t do it now, it won’t work until your org is GPU-blessed):
fly deploy --vm-gpu-kind a100-pcie-40gb --volume-initial-size 100
What Can I Use Them For?
Judging by our first batch of GPU users, a variety of things!
AI is, of course, popular - mostly inference with existing open-source models but also with custom models. We’re not set up for training large models, but I could imagine someone using us to fine-tune the last few layers of a model on their own data.
Some users have asked about rendering, and we’re into that. And of course some of you are exploring running your platform on top of Fly GPUs, which we love.
It’s still early, but you can check out some of the demos we’ve put together in our docs.
What’s the Cost?
Right now, the on-demand cost is $2.50/hr per GPU for the A100 40G PCIes and $3.50/hr per GPU for the A100 80G SXMs. There’s no minimums for usage. You decide the CPU, RAM, and storage you need. We have discounted pricing for reserved GPU machines and dedicated hosts.
GPU access requires a paid plan that includes support. We’re not trying to add to the cost - the $29/month Launch plan comes with $29/month of usage. If you think you’ll use $29/month worth of GPUs and accoutrements (CPUs, volumes, etc), then you’re signing up for a free support email and help from our team.
How Do I Get One?
We have been reaching out to folks on the waitlist to learn more about their use cases, and if we think we can help them do what they’re trying to do, we give them GPUs.
We can’t wait to open these for anyone to experiment on, but we’re focused on company-esque entities right now. Like, you have a product with users and someone we can talk to about future plans and capacity requirements. This will change, but it means we can get GPUs out faster so we’re doing it.
All of this is to say: sign up! If you leave a good explanation of who you are and what you want to build with GPUs on the waitlist, you’ll hear from us pretty quickly.
What are YOU interested in building with Fly GPUs? What other GPU-enabled shenanigans would you like to see Fly support? Leave us a note.
I wrote a thing on using GPU machines, which has some examples/info on how to work with GPU Machines when you get access:
on using 