Configure console Machine size (including GPUs!) in fly.toml

Fresh Produce
1

If you’re not familiar with it, fly console lets you run a Rails console, Django’s manage.py, a Bash shell, or … well, pretty much any command you want in an ephemeral Machine, where you can work without disrupting your app in production. (Find the original Fresh Produce here!)

If you know you’ll need a lot of resources, then it’s convenient to configure the size of the console Machine in fly.toml using a [[vm]] section, just like you would for the rest of the Machines in your app. This was already possible—but for configuring RAM only. Also, it was kind of broken.

Well, we fixed it in flyctl v0.2.29, and in the process, we made it so that all of the [[vm]] section keys work. Just write a [[vm]] section that applies to the console process group. Something like this demo Ollama app:

app = "..."
primary_region = "ord"

[build]
  image = "ollama/ollama"

[[vm]]
  size = "performance-8x"
  memory = "16gb"
  gpu_kind = "l40s" # !!!
  processes = ["app", "console"]

Now, we’re not entirely sure what a console Machine with a GPU is useful for, but it does work, and it can be surprisingly quick to spin up—just make sure to run it in a region that has the GPU kind you’ve set (e.g. for the L40S, fly console --region ord).

Happy console-ing—and if you discover a good use for console Machines with GPUs, let us know! :grinning:

1 Like