We are going to start collecting charges for Stopped Machines RootFS starting April 25th

tl;dr: $0.15/GB of RootFS for a machine stopped for 30 days.

Back when we announced machines in May 2022 we were still figuring some billing for it. We mentioned we would charge for running machines the same as we did for Nomad VMs but for stopped machines we asked for community feedback.

Fast forward almost 2 years later (I can’t write a billing post without mentioning how bad we are at charging customers, ikr) and we are preparing things for finally doing it and I’m here to answer all your questions too. We are not activating any billing job and we are giving you 3 months heads up too. And this time we will nag everyone via email a lot too.

We are going to charge $0.15 per GB/month of used RootFS while your VM is stopped. And no this is not going to kill our free allowances (more below).

So what exactly are we charging you for?

Machines are fast booting VMs. That means we have to keep your built RootFS around so when the machine starts it already has all your NodeJS/Elixir/Erlang/whatever runtime is already installed.

Take for example the machine I just showed above: if I stop this machine, our servers will have a 905MB volume sitting there until the machine starts again, which could be in an hour or in months. We do have folks with stopped machines for months and those cost us.

What the heck is RootFS?

Whenever you spawn a machine we always give you a secret-not-so-secret volume called RootFS that is mounted at, you guessed it, your root directory. Here’s an example:

[I] ➜ fly ssh console -a fly-replay-app
root@6e82554b262e87:/app# df -h
Filesystem      Size  Used Avail Use% Mounted on
devtmpfs         97M     0   97M   0% /dev
/dev/vda        7.8G  191M  7.2G   3% /.      <------ LOOK HERE
shm             109M     0  109M   0% /dev/shm
tmpfs           109M     0  109M   0% /sys/fs/cgroup

Its usually called /dev/vda. This RootFS is 8GB long but only used 191MB. We are going to charge only for the used RootFS for stopped machines.

When you build your machines you generate an OCI image that is needed to be used in your deploys. When your machine is first created we unpack this image into a real File System on your machine.

How is this related to my actual machines anyways? RootFS is ephemeral meaning it resets after the machine is stopped and it goes back to the starting usage. Take from example the image above, my RootFS usage showd on the dashboard is 905MB meaning if I stop my machine it will need from the host server 905MB of storage as long as this machine is stopped. It does not matter if at runtime I download many GB of cache files or if I remove many MBs of files, the RootFS will still be 905MB as long as the machine is stopped.

How do I calculate this charge?

  • Usage is only calculated for stopped machines, if your machine is running 24/7 you don’t have to calculate anything, RootFS is already included on the price you already pay for running machines regardless of how much you use it.
  • Usage is calculated on the amount of RootFS needed when the machine is stopped. RootFS is ephemeral meaning anything you modify after the machine starts running will not be charged.
  • Each used GB costs $0.15 if it runs for 30 days, we prorate per hour.

Here’s a few examples:

My machine was stopped for 30 days this month and the rootfs usage is 100MB

That’s an easy one: $0.15 (1GB/month charge) * 0.10 (only 100MB) = $0.01

My machine spent 10 days stopped then ran for 20 days this month. It had 2GB of used rootfs.

$0.30 (2GB/month charge) * 0.33 (only 1/3 of the month) =~ $0.099

Is this the end of Fly.io free allowances?

No. Our free allowances let you run ‘Up to 3 shared-cpu-1x 256mb VMs’ 24/7. We also have been discounting up to $5.00 of usage on invoices so that means if you have 3 stopped machines for the entire month each having 1GB of RootFS usage that is $0.45 so far from breaking your free allowances.

How to verify how much I’m going to get charged?

We also shipped a preview in your invoice page that tells you how much dedicated usage you have so far in the current month prorated by hour. These will not be added to your invoice right now but by April 25th they will be added for May’s invoice.

We started collecting those preview charges in January 22th. Another way to see your usage is to go to our usage dashboard at https://fly.io/dashboard/personal/usage/stopped-machine.

How to verify I’m not going to be charged on the free allowances?

You can let your 3 shared-cpu-1x 256mb VMs running for the entire month or another way is to check that your stopped machines won’t add up to several GBs of RootFS usage together. You’d need more than 33GB of RootFS usage assuming all your machines were stopped for the entire month to exceed the $5.00 usage discount, for example.

How to pay less?

Another pattern y’all can find in my billing posts is “how to pay our company less money” so here we go.

It all depends on what you install on your OCI image. RootFS essentially is what your OCI image has after its packed + a few Containerd tweaks to make the file system usable (that’s why the value you see from docker build . might differ from our dashboard values for RootFS).

If your image install less stuff it means you need less base RootFS which means we are going to charge you even less. Believe it or not we do want you to have smaller images because that means faster builds too! Take @rubys work on dockerfile-node for example: we use multi-stage builds to make your images weight even less by not storing all the things needed just to compile your app like babel compiler or whatever.

So try multi-stage builds, try alpine if you want, it might save you some money and make your deploys even faster although you might want try to calculate if this is going to save you cents or fractions of cents. Unless you’re one of the customers racking several stopped machines you don’t really need to overthink this.

What’s next?

We are going to send emails. A lot of them. Plus we still need to show RootFS usage to our CLI, currently only our UI shows it. We need to update our pricing docs too!

We still want to hear it from you though. Let us know if you have any questions about all of this.

15 Likes

For those using Go and wondering how to avoid some fees here, this is the Dockerfile I use to avoid the almost 1GB Go binary:

FROM golang:1.21.6 as build

WORKDIR /app

COPY . .

RUN go mod download
RUN CGO_ENABLED=0 go build -o /go/bin/app

FROM gcr.io/distroless/base-debian12

COPY --from=build /go/bin/app /
COPY --from=build /app/content /content
COPY --from=build /app/pages /pages
COPY --from=build /app/public /public

CMD [ "/app" ]

My docker images went from almost a gig down to ~20-40MB. This is even more efficient than the buildpack that Fly uses by default. From my experience that is still good, but gives you around 100MB for the image. Hope this helps someone!

P.S. of course, the paths you copy depend on your project structure here. The rest can be used by any project.

4 Likes

While this was something we knew was coming, this technically means fly no longer has a zero cost autoscaling solution with scale-to-zero.

This won’t really impact our organization as a back-of-the-napkin calculation says we’ll only see roughly a $1 - $2 a month charge, but I could see it impacting other organizations a lot more.

5 Likes

I agree with @charsleysa . I don’t love what this incentivizes when it comes to auto scaling. On one hand the fee is rather small unless you’re operating at pretty large scale, on the other hand the fee existing at all sort of discourages (even if just slightly) the sort of scaling I think a lot of us aspire to see on fly (encouraging me to create fewer fat machines rather than many thin).

Assuming the costs are an issue, it essentially requires me to either create my own OCI image optimization trading startup to time for image size (download a large amount of data on machine startup, effectively delaying startup) or to be extremely cautious with provisioning and possibly create my own meta load balanced to clone and delete machines with the fly API.

Ultimately this will just shift some more machine management logic from the platform itself to developers (for example the FaaS example for machines would now need garbage collecting of seldom called functions implemented) which sort of rubs me the wrong way for something referred to as scale from 0. I’d much prefer a flat fee for images stored in your image repository (even if it was at a higher per gigabyte cost to amortize the average parallelization of machines).

2 Likes

One note about garbage collection though is that we have an auto_destroy flag upon machine creation so that you can ensure the machine will go from running to destroyed directly.

  • If you need to run code once and then the machine should be destroyed: use auto_destroy and exit when you finish the task.
  • If you need to constantly use a machine but don’t want to garbage collect it: use auto_destroy but do not immediately exit it, chose a trigger such as “exit when idle for X seconds” and then it will be garbage collected. Granted this has some caretaking from the developer too.
4 Likes

Does this also apply to App Builders as well? Understandably, the App Builder machines are also included in my Idle Machine Usage report, but on the App Builders page builder apps are listed as “Free builder”.
FreeBuilder
If they are indeed being charged for the idle machine usage, this might be slightly confusing, and might lead users to delete their App Builders.

2 Likes

They are and will still be free. We will update our usage UIs to tell this too asap, thanks for pointing that out!

6 Likes

For Go, that’s still a lot! Most application without C dependencies (they make static linking more complicated) should be able to easily get by with a binary, a timezone information database, and list of trusted root CAs.

I’ve done something like that using Earthly in the past (see below). It should be easy to replicate a similar setup using a multistage build in plain Docker.

3 Likes

But I delete the machines and my created content is lost… I create the content again and other machines are created that stop by themselves again and for that reason I am charged, what can I do? aid

True. My setup includes markdown files and static files the binary has to serve. Removing those gets you a much smaller image.

Hi @AdddrianLF

Im not sure I follow your question. You deleted old machines and created new ones? Can you explain a bit more, perhaps in another topic too (feel free to @ me)

Fair, fair - still most of the Linux distribution files can be cut out for a lot of Go apps.

@lubien Was there a recent change to how the estimated cost is calculated? I just noticed the amount get halved.

1 Like

Our query for our preview and usage data from Machine Events to Stopped Time showing more seconds than normal. Since this is a very expensive query we’ve been backfilling it so that’s why its lowering. Hopefully in a few more hours it will be fine and new values since last night should follow the correct query. Sorry about any jumpscares!

1 Like

Makes sense! :slight_smile:

The stopped machine page shows deleted machines. Shouldn’t they be excluded?

1 Like

We show machines deleted on the usage period (so if they were deleted this month they will appear there until the end of the month), can you verify that?

That being said it’s probably better for us if we update the UI to be more clear on that, thanks for pointing that out!

I see, but it doesn’t show any timestamp so I have no way to tell if it was deleted in the last 30 days or a year ago.

1 Like

We shipped a small tweak to the stopped machines usage page that should show the current machine state, including destroyed state :slight_smile:

Do I understand correctly that if my monthly invoice is below $5, then I won’t be charged?