Autoscaling on CPU utilization?

[pier]

So how my setup works is I have an app with multiple machines created beforehand that can handle peak demand. Asleep machines are very cheap (can’t find the pricing now) so it’s ok to just manually create as many as you think you will need.

In the future I will create machines on demand but I haven’t had the time to play with the machines API yet. And this setup is working great so far.

I explained a scale to zero approach in this post:

Implementing scale to zero is super easy!

Just implemented a Node v2 app with multiple machines to encode audio with scale to zero. Couldn’t have been easier. 1. Sleep your app after some inactivity This will be different depending on what you’re using. In Node it can be as simple as doing this, and updating the timeout on every new request. let timeout; export function resetCountdownSleep () { clearTimeout(timeout); timeout = setTimeout(() => { process.exit(); }, 60000); } In this case, the server will sleep after 60 seconds…

I also have another app, let’s call it the job worker, which is permanently awake and orchestrates the whole thing. It responds to a trigger in the Jobs table in Postgres with listen/notify. It could some pub/sub service too.

So the worker sends the HTTP requests to the encoding app to trigger the audio encoding. Basically, Fly’s routing layer wakes the machines as needed when new requests are made to the encoding app. With the auto scaling settings (see the post) Fly will try to use a single request per machine as long as there are available machines to wake up.

One thing to take into consideration is that requests will timeout after 60 seconds if no bytes go through Fly routing layer. So either complete the job before 60s, or stream some bytes so that Fly doesn’t timeout, or just respond the request asap and then do the CPU job.

@jakeshead

How do you track the number of concurrent encodings across your cluster? (I’m assuming Redis or Postgres).

I don’t really track it, but if I needed to do it I could just do something like this:

select * from "AudioEncoding" where "state" = "ENCODING"

@jakeshead

How do you avoid all of your machines shutting down?

Like I said, all the encoding machines shut down eventually. The job worker who orchestrates all this is always awake.

@jakeshead

the tricky part is that this kicks off a minutes long job, which I run asynchronously, so from the fly proxy’s perspective that machine is no longer serving a request

Like I said, to solve this you could stream some bytes before ending the response. Streaming is a bit of a headache though because you can’t change the HTTP status and there are other considerations.

If you don’t have control over the app making the request, you could ask them to give you a webhook URL you could call when the long running task is complete.