How to speed up start-up time for a Bumblebee app on Fly.io?

Questions / Help
1

Hey there :wave:

I have a simple application that is running Bumblebee, a framework in Elixir that allows me to run machine learning models with Phoenix. The repo is at GitHub - dwyl/image-classifier: 🖼️ Classify images and attempt to extract data from or describe their contents.

I have this application deployed on fly.io, which sleeps after a period of inactivity. I’ve followed the instructions in Speed up your boot times with this one Dockerfile trick · The Phoenix Files to speed up the boot time of the fly.io instance and it works to a certain extent - the model is properly cached in a volume and is not re-downloaded every time the instance is back up, being loaded when the app goes from sleep to active.

I’ve inclusively created a small guide to deploy this with fly.io, in case anyone is interested. This was also discussed in Build failed with elixir bumblebee - #4 by matthewford.

While the app works normally, even if the models are cached, the amount of time the app takes to go from sleeping to active is roughly 25 seconds (I’m using a fairly large model - Salesforce/blip-image-captioning-large · Hugging Face - on a performance-4x machine instance). I don’t know if this is normal or not and I realise it might be beyond my control.

2023-11-21T09:33:01.113 -  Starting machine
2023-11-21T09:33:26.140 - Access AppWeb.Endpoint at https://imgai.fly.dev

I assume it’s taking 25 seconds to load the model into the CPU.

However, is there anything I can do to hopefully reduce this time? This is very much niche, but I’m curious if there’s anything I can do in practice besides scaling up the machines.

Thank you so much for reading :slight_smile:

2

What you will want to do is add a Volume to store your model instead of in the dockerfile, the larger the docker image gets the more lag it adds to everything.

So add a volume to your app, then set BUMBLEBEE_CACHE_DIR="/data/cache/bumblebee" or w/e you call the volume.

1 Like
3

Thanks for the reply!

That’s effectively what I’ve done. I’ve linked the article just so people know that I tried to follow it. I already have a volume attached and the models are being cached and loaded on start-up without re-downloading.

I’m just wondering if there’s anything else I can do to make the boot time even better :smiley:

4

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.