New Feature: Graceful VM shutdown options

You can now give your VMs up to 24 hours to cleanly shutdown after receiving a kill signal. You can also specify the kill signal to send. Add these to your fly.toml to check it out:

kill_timeout: Number of seconds to wait before we force kill a VM. Minimum is 1 second, shared CPU VMs allow up to 5 minutes, dedicated CPU VMs allow up to 24 hours. Default is 5s.

kill_signal: Default is SIGINT, also allows SIGTERM , SIGQUIT , SIGUSR1 , SIGUSR2 , SIGKILL , or SIGSTOP.

If you’re curious how we terminate VMs, head to our blog to read more.


Would you consider shutdown scripts in addition to this?

  1. Execute the script
  2. Send signal

It’d be helpful for applications that have a http shutdown hook but shutdown too quickly on SIGINT/SIGTERM (eg Envoy and some other proxies)

The shutdown script would curl out to the shutdown http endpoint, sleep for 30/60s, then exit.

This is somewhat easily achieved with a sidecar container that can listen to the signal and start the slow and orderly shutdown of the main application.

Or send post to a global webhook receiver which could then trigger the shutdown logic based on instance id/ipv6, etc…

I’ve been doing a lot of work with this for DBs to make sure a leader steps down properly before it goes away. I think the right way to handle this is a supervisor process. I found a nice bash script to handle this:

    unset term_child_pid
    unset term_kill_needed
    trap 'handle_term' TERM INT
    if [ "${term_child_pid}" ]; then
        kill -TERM "${term_child_pid}" 2>/dev/null
    if [ "${term_kill_needed}" ]; then
        kill -TERM "${term_child_pid}" 2>/dev/null 
    wait ${term_child_pid} 2>/dev/null
    trap - TERM INT
    wait ${term_child_pid} 2>/dev/null
/bin/something & # run your server

You should be able to adjust wait_term to do what Envoy needs.

I did not expect to get good at bash while working on Fly but here we are.

Incidentally, if you feel like sharing your Envoy setup I think it would make a fabulous example app.

1 Like

I was afraid of having to do this in bash, but your example is not as intimidating as I imagined it.

Once that’s done I’ll definitely share my envoy example.

Never thought I’d spend the last days of 2020 writing bash :sweat_smile:

1 Like

It is strangely satisfying. So far it’s good enough that we haven’t built a simple process supervisor to do these things, but sometimes I really wish I had a simple Go binary to interpolate environment variables into a config file.

Would something like this help:

It would! envsubst is also useful: GNU gettext utilities: envsubst Invocation

I have discovered that I want to do more logic than just environment variable substitutions, though. Go with os.Exec has been a godsend.

Neat! It always amazes me what’s buried in the standard utilities. That’s one I hadn’t heard of.

As far a simple template rendering, I just found this:

I’m running some tests with that now for the same reason a your questions.