New Feature: Graceful VM shutdown options

1

You can now give your VMs up to 24 hours to cleanly shutdown after receiving a kill signal. You can also specify the kill signal to send. Add these to your fly.toml to check it out:

kill_timeout: Number of seconds to wait before we force kill a VM. Minimum is 1 second, shared CPU VMs allow up to 5 minutes, dedicated CPU VMs allow up to 24 hours. Default is 5s.

kill_signal: Default is SIGINT, also allows SIGTERM , SIGQUIT , SIGUSR1 , SIGUSR2 , SIGKILL , or SIGSTOP.

If you’re curious how we terminate VMs, head to our blog to read more.

2

Would you consider shutdown scripts in addition to this?

  1. Execute the script
  2. Send signal

It’d be helpful for applications that have a http shutdown hook but shutdown too quickly on SIGINT/SIGTERM (eg Envoy and some other proxies)

The shutdown script would curl out to the shutdown http endpoint, sleep for 30/60s, then exit.

This is somewhat easily achieved with a sidecar container that can listen to the signal and start the slow and orderly shutdown of the main application.

Or send post to a global webhook receiver which could then trigger the shutdown logic based on instance id/ipv6, etc…

3

I’ve been doing a lot of work with this for DBs to make sure a leader steps down properly before it goes away. I think the right way to handle this is a supervisor process. I found a nice bash script to handle this:

prep_term()
{
    unset term_child_pid
    unset term_kill_needed
    trap 'handle_term' TERM INT
}
handle_term()
{
    if [ "${term_child_pid}" ]; then
        kill -TERM "${term_child_pid}" 2>/dev/null
    else
        term_kill_needed="yes"
    fi
}
wait_term()
{
    term_child_pid=$!
    if [ "${term_kill_needed}" ]; then
        kill -TERM "${term_child_pid}" 2>/dev/null 
    fi
    wait ${term_child_pid} 2>/dev/null
    trap - TERM INT
    wait ${term_child_pid} 2>/dev/null
}
# EXAMPLE USAGE
prep_term
/bin/something & # run your server
wait_term

You should be able to adjust wait_term to do what Envoy needs.

I did not expect to get good at bash while working on Fly but here we are.

Incidentally, if you feel like sharing your Envoy setup I think it would make a fabulous example app.

4

I was afraid of having to do this in bash, but your example is not as intimidating as I imagined it.

Once that’s done I’ll definitely share my envoy example.

Never thought I’d spend the last days of 2020 writing bash :sweat_smile:

5

It is strangely satisfying. So far it’s good enough that we haven’t built a simple process supervisor to do these things, but sometimes I really wish I had a simple Go binary to interpolate environment variables into a config file.

6

Would something like this help:

7

It would! envsubst is also useful: GNU gettext utilities: envsubst Invocation

I have discovered that I want to do more logic than just environment variable substitutions, though. Go with os.Exec has been a godsend.

8

Neat! It always amazes me what’s buried in the standard utilities. That’s one I hadn’t heard of.

As far a simple template rendering, I just found this:

I’m running some tests with that now for the same reason a your questions.