Smoke Checks during Apps V2 deploy

Fresh Produce
1

Deploys fail, especially if you just started developing an app. Sometimes,
a machine is not properly sized and the app is terminated by OOM killer,
sometimes, a mandatory environment variable is not provided and the app can’t
start. Health checks help, but may be hard to diagnose - a failing health check
doesn’t really tell you what’s wrong.

Starting from version v0.0.553, flyctl will perform smoke checks during Apps V2 deploy.

What’s checked

During deploy, after a machine has transitioned to the started state,
flyctl will check for 10 seconds whether the machine is up and running and not
constantly restarting (crashing) with a non-zero exit code.

If flyctl detects that the app is crashing, it will abort the deployment and dump
the last log lines onto the terminal to help you diagnose what’s happening.

How it looks

❯ fly deploy
==> Verifying app config
Validating /home/pbor/apps/smoke-checks-test/fly.toml
Platform: machines
✓ Configuration is valid
--> Verified app config
==> Building image
Searching for image 'library/mariadb:10' remotely...
image found: img_8y6w4z091k2p7rn3

Watch your app at https://fly.io/apps/smoke-checks-test/monitoring

Updating existing machines in 'smoke-checks-test' with rolling strategy
  Machine 28744e2b071248 [app] has state: started
  [1/1] Checking that 28744e2b071248 [app] is up and running
Smoke checks for 28744e2b071248 failed: the app appears to be crashing
Check its logs: here's the last lines below, or run 'fly logs -i 28744e2b071248':
  Starting init (commit: 15c0f38)...
  Preparing to run: `docker-entrypoint.sh mariadbd` as root
  2023/05/03 10:11:31 listening on [fdaa:1:d615:a7b:e0:935f:9112:2]:22 (DNS: [fdaa::3]:53)
  2023-05-03 10:11:31+00:00 [Note] [Entrypoint]: Entrypoint script for MariaDB Server 1:10.11.2+maria~ubu2204 started.
  2023-05-03 10:11:31+00:00 [Note] [Entrypoint]: Switching to dedicated user 'mysql'
  2023-05-03 10:11:31+00:00 [Note] [Entrypoint]: Entrypoint script for MariaDB Server 1:10.11.2+maria~ubu2204 started.
  2023-05-03 10:11:31+00:00 [ERROR] [Entrypoint]: Database is uninitialized and password option is not specified
        You need to specify one of MARIADB_ROOT_PASSWORD, MARIADB_ROOT_PASSWORD_HASH, MARIADB_ALLOW_EMPTY_ROOT_PASSWORD and MARIADB_RANDOM_ROOT_PASSWORD
  Starting clean up.
  [    2.114360] reboot: Restarting system

How to disable

If you don’t really need the checks, they can be disabled by passing --smoke-checks=false flag to fly deploy.

If you have feedback, comments or questions, please share!

9 Likes