Health check on port 8000 has failed. Your app is not responding properly. Services exposed on ports [80, 443] will have intermittent failures until the health check passes.

leonardohuttner · April 12, 2023, 1:54am

I have an error with django 4.0.10 when i deployed a new version with sentry.io monitoring and news endpoints. But this logs happens

Logs:

2023-04-12T01:45:23Z   [info]Preparing to run: `/bin/sh -c ["gunicorn", "--bind", ":8000", "--workers", "2", "api.wsgi:application" "--access-logfile '-'" "--error-logfile '-'" "--log-level debug"]` as root
2023-04-12T01:45:23Z   [info]2023/04/12 01:45:23 listening on [fdaa:1:28e9:a7b:e3:4271:d7c6:2]:22 (DNS: [fdaa::3]:53)
2023-04-12T01:45:24Z   [info]Starting clean up.
2023-04-12T01:45:29Z   [info]Starting instance
2023-04-12T01:45:29Z   [info]Configuring virtual machine
2023-04-12T01:45:29Z   [info]Pulling container image
2023-04-12T01:45:30Z   [info]Unpacking image
2023-04-12T01:45:30Z   [info]Preparing kernel init
2023-04-12T01:45:30Z   [info]Configuring firecracker
2023-04-12T01:45:30Z   [info]Starting virtual machine
2023-04-12T01:45:30Z   [info]Starting init (commit: e878f33)...
2023-04-12T01:45:30Z   [info]Preparing to run: `/bin/sh -c ["gunicorn", "--bind", ":8000", "--workers", "2", "api.wsgi:application" "--access-logfile '-'" "--error-logfile '-'" "--log-level debug"]` as root
2023-04-12T01:45:30Z   [info]2023/04/12 01:45:30 listening on [fdaa:1:28e9:a7b:e3:4271:d7c6:2]:22 (DNS: [fdaa::3]:53)
2023-04-12T01:45:31Z   [info]Starting clean up.
2023-04-12T01:45:32Z   [error]Health check on port 8000 has failed. Your app is not responding properly. Services exposed on ports [80, 443] will have intermittent failures until the health check passes.
--> v23 failed - Failed due to unhealthy allocations - rolling back to job version 22 and deploying as v24

Fly toml

# fly.toml file generated for api-jober on 2023-03-21T23:15:29-03:00

app = "api-jober"
kill_signal = "SIGINT"
kill_timeout = 5
processes = []

#[deploy]
#  release_command = "python manage.py migrate"

[env]
  PORT = "8000"

[experimental]
  auto_rollback = true

[[services]]
  http_checks = []
  internal_port = 8000
  processes = ["app"]
  protocol = "tcp"
  script_checks = []
  [services.concurrency]
    hard_limit = 25
    soft_limit = 20
    type = "connections"

  [[services.ports]]
    force_https = true
    handlers = ["http"]
    port = 80

  [[services.ports]]
    handlers = ["tls", "http"]
    port = 443

  [[services.tcp_checks]]
    grace_period = "1s"
    interval = "15s"
    restart_limit = 0
    timeout = "2s"

[[statics]]
  guest_path = "/app/public"
  url_prefix = "/static/"

kurt · April 12, 2023, 3:27am

fly status --all and fly vm status <id> may show you the app exiting with a weird exit code. It looks like it’s starting, then stopping very fast. But not really logging anything useful.

That error just means we can’t connect. Which would make sense if the process is crashing.

leonardohuttner · April 12, 2023, 11:44am

fly status --all

[Uploading: image.png...]()
App
  Name     = api-jober
  Owner    = personal
  Version  = 40
  Status   = dead
  Hostname = api-jober.fly.dev
  Platform = nomad

Instances
ID              PROCESS VERSION REGION  DESIRED STATUS          HEALTH CHECKS           RESTARTS        CREATED
4109efb7        app     40 ⇡    gru     run     failed          1 total                 2               8h22m ago
f8d8ff93        app     39      gru     stop    failed          1 total                 2               8h26m ago
0466fb7d        app     38      gru     run     failed                                  2               8h27m ago
955f6430        app     37      gru     stop    failed          1 total                 2               8h31m ago
66422024        app     36      gru     stop    failed          1 total                 2               8h34m ago
23d2863e        app     35      gru     stop    failed                                  2               8h32m ago
cdd00ccc        app     35      gru     stop    failed                                  2               8h34m ago
d654ca39        app     35      gru     stop    failed          1 total                 2               8h35m ago
609b4202        app     34      gru     stop    failed                                  2               8h39m ago
0c16a6c6        app     33      gru     stop    failed          1 total, 1 critical     2               8h44m ago
f5d2313a        app     32      gru     run     failed                                  2               8h45m ago
c06709f5        app     31      gru     stop    failed          1 total                 2               8h56m ago

fly vm status

2023-04-12T03:12:33.171 app[148ed555c0d289] gru [info] time="2023-04-12T03:12:33.171616242Z" level=debug msg="Cleaning up old mountid : done."

2023-04-12T03:12:33.171 app[148ed555c0d289] gru [info] time="2023-04-12T03:12:33.171749122Z" level=debug msg="unmounting daemon root" mountpoint=/data/docker

2023-04-12T03:12:33.172 app[148ed555c0d289] gru [info] time="2023-04-12T03:12:33.171893043Z" level=debug msg="Clean shutdown succeeded"

2023-04-12T03:12:33.172 app[148ed555c0d289] gru [info] time="2023-04-12T03:12:33.171912283Z" level=info msg="Daemon shutdown complete"

2023-04-12T03:12:33.172 app[148ed555c0d289] gru [info] time="2023-04-12T03:12:33.171948933Z" level=info msg="stopping healthcheck following graceful shutdown" module=libcontainerd

2023-04-12T03:12:33.172 app[148ed555c0d289] gru [info] time="2023-04-12T03:12:33.172003993Z" level=info msg="stopping event stream following graceful shutdown" error="context canceled" module=libcontainerd namespace=plugins.moby

2023-04-12T03:12:33.172 app[148ed555c0d289] gru [info] time="2023-04-12T03:12:33.172157883Z" level=debug msg="received signal" signal=terminated

2023-04-12T03:12:33.172 app[148ed555c0d289] gru [info] time="2023-04-12T03:12:33.172253853Z" level=debug msg="sd notification" error="<nil>" notified=false state="STOPPING=1"

2023-04-12T03:12:33.282 app[148ed555c0d289] gru [info] time="2023-04-12T03:12:33.282490919Z" level=debug msg="checking docker activity"

2023-04-12T03:12:33.283 app[148ed555c0d289] gru [info] time="2023-04-12T03:12:33.282785389Z" level=debug msg="Calling GET /v1.41/containers/json?filters=%7B%22status%22%3A%7B%22running%22%3Atrue%7D%7D&limit=0"

2023-04-12T03:12:34.178 app[148ed555c0d289] gru [info] time="2023-04-12T03:12:34.178268288Z" level=info msg="dockerd has exited"

2023-04-12T03:12:34.616 app[148ed555c0d289] gru [info] Starting clean up.

2023-04-12T03:12:34.616 app[148ed555c0d289] gru [info] Umounting /dev/vdb from /data

2023-04-12T03:12:35.989 app[148ed555c0d289] gru [info] [ 1693.605477] reboot: Restarting system

2023-04-12T03:12:37.414 runner[148ed555c0d289] gru [info] machine exited with exit code 0, not restarting

flyctl logs -a api-jober

$ flyctl logs -a api-jober
Update available 0.0.450 -> v0.0.513.
Run "flyctl.exe version update" to upgrade.
2023-04-12T02:59:24Z app[0466fb7d] gru [info]Starting init (commit: e878f33)...
2023-04-12T02:59:24Z app[0466fb7d] gru [info]Preparing to run: `/bin/sh -c ["gunicorn", "-b", ":8008", "--workers", "2", "api.wsgi:application" "--access-logfile '-'" "--error-logfile '-'" "--log-level debug"]` as root
2023-04-12T02:59:24Z app[0466fb7d] gru [info]2023/04/12 02:59:24 listening on [fdaa:1:28e9:a7b:de:466:fb7d:2]:22 (DNS: [fdaa::3]:53)      
2023-04-12T02:59:25Z app[0466fb7d] gru [info]Starting clean up.
2023-04-12T02:59:29Z runner[f8d8ff93] gru [info]Starting instance
2023-04-12T02:59:30Z runner[f8d8ff93] gru [info]Configuring virtual machine
2023-04-12T02:59:30Z runner[f8d8ff93] gru [info]Pulling container image
2023-04-12T02:59:30Z runner[f8d8ff93] gru [info]Unpacking image
2023-04-12T02:59:30Z runner[f8d8ff93] gru [info]Preparing kernel init
2023-04-12T02:59:30Z runner[f8d8ff93] gru [info]Configuring firecracker
2023-04-12T02:59:31Z runner[f8d8ff93] gru [info]Starting virtual machine
2023-04-12T02:59:31Z app[f8d8ff93] gru [info]Starting init (commit: e878f33)...
2023-04-12T02:59:31Z app[f8d8ff93] gru [info]Preparing to run: `/bin/sh -c ["gunicorn", "-b", ":8000", "--workers", "2", "api.wsgi:application" "--access-logfile '-'" "--error-logfile '-'" "--log-level debug"]` as root
2023-04-12T02:59:31Z app[f8d8ff93] gru [info]2023/04/12 02:59:31 listening on [fdaa:1:28e9:a7b:103:f8d8:ff93:2]:22 (DNS: [fdaa::3]:53)    
2023-04-12T02:59:32Z app[f8d8ff93] gru [info]Starting clean up.
2023-04-12T02:59:37Z runner[f8d8ff93] gru [info]Starting instance
2023-04-12T02:59:37Z runner[f8d8ff93] gru [info]Configuring virtual machine
2023-04-12T02:59:37Z runner[f8d8ff93] gru [info]Pulling container image
2023-04-12T02:59:37Z runner[f8d8ff93] gru [info]Unpacking image
2023-04-12T02:59:38Z runner[f8d8ff93] gru [info]Preparing kernel init
2023-04-12T02:59:38Z runner[f8d8ff93] gru [info]Configuring firecracker
2023-04-12T02:59:38Z runner[f8d8ff93] gru [info]Starting virtual machine
2023-04-12T02:59:38Z app[f8d8ff93] gru [info]Starting init (commit: e878f33)...
2023-04-12T02:59:38Z app[f8d8ff93] gru [info]Preparing to run: `/bin/sh -c ["gunicorn", "-b", ":8000", "--workers", "2", "api.wsgi:application" "--access-logfile '-'" "--error-logfile '-'" "--log-level debug"]` as root
2023-04-12T02:59:38Z app[f8d8ff93] gru [info]2023/04/12 02:59:38 listening on [fdaa:1:28e9:a7b:103:f8d8:ff93:2]:22 (DNS: [fdaa::3]:53)    
2023-04-12T02:59:39Z app[f8d8ff93] gru [info]Starting clean up.
2023-04-12T02:59:40Z health[f8d8ff93] gru [warn]Health check on port 80 is in a 'warning' state. Your app may not be responding properly. Services exposed on ports [80, 443] may have intermittent failures until the health check passes.
2023-04-12T02:59:44Z runner[f8d8ff93] gru [info]Starting instance
2023-04-12T02:59:44Z runner[f8d8ff93] gru [info]Configuring virtual machine
2023-04-12T02:59:44Z runner[f8d8ff93] gru [info]Pulling container image
2023-04-12T02:59:45Z runner[f8d8ff93] gru [info]Unpacking image
2023-04-12T02:59:45Z runner[f8d8ff93] gru [info]Preparing kernel init
2023-04-12T02:59:45Z runner[f8d8ff93] gru [info]Configuring firecracker
2023-04-12T02:59:45Z runner[f8d8ff93] gru [info]Starting virtual machine
2023-04-12T02:59:46Z app[f8d8ff93] gru [info]Starting init (commit: e878f33)...
2023-04-12T02:59:46Z app[f8d8ff93] gru [info]Preparing to run: `/bin/sh -c ["gunicorn", "-b", ":8000", "--workers", "2", "api.wsgi:application" "--access-logfile '-'" "--error-logfile '-'" "--log-level debug"]` as root
2023-04-12T02:59:46Z app[f8d8ff93] gru [info]2023/04/12 02:59:46 listening on [fdaa:1:28e9:a7b:103:f8d8:ff93:2]:22 (DNS: [fdaa::3]:53)    
2023-04-12T02:59:47Z app[f8d8ff93] gru [info]Starting clean up.
error.message="timed out while connecting to your instance. this indicates a problem with your app (hint: look at your logs and metrics)" 2023-04-12T03:00:00Z proxy[f8d8ff93] gru [error]request.method="GET" request.id="01GXSPFPF3CFZN3R4E0XTFB2F6-gru"
error.message="no known healthy instances found for route tcp/443. (hint: is your app shutdown? is there an ongoing deployment with a volume or using the 'immediate' strategy? if not, this could be a delayed state issue)" 2023-04-12T03:00:02Z proxy gru [warn]request.method="GET" request.url="https://api-jober.fly.dev/" request.id="01GXSPFSTM1195PA0ZJ1W0ZJTY-gru"
error.message="no known healthy instances found for route tcp/443. (hint: is your app shutdown? is there an ongoing deployment with a volume or using the 'immediate' strategy? if not, this could be a delayed state issue)" 2023-04-12T03:00:06Z proxy gru [warn]request.method="GET" request.url="https://api-jober.fly.dev/" request.id="01GXSPFSTM1195PA0ZJ1W0ZJTY-gru"
error.message="no known healthy instances found for route tcp/443. (hint: is your app shutdown? is there an ongoing deployment with a volume or using the 'immediate' strategy? if not, this could be a delayed state issue)" 2023-04-12T03:00:16Z proxy gru [warn]request.method="GET" request.url="https://api-jober.fly.dev/" request.id="01GXSPFSTM1195PA0ZJ1W0ZJTY-gru"
error.message="no known healthy instances found for route tcp/443. (hint: is your app shutdown? is there an ongoing deployment with a volume or using the 'immediate' strategy? if not, this could be a delayed state issue)" 2023-04-12T03:00:26Z proxy gru [warn]request.method="GET" request.url="https://api-jober.fly.dev/" request.id="01GXSPFSTM1195PA0ZJ1W0ZJTY-gru"
error.message="no known healthy instances found for route tcp/443. (hint: is your app shutdown? is there an ongoing deployment with a volume or using the 'immediate' strategy? if not, this could be a delayed state issue)" 2023-04-12T03:00:36Z proxy gru [warn]request.method="GET" request.url="https://api-jober.fly.dev/" request.id="01GXSPFSTM1195PA0ZJ1W0ZJTY-gru"
error.message="no known healthy instances found for route tcp/443. (hint: is your app shutdown? is there an ongoing deployment with a volume or using the 'immediate' strategy? if not, this could be a delayed state issue)" 2023-04-12T03:00:46Z proxy gru [warn]request.method="GET" request.url="https://api-jober.fly.dev/" request.id="01GXSPFSTM1195PA0ZJ1W0ZJTY-gru"
error.message="no known healthy instances found for route tcp/443. (hint: is your app shutdown? is there an ongoing deployment with a volume or using the 'immediate' strategy? if not, this could be a delayed state issue)" 2023-04-12T03:00:56Z proxy gru [warn]request.method="GET" request.url="https://api-jober.fly.dev/" request.id="01GXSPFSTM1195PA0ZJ1W0ZJTY-gru"
error.message="no known healthy instances found for route tcp/443. (hint: is your app shutdown? is there an ongoing deployment with a volume or using the 'immediate' strategy? if not, this could be a delayed state issue)" 2023-04-12T03:01:06Z proxy gru [warn]request.method="GET" request.url="https://api-jober.fly.dev/" request.id="01GXSPFSTM1195PA0ZJ1W0ZJTY-gru"
error.message="no known healthy instances found for route tcp/443. (hint: is your app shutdown? is there an ongoing deployment with a volume or using the 'immediate' strategy? if not, this could be a delayed state issue)" 2023-04-12T03:01:15Z proxy gru [warn]request.method="GET" request.url="https://api-jober.fly.dev/" request.id="01GXSPFSTM1195PA0ZJ1W0ZJTY-gru"
error.message="no known healthy instances found for route tcp/443. (hint: is your app shutdown? is there an ongoing deployment with a volume or using the 'immediate' strategy? if not, this could be a delayed state issue)" 2023-04-12T03:01:25Z proxy gru [warn]request.method="GET" request.url="https://api-jober.fly.dev/" request.id="01GXSPFSTM1195PA0ZJ1W0ZJTY-gru"
error.message="no known healthy instances found for route tcp/443. (hint: is your app shutdown? is there an ongoing deployment with a volume or using the 'immediate' strategy? if not, this could be a delayed state issue)" 2023-04-12T03:01:26Z proxy gru [error]request.method="GET" request.url="https://api-jober.fly.dev/" request.id="01GXSPFSTM1195PA0ZJ1W0ZJTY-gru" response.status=503
2023-04-12T03:02:59Z runner[34df7642] gru [info]Starting instance
2023-04-12T03:02:59Z runner[34df7642] gru [info]Configuring virtual machine
2023-04-12T03:02:59Z runner[34df7642] gru [info]Pulling container image
2023-04-12T03:03:02Z runner[34df7642] gru [info]Unpacking image
2023-04-12T03:03:06Z runner[34df7642] gru [info]Preparing kernel init
2023-04-12T03:03:06Z runner[34df7642] gru [info]Configuring firecracker
2023-04-12T03:03:06Z runner[34df7642] gru [info]Starting virtual machine
2023-04-12T03:03:06Z app[34df7642] gru [info]Starting init (commit: e878f33)...
2023-04-12T03:03:06Z app[34df7642] gru [info]Preparing to run: `python manage.py migrate` as root
2023-04-12T03:03:06Z app[34df7642] gru [info]2023/04/12 03:03:06 listening on [fdaa:1:28e9:a7b:de:34df:7642:2]:22 (DNS: [fdaa::3]:53)     
2023-04-12T03:03:09Z app[34df7642] gru [info]Operations to perform:
2023-04-12T03:03:09Z app[34df7642] gru [info]  Apply all migrations: admin, auth, contenttypes, sessions, v1
2023-04-12T03:03:09Z app[34df7642] gru [info]Running migrations:
2023-04-12T03:03:09Z app[34df7642] gru [info]  No migrations to apply.
2023-04-12T03:03:09Z app[34df7642] gru [info]Starting clean up.
2023-04-12T03:03:16Z runner[4109efb7] gru [info]Starting instance
2023-04-12T03:03:16Z runner[4109efb7] gru [info]Configuring virtual machine
2023-04-12T03:03:16Z runner[4109efb7] gru [info]Pulling container image
2023-04-12T03:03:16Z runner[4109efb7] gru [info]Unpacking image
2023-04-12T03:03:16Z runner[4109efb7] gru [info]Preparing kernel init
2023-04-12T03:03:16Z runner[4109efb7] gru [info]Configuring firecracker
2023-04-12T03:03:19Z runner[4109efb7] gru [info]Starting virtual machine
2023-04-12T03:03:19Z app[4109efb7] gru [info]Starting init (commit: e878f33)...
2023-04-12T03:03:19Z app[4109efb7] gru [info]Preparing to run: `/bin/sh -c ["gunicorn", "-b", ":8000", "--workers", "2", "api.wsgi:application" "--access-logfile '-'" "--error-logfile '-'" "--log-level debug"]` as root
2023-04-12T03:03:19Z app[4109efb7] gru [info]2023/04/12 03:03:19 listening on [fdaa:1:28e9:a7b:de:4109:efb7:2]:22 (DNS: [fdaa::3]:53)     
2023-04-12T03:03:20Z app[4109efb7] gru [info]Starting clean up.
2023-04-12T03:03:21Z health[4109efb7] gru [warn]Health check on port 8000 is in a 'warning' state. Your app may not be responding properly. Services exposed on ports [80, 443] may have intermittent failures until the health check passes.
2023-04-12T03:03:25Z runner[4109efb7] gru [info]Starting instance
2023-04-12T03:03:25Z runner[4109efb7] gru [info]Configuring virtual machine
2023-04-12T03:03:25Z runner[4109efb7] gru [info]Pulling container image
2023-04-12T03:03:26Z runner[4109efb7] gru [info]Unpacking image
2023-04-12T03:03:26Z runner[4109efb7] gru [info]Preparing kernel init
2023-04-12T03:03:26Z runner[4109efb7] gru [info]Configuring firecracker
2023-04-12T03:03:27Z runner[4109efb7] gru [info]Starting virtual machine
2023-04-12T03:03:27Z app[4109efb7] gru [info]Starting init (commit: e878f33)...
2023-04-12T03:03:27Z app[4109efb7] gru [info]Preparing to run: `/bin/sh -c ["gunicorn", "-b", ":8000", "--workers", "2", "api.wsgi:application" "--access-logfile '-'" "--error-logfile '-'" "--log-level debug"]` as root
2023-04-12T03:03:27Z app[4109efb7] gru [info]2023/04/12 03:03:27 listening on [fdaa:1:28e9:a7b:de:4109:efb7:2]:22 (DNS: [fdaa::3]:53)     
2023-04-12T03:03:28Z app[4109efb7] gru [info]Starting clean up.
2023-04-12T03:03:33Z runner[4109efb7] gru [info]Starting instance
2023-04-12T03:03:33Z runner[4109efb7] gru [info]Configuring virtual machine
2023-04-12T03:03:33Z runner[4109efb7] gru [info]Pulling container image
2023-04-12T03:03:33Z runner[4109efb7] gru [info]Unpacking image
2023-04-12T03:03:33Z runner[4109efb7] gru [info]Preparing kernel init
2023-04-12T03:03:34Z runner[4109efb7] gru [info]Configuring firecracker
2023-04-12T03:03:34Z runner[4109efb7] gru [info]Starting virtual machine
2023-04-12T03:03:34Z app[4109efb7] gru [info]Starting init (commit: e878f33)...
2023-04-12T03:03:34Z app[4109efb7] gru [info]Preparing to run: `/bin/sh -c ["gunicorn", "-b", ":8000", "--workers", "2", "api.wsgi:application" "--access-logfile '-'" "--error-logfile '-'" "--log-level debug"]` as root
2023-04-12T03:03:34Z app[4109efb7] gru [info]2023/04/12 03:03:34 listening on [fdaa:1:28e9:a7b:de:4109:efb7:2]:22 (DNS: [fdaa::3]:53)     
2023-04-12T03:03:35Z app[4109efb7] gru [info]Starting clean up.
2023-04-12T03:03:36Z health[4109efb7] gru [warn]Health check on port 8000 is in a 'warning' state. Your app may not be responding properly. Services exposed on ports [80, 443] may have intermittent failures until the health check passes.

Cy11 · April 12, 2023, 12:41pm

I’m having a similar issue over here with my Rust Web App. Deployments worked fine before, I haven’t changed anything.

Cy11 · April 12, 2023, 3:10pm

Since I posted that comment, the issue has gone away even though I haven’t changed anything. Might be worth trying to deploy again.

leonardohuttner · April 13, 2023, 2:55pm

i tried yesterday, but yet not working

leonardohuttner · April 13, 2023, 3:37pm

The django version was switched to 4.1 and gunicorn 20.1, and it worked without further changes.

Topic		Replies	Views
Health check on port 8080 has failed Questions / Help postgres	2	694	March 21, 2023
After updating to v2 and v3 automatically my service has been interrupted and my application does not work	6	329	August 25, 2023
Failing Health Checks after Deploy Questions / Help	3	1506	January 22, 2022
Virtual machine repeatedly shutting down Build debugging	2	1173	November 18, 2022
Health check failure Questions / Help	6	902	April 26, 2023

Health check on port 8000 has failed. Your app is not responding properly. Services exposed on ports [80, 443] will have intermittent failures until the health check passes.

Related Topics