Health check failure

Questions / Help
1

Hey,

When deploying my app it seems like it:

  1. Starts the container
  2. Fails the healthcheck (because the server hasn’t fully started yet)
  3. The server starts
  4. Waits 3.5min until the runner shuts down the virtual machine

Logs as well as the fly.toml file attached below:

 2023-04-19T14:35:17.856 runner[f0312282] arn [info] Starting instance

2023-04-19T14:35:17.896 runner[f0312282] arn [info] Configuring virtual machine

2023-04-19T14:35:17.902 runner[f0312282] arn [info] Pulling container image

2023-04-19T14:36:00.892 runner[f0312282] arn [info] Unpacking image

2023-04-19T14:36:42.793 runner[f0312282] arn [info] Preparing kernel init

2023-04-19T14:36:43.143 runner[f0312282] arn [info] Configuring firecracker

2023-04-19T14:36:43.278 runner[f0312282] arn [info] Starting virtual machine

2023-04-19T14:36:43.468 app[f0312282] arn [info] Starting init (commit: ed64554)...

2023-04-19T14:36:43.498 app[f0312282] arn [info] Preparing to run: `/cnb/process/web` as 1000

2023-04-19T14:36:43.515 app[f0312282] arn [info] 2023/04/19 14:36:43 listening on [fdaa:1:700f:a7b:e8:f031:2282:2]:22 (DNS: [fdaa::3]:53)

2023-04-19T14:36:44.823 health[f0312282] arn [warn] Health check on port 8080 is in a 'warning' state. Your app may not be responding properly. Services exposed on ports [80, 443] may have intermittent failures until the health check passes.

2023-04-19T14:36:49.823 health[f0312282] arn [error] Health check on port 8080 has failed. Your app is not responding properly. Services exposed on ports [80, 443] will have intermittent failures until the health check passes.

2023-04-19T14:36:50.965 app[f0312282] arn [info] [nltk_data] Downloading package punkt to /home/cnb/nltk_data...

2023-04-19T14:36:51.029 app[f0312282] arn [info] [nltk_data] Unzipping tokenizers/punkt.zip.

2023-04-19T14:36:51.297 app[f0312282] arn [info] [nltk_data] Downloading package stopwords to /home/cnb/nltk_data...

2023-04-19T14:36:51.312 app[f0312282] arn [info] [nltk_data] Unzipping corpora/stopwords.zip.

2023-04-19T14:36:57.397 app[f0312282] arn [info] INFO: Started server process [520]

2023-04-19T14:36:57.397 app[f0312282] arn [info] INFO: Waiting for application startup.

2023-04-19T14:36:59.042 app[f0312282] arn [info] INFO: Application startup complete.

2023-04-19T14:36:59.043 app[f0312282] arn [info] INFO: Uvicorn running on http://127.0.0.1:8080 (Press CTRL+C to quit)

2023-04-19T14:40:31.753 runner[f0312282] arn [info] Shutting down virtual machine

2023-04-19T14:40:31.878 app[f0312282] arn [info] Sending signal SIGINT to main child process w/ PID 520

2023-04-19T14:40:31.929 app[f0312282] arn [info] INFO: Shutting down

2023-04-19T14:40:32.029 app[f0312282] arn [info] INFO: Waiting for application shutdown.

2023-04-19T14:40:32.030 app[f0312282] arn [info] INFO: Application shutdown complete.

2023-04-19T14:40:32.030 app[f0312282] arn [info] INFO: Finished server process [520]

2023-04-19T14:40:32.030 app[f0312282] arn [info] loading bm25

2023-04-19T14:40:33.786 app[f0312282] arn [info] Starting clean up.

fly.toml

app = "name"
kill_signal = "SIGINT"
kill_timeout = 5
processes = []

[build]
  builder = "paketobuildpacks/builder:full"

[env]
  PORT = "8080"

[experimental]
  auto_rollback = true

[[services]]
  http_checks = []
  internal_port = 8080
  processes = ["app"]
  protocol = "tcp"
  script_checks = []
  [services.concurrency]
    hard_limit = 25
    soft_limit = 20
    type = "connections"

  [[services.ports]]
    force_https = true
    handlers = ["http"]
    port = 80

  [[services.ports]]
    handlers = ["tls", "http"]
    port = 443

  [[services.tcp_checks]]
    grace_period = "1s"
    interval = "15s"
    restart_limit = 0
    timeout = "2s"
  [services.ports.http_options.response.headers]
    ...
2

Hi @johan1

Can you reach your app through the app-name.fly.dev domain? If it’s not reachable it’s an indicator that the app is not listening to the correct port and from the log line above it seems your app is listening 22 instead of 8080 which is configured on internal_port = 8080.

3

I can still access the previous version of the app. The two main changes I’ve made since then are:

  • uvicorn (web: uvicorn app:app --port 8080) instead of gunicorn
  • builder = "paketobuildpacks/builder:full" instead of builder = "paketobuildpacks/builder:base"
4

I also tried extending the graceperiod in fly.toml: grace_period = "120s"

5

One note about my previous message: that thing listening on port 22 is hallpass not your app, my fault.

Looking again at your logs I can see your app is listening to 127.0.0.1. Your app needs to listen on 0.0.0.0:8080. Looking at uvicorn docs you might be able to do that with --host 0.0.0.0. Let me know if that helps

6

That worked, thank you!

1 Like
7

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.