Healthcheck fails even when application is available through browser

Questions / Help
1

Hello. I have a problem with healthcheck which fails. I spent already 10 hours to solve it. My application is available from browser with “https://my-app.fly.dev”, it’s available on VM through SSH with “curl -I http://0.0.0.0:3000/”, but healthcheck fails and it stops the building.
On first fail there was flyctl 0.3.45, but after that I upgraded it to the latest 0.3.87. It didn’t help

I didn’t change anything in .toml config since last successful deployment, but the latest deployment was 3 months ago. So I expected changes from Fly side.
I tried to remove all machines and created a new one. Many experiments with healthcheck intervals, timeouts etc.
I also installed tcpdump to listen all GET requests on machine and it doesn’t catch anything when healthcheck in progress (but catched my requests from browser).

All I see from console is Error: failed to wait for health checks to pass: context deadline exceeded (and option --debug doesn’t provide more information)

My .toml is pretty simple:

app = "app-name"
primary_region = "lhr"

[http_service]
  internal_port = 3000
  force_https = true
  auto_stop_machines = true
  auto_start_machines = true
  min_machines_running = 1
  processes = ["app"]

  [[http_service.checks]]
    grace_period = "10s"
    interval = "30s"
    method = "GET"
    timeout = "1s"
    path = "/favicon.ico"

My docker runs through CMD HOSTNAME="0.0.0.0" PORT="3000" node server.js and I believe it listens port 3000 all local IPs

image
image818×371 7.26 KB

4

I wonder if readers need to clarify your question first. That doesn’t look like a build, in the Docker sense of things. It looks like you’re updating secrets, in which case I’d assume that it is bouncing your containers one-by-one, and resetting the env vars exposed inside each.

Can you try the same command locally? Also, what version of the flyctl binary are you running remotely?

(I agree the Go code crash is a bit worrying though, maybe you’ve stumbled into a bug).

5

Updated the post. Yes, the reason was not with Docker building. Deployment stops because health-check fails, but I can’t understand why

7

Hmm, OK. Since this is on the web, I think listening on IPv4 is the right approach. But if you’re completely stuck, then maybe adding a listener on IPv6 is worth a try. I do that for my apps that only listen on internal private networks.

8

OMG, I just changed path = "/favicon.ico" to another file and it works! I don’t know what happened with “favicon.ico” (it was available through browser and still is available), but it stolen many hours of my life…

image
image737×310 16 KB

2 Likes
9

Glad to hear that you got it working… If you SSH in again and try curl --dump-header h http://localhost:3000/favicon.ico > /dev/null (instead of just asking for the root) then you may see the discrepancy.

(The health-check system is super-strict about which exact HTTP response codes it will consider successful. Even a redirect counts as failure…)

10

I don’t see the difference with another svg file which is working. But anyway I will add some API endpoint with small response. It should be better

image
image659×122 5 KB

11

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.