Health checks consistently timing out

1

Hello, I’ve deployed my ruby on rails app and am intermittently getting the following error:

 ✔ release_command 3d8d0e6be0e458 completed successfully
-------
-------
Updating existing machines in 'music-assistant-dev' with rolling strategy

WARNING The app is not listening on the expected address and will not be reachable by fly-proxy.
You can fix this by configuring your app to listen on the following addresses:
  - 0.0.0.0:3000
Found these processes inside the machine with open listening sockets:
  PROCESS        | ADDRESSES                             
-----------------*---------------------------------------
  /.fly/hallpass | [fdaa:9:bc92:a7b:2ae:fae0:9c94:2]:22  




-------
 ✔ Cleared lease for 90801e09c9d238
-------
Error: failed to update machine 90801e09c9d238: Unrecoverable error: timeout reached waiting for health checks to pass for machine 90801e09c9d238: failed to get VM 90801e09c9d238: Get "https://api.machines.dev/v1/apps/music-assistant-dev/machines/90801e09c9d238": net/http: request canceled

I’ve looked at other threads and this usually indicated a mismatched port, however when running fly logs I can see that my server is running appropriately:

2024-08-06T19:48:59Z app[3d8d0e6be0e458] sjc [info]W, [2024-08-06T19:48:59.106928 #319]  WARN -- : You are running SQLite in production, this is generally not recommended. You can disable this warning by setting "config.active_record.sqlite3_production_warning=false".
2024-08-06T19:48:59Z app[3d8d0e6be0e458] sjc [info] INFO Main child exited normally with code: 0
2024-08-06T19:48:59Z app[3d8d0e6be0e458] sjc [info] INFO Starting clean up.
2024-08-06T19:48:59Z app[3d8d0e6be0e458] sjc [info] WARN could not unmount /rootfs: EINVAL: Invalid argument
2024-08-06T19:48:59Z app[3d8d0e6be0e458] sjc [info][    3.650530] reboot: Restarting system
2024-08-06T19:49:00Z runner[3d8d0e6be0e458] sjc [info]machine restart policy set to 'no', not restarting
2024-08-06T19:49:01Z runner[90801e09c9d238] sjc [info]Pulling container image registry.fly.io/music-assistant-dev:deployment-01J4MKS6QZBE409YQBBRDWNYZA
2024-08-06T19:49:01Z runner[90801e09c9d238] sjc [info]Successfully prepared image registry.fly.io/music-assistant-dev:deployment-01J4MKS6QZBE409YQBBRDWNYZA (502.954355ms)
2024-08-06T19:49:01Z runner[90801e09c9d238] sjc [info]Setting up volume 'litestack_data'
2024-08-06T19:49:01Z runner[90801e09c9d238] sjc [info]Opening encrypted volume
2024-08-06T19:49:02Z app[90801e09c9d238] sjc [info] INFO Sending signal SIGINT to main child process w/ PID 327
2024-08-06T19:49:02Z app[90801e09c9d238] sjc [info]- Gracefully stopping, waiting for requests to finish
2024-08-06T19:49:02Z app[90801e09c9d238] sjc [info]=== puma shutdown: 2024-08-06 19:49:02 +0000 ===
2024-08-06T19:49:02Z app[90801e09c9d238] sjc [info]- Goodbye!
2024-08-06T19:49:02Z app[90801e09c9d238] sjc [info]Exiting
2024-08-06T19:49:03Z app[90801e09c9d238] sjc [info] INFO Main child exited normally with code: 0
2024-08-06T19:49:03Z app[90801e09c9d238] sjc [info] INFO Starting clean up.
2024-08-06T19:49:03Z app[90801e09c9d238] sjc [info] INFO Umounting /dev/vdc from /data
2024-08-06T19:49:03Z app[90801e09c9d238] sjc [info] WARN could not unmount /rootfs: EINVAL: Invalid argument
2024-08-06T19:49:03Z app[90801e09c9d238] sjc [info][  330.065822] reboot: Restarting system
2024-08-06T19:49:06Z app[90801e09c9d238] sjc [info]2024-08-06T19:49:06.733714464 [01J4MKSRBS4PFM46SB1RG5F9EG:main] Running Firecracker v1.7.0
2024-08-06T19:49:07Z app[90801e09c9d238] sjc [info][    0.278175] PCI: Fatal: No config space access function found
2024-08-06T19:49:07Z app[90801e09c9d238] sjc [info] INFO Starting init (commit: 20f21dc5f)...
2024-08-06T19:49:07Z app[90801e09c9d238] sjc [info] INFO Mounting /dev/vdc at /data w/ uid: 1000, gid: 1000 and chmod 0755
2024-08-06T19:49:07Z app[90801e09c9d238] sjc [info] INFO Resized /data to 1056964608 bytes
2024-08-06T19:49:07Z app[90801e09c9d238] sjc [info] INFO starting statics vsock server
2024-08-06T19:49:07Z app[90801e09c9d238] sjc [info] INFO Preparing to run: `/rails/bin/docker-entrypoint ./bin/rails server` as rails
2024-08-06T19:49:07Z app[90801e09c9d238] sjc [info] INFO [fly api proxy] listening at /.fly/api
2024-08-06T19:49:07Z app[90801e09c9d238] sjc [info]2024/08/06 19:49:07 INFO SSH listening listen_address=[fdaa:9:bc92:a7b:2ae:fae0:9c94:2]:22 dns_server=[fdaa::3]:53
2024-08-06T19:49:07Z runner[90801e09c9d238] sjc [info]Machine created and started in 6.009s
2024-08-06T19:49:09Z app[90801e09c9d238] sjc [info]=> Booting Puma
2024-08-06T19:49:09Z app[90801e09c9d238] sjc [info]=> Rails 7.1.3.4 application starting in production
2024-08-06T19:49:09Z app[90801e09c9d238] sjc [info]=> Run `bin/rails server --help` for more startup options
2024-08-06T19:49:10Z app[90801e09c9d238] sjc [info]W, [2024-08-06T19:49:09.595496 #327]  WARN -- : You are running SQLite in production, this is generally not recommended. You can disable this warning by setting "config.active_record.sqlite3_production_warning=false".
2024-08-06T19:49:10Z app[90801e09c9d238] sjc [info]Puma starting in single mode...
2024-08-06T19:49:10Z app[90801e09c9d238] sjc [info]* Puma version: 6.4.2 (ruby 3.3.0-p0) ("The Eagle of Durango")
2024-08-06T19:49:10Z app[90801e09c9d238] sjc [info]*  Min threads: 5
2024-08-06T19:49:10Z app[90801e09c9d238] sjc [info]*  Max threads: 5
2024-08-06T19:49:10Z app[90801e09c9d238] sjc [info]*  Environment: production
2024-08-06T19:49:10Z app[90801e09c9d238] sjc [info]*          PID: 327
2024-08-06T19:49:10Z app[90801e09c9d238] sjc [info]* Listening on http://0.0.0.0:3000
2024-08-06T19:49:10Z app[90801e09c9d238] sjc [info]Use Ctrl-C to stop

And my port is set to 3000 in my fly toml:

# fly.toml app configuration file generated for music-assistant-dev-bold-surf-3433 on 2024-08-05T09:17:48-06:00
#
# See https://fly.io/docs/reference/configuration/ for information about how to use this file.
#

app = 'music-assistant-dev'
primary_region = 'sjc'
console_command = '/rails/bin/rails console'

[build]

[deploy]
  release_command = './bin/rails db:prepare'

[http_service]
  internal_port = 3000
  force_https = true
  auto_stop_machines = 'stop'
  auto_start_machines = true
  min_machines_running = 0
  processes = ['app']

[[vm]]
  memory = '1gb'
  cpu_kind = 'shared'
  cpus = 1

[[statics]]
  guest_path = '/rails/public'
  url_prefix = '/'

[mounts]
  source="litestack_data"
  destination="/data"

[[services.http_checks]]
  interval = "10s"
  grace_period = "5s"
  method = "get"
  path = "/"
  protocol = "http"
  timeout = "2s"
  tls_skip_verify = false
[services.http_checks.headers]
  Host = "chordal.app"
2

Is that your site? I don’t think it makes sense to have the internal fly service check for this host header.

3

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.