Health checks and routing during recent outage

Questions / Help
1

Hello. During the recent platform outage, two of my three EWR machines were restarting endlessly, so I spun up another machine in IAD. That restored service. One EWR machine remained healthy throughout, and the new IAD machine worked for roughly 1–2 days.

Today, I received an email from Fly noting that IAD was experiencing restarts. My IAD machine also stopped working, so I powered it down. The remaining EWR machine stayed healthy.

Could the Fly.io team help me understand why traffic continued to reach the affected machine instead of being routed to the remaining healthy machine? I received both an internal application downtime alert and Fly’s notification about the regional issue, but manual intervention was still needed.

For context, the affected machine could not successfully query our externally hosted database, so any application functionality requiring database access was broken. I understand that Fly wouldn’t necessarily detect that without an application health check covering database connectivity.

Is there a health-check configuration that would automatically remove a machine from traffic in this situation and route requests to the healthy machine in another region? If the machines need to be explicitly identified as interchangeable, is there a configuration option for that?

I’d appreciate guidance on whether this is something I should configure or a limitation of how the platform handles these incidents.

Thank you.

4

Hi… Fly.io’s support queue is still unusually long, so as just a couple of pointers in the meantime…

Any sustained failure of a health check defined under [[services.http_checks]] or [http_service.checks] should remove the Machine from routing, at least for anything that comes in through the Fly Proxy.

https://docs.fly.io/reference/health-checks

If it bounces back and forth between healthy and unhealthy, then you can still get some traffic, though.

Posting your full fly.toml is often wise when you have questions about this kind of thing, since people like myself who happen to be reading the community forum here may notice a glitch, etc. (flyctl itself doesn’t always complain about syntactic and logical errors.)

You can’t really influence this, unfortunately. (Not that I know of, anyway.) Machines in different regions aren’t considered fully interchangeable, although it should route to another region if a given region is reduced to no healthy Machines left…

Cross-region routing only happens when all Machines in the local region are unhealthy or at their hard_limit.

https://docs.fly.io/reference/load-balancing

https://community.fly.io/t/healthcheck-based-routing/12997

5

Thanks for the reply, this is very helpful. Here’s the toml file in case it is useful.

# fly.toml app configuration file generated for redacted on 2024-11-15T20:49:54+07:00
#
# See https://fly.io/docs/reference/configuration/ for information about how to use this file.
#

app = 'redacted'
primary_region = 'ewr'
console_command = '/code/manage.py shell'

[env]
  CURRENT_ENVIRONMENT = 'production'
  PORT = 'redacted'

[http_service]
  internal_port = redacted
  force_https = true
  auto_stop_machines = 'off'
  auto_start_machines = false
  min_machines_running = 1
  processes = ['app']

[[vm]]
  memory = '2gb'
  cpu_kind = 'shared'
  cpus = 1

[[statics]]
  guest_path = '/code/static'
  url_prefix = '/static/'

Looking at this myself right now, I see auto_stop_machines OFF - per the docs, it appears to be more about not sitting idle during low traffic periods, so I think not applicable for outages / errors… Autostop/autostart Machines - Fly.io

6

Basically, you’d want to append something like the following to that stanza:

  [[http_service.checks]]
    method = "GET"
    path = "/health/"
    interval = "5s"

The /health/ endpoint that you implement should verify that the database is reachable (like you were saying) along with any other resources that it definitely needs to serve traffic. If some of those prerequisites are missing, then report failure (i.e., any HTTP response code other than 200).

(If this is a Django server, then there may actually already be such an implementation squirreled away somewhere. I don’t know that ecosystem very well, myself.)

The interval is how often the Fly.io infrastructure will probe that URL.

Note that Fly.io has a confusing tradition wherein “Last Updated” means the time when the status last changed, not when it was last probed.

There are more knobs here in the docs:

https://docs.fly.io/reference/configuration#http_service-checks

7

Marked as the solution - This is just what I need, paired with your first message. Appreciate you.