sjc host e7d1 unreachable (PU03) for 9+ hours; host_status stayed "ok", no host incident

Questions / Help
1

Hi Fly team, reporting a host problem in case it’s still affecting others.

2026-10-23, sjc, host e7d1 (app nudge-surf-api, machine 815745c90624d8, now destroyed)

  • 22:56 UTC: the machine stopped serving; its metrics stopped at the same moment.
  • The reboots we have console logs for all failed DNS inside the VM (getaddrinfo EAI_AGAIN for our database’s hostname). It hit the 10-restart limit at 23:05:55 UTC and stayed stopped.
  • From 23:07 UTC until we moved it at 08:27 UTC the next day, the proxy logged [PU03] unreachable worker host. the host may be unhealthy. this is a Fly issue. on every request.
  • The whole time, the Machines API said host_status: ok, fly incidents hosts list was empty, and the status page showed nothing.
  • Cloning the machine onto another host (03d5) fixed it straight away.

Same host, 2026-10-16, during the platform-wide reboots: after the reboot the VM came back with no network (no eth0 traffic), failed 10 boots and sat stopped until the proxy started it at 19:14 UTC, about 55 minutes later.

Questions:

  1. Is e7d1 still unhealthy, and can it be taken out of rotation?
  2. Why did host_status stay “ok”, with no host issue posted, while the proxy reported PU03 for 9 hours? We found out from our own alerts.
  3. Is there a way to be notified when a host problem affects one of our machines?

Thanks!