jnb host instability since 16 Oct ~18:19 UTC: unreachable volume markers, SSH/exec timeouts to a serving machine

Questions / Help
1

Hello,

Since about 18:19 UTC on 16 Oct 2026 we have had host instability in jnb affecting two apps in our organisation. This is the state
we observed on 17 Oct between 13:30 and 14:15 UTC.

App eziopro-prod-db (3 machines):

  • 287154db111698: started, health checks 3/3, serving traffic normally. However, fly ssh console --machine 287154db111698 times
    out on port 22, and a command that execs on it fails with “failed to exec on VM 287154db: deadline_exceeded”. So SSH and exec
    into this machine fail while it serves normally.
  • 2871549f564078: started, checks 3/3.
  • 784e197c625118: its host became unreachable during the incident on 16 Oct (we do not know exactly when). The machine came
    back around 20:30 UTC and is now started, but with checks 1/3 (last updated 20:29:50 UTC, 16 Oct).
  • All three volumes of this app now show as attached, with no “could not be reached” marker. During the incident all three were
    flagged as on unreachable hosts.

App eziopro-prod:

  • The machine beginning 683d597c is started, checks 1/1.
  • The machines beginning 08016e7a and 2865901f563d48 are stopped (0/1).
  • Volumes vol_r68qo2oll88ed9j4 (machine beginning 08016e7a) and vol_r1jg187gzp3l81wr (2865901f563d48) are still marked
    “hosts could not be reached”, and no attached VM is shown for either.

Questions:

  1. Was there a host or network incident in jnb from about 18:19 UTC on 16 Oct, and is it resolved? What happened to the hosts of
    287154db111698 and 784e197c625118?
  2. Why do SSH (port 22) and exec to 287154db111698 time out while the machine passes its checks and serves traffic? Is the host’s
    management path degraded, and is the machine at risk?
  3. Will the hosts of volumes vol_r68qo2oll88ed9j4 and vol_r1jg187gzp3l81wr become reachable again? Is the data on them intact?
  4. If we destroy and replace 784e197c625118, or either stopped app machine, will the destroy be honoured on those hosts, and will
    the old machine stay down if its host has problems again?
  5. Please confirm that the daily snapshots of the eziopro-prod-db volumes are unaffected. The newest we see is about 14 hours old.

We are not asking for Postgres support, only about hosts, machines, volumes and SSH/exec access. Thank you.