Stalled instances on new deploy leading to broken website

1

A new deploy has caused older instances to stick around in a “pending” state but they’re still running and serving responses leading to broken website experiences.

This has happened a few times now and I can sometimes force it by changing the regions but the reliability of deploys has become an issue:

PS C:\dev\mg> fly status
App
  Name     = mg1
  Owner    = personal
  Version  = 149
  Status   = running
  Hostname = mg1.fly.dev

Deployment Status
  ID          = 9cab9f39-104d-410f-8bc8-2f67033d1e8f
  Version     = v149
  Status      = successful
  Description = Deployment completed successfully
  Instances   = 1 desired, 1 placed, 1 healthy, 0 unhealthy

Instances
ID              PROCESS VERSION REGION  DESIRED STATUS  HEALTH CHECKS   RESTARTS        CREATED
f0c3dc3f        app     149 ⇡   sjc     run     running                 0               2m51s ago
b340eaee        app     147     dfw     stop    pending                 0               23m24s ago
ef901f1b        app     144     dfw     stop    running                 0               6h53m ago
2

changed regions and set scale count 0 but there’s still an instance stuck in running status:

PS C:\dev\mg> fly status
App
  Name     = mg1
  Owner    = personal
  Version  = 152
  Status   = dead
  Hostname = mg1.fly.dev

Instances
ID              PROCESS VERSION REGION  DESIRED STATUS  HEALTH CHECKS   RESTARTS        CREATED
ef901f1b        app     144     dfw     stop    running                 0               10h36m ago
3

I am running into this as well. I scale to 0 and then to 1 and it has two instances that are different versions. I even destroyed the app and restarted it and the same problem arrises.

image
image2722×946 439 KB