Can't deploy my app in iad

I’m trying to deploy a new version of my app in the IAD DC, but it just gets stuck in pending, and I can’t roll back to an old version, so my app is offline:

$ fly status --all
App
  Name     = picoshare          
  Owner    = personal           
  Version  = 393                
  Status   = pending            
  Hostname = picoshare.fly.dev  

Deployment Status
  ID          = 66e65bb5-1303-706d-5540-ffd3aa6f73c2         
  Version     = v393                                         
  Status      = running                                      
  Description = Deployment is running                        
  Instances   = 1 desired, 1 placed, 0 healthy, 0 unhealthy  

Instances
ID              PROCESS VERSION REGION  DESIRED STATUS          HEALTH CHECKS           RESTARTS        CREATED              
555d2f57        app     393 ⇡   iad     stop    lost                                    0               16m14s ago          
6aaa03f1        app     392     iad     stop    lost                                    0               39m20s ago          
2133f5a1        app     391     iad     stop    lost                                    0               44m30s ago          
36230583        app     390     iad     stop    complete        1 total, 1 passing      0               2022-09-11T19:11:09Z

Possibly related:

Also seeing my apps in IAD struggle, and several of them exited unexpectedly.

We had a host go down for a few minutes. Errors or a status of “lost” are related to that, and scheduling was slow up for ~5 minutes while VMs moved to other hosts.

Thanks for the update.

One of my apps seems to now be in a stuck state. Nothing appears to prod it awake.

 2022-10-09T17:22:54.331 app[f9f59b7f] iad [info] Starting clean up.

2022-10-09T17:22:54.349 app[f9f59b7f] iad [info] Umounting /dev/vdc from /data/db

2022-10-09T17:22:54.352 app[f9f59b7f] iad [info] error umounting /data/db: EBUSY: Device or resource busy, retrying in a bit 

It’s msc-mongodb and has a volume so perhaps that broke when the host went down?

Looking into this!

1 Like

Thanks, @michael. I’m seeing my app back online.

App is back up now, and my services are fully restored.

Thanks!

Excellent! It looks like device mapper was hanging on several volumes on the busted host which was preventing a few apps from starting, including yours. Sorry about that. We’ve been debugging this issue for a little while, hopefully we’ll have a permanent fix soon.

2 Likes