Can't deploy my app in iad

1

I’m trying to deploy a new version of my app in the IAD DC, but it just gets stuck in pending, and I can’t roll back to an old version, so my app is offline:

$ fly status --all
App
  Name     = picoshare          
  Owner    = personal           
  Version  = 393                
  Status   = pending            
  Hostname = picoshare.fly.dev  

Deployment Status
  ID          = 66e65bb5-1303-706d-5540-ffd3aa6f73c2         
  Version     = v393                                         
  Status      = running                                      
  Description = Deployment is running                        
  Instances   = 1 desired, 1 placed, 0 healthy, 0 unhealthy  

Instances
ID              PROCESS VERSION REGION  DESIRED STATUS          HEALTH CHECKS           RESTARTS        CREATED              
555d2f57        app     393 ⇡   iad     stop    lost                                    0               16m14s ago          
6aaa03f1        app     392     iad     stop    lost                                    0               39m20s ago          
2133f5a1        app     391     iad     stop    lost                                    0               44m30s ago          
36230583        app     390     iad     stop    complete        1 total, 1 passing      0               2022-09-11T19:11:09Z

Possibly related:

2

Also seeing my apps in IAD struggle, and several of them exited unexpectedly.

3

We had a host go down for a few minutes. Errors or a status of “lost” are related to that, and scheduling was slow up for ~5 minutes while VMs moved to other hosts.

4

Thanks for the update.

One of my apps seems to now be in a stuck state. Nothing appears to prod it awake.

 2022-10-09T17:22:54.331 app[f9f59b7f] iad [info] Starting clean up.

2022-10-09T17:22:54.349 app[f9f59b7f] iad [info] Umounting /dev/vdc from /data/db

2022-10-09T17:22:54.352 app[f9f59b7f] iad [info] error umounting /data/db: EBUSY: Device or resource busy, retrying in a bit

It’s msc-mongodb and has a volume so perhaps that broke when the host went down?

5

Looking into this!

1 Like
6

Thanks, @michael. I’m seeing my app back online.

7

App is back up now, and my services are fully restored.

Thanks!

8

Excellent! It looks like device mapper was hanging on several volumes on the busted host which was preventing a few apps from starting, including yours. Sorry about that. We’ve been debugging this issue for a little while, hopefully we’ll have a permanent fix soon.

2 Likes