Fly deployment to SIN stuck at pending, not creating VM

Questions / Help Build debugging
#1

Hi, had trouble with one of my apps corpus-x so I decided to delete it and see if I could resolve it by creating a new app lawdata. Unfortunately, the same problem occurs: the deployment status continues to remain pending.

 fly deploy --local-only

... Pushing image done
image: registry.fly.io/lawdata:deployment-01GPG01KJHH2MCR20K79RPGY9Z
image size: 261 MB
==> Creating release
--> release v2 created

--> You can detach the terminal anytime without stopping the deployment
==> Monitoring deployment
Logs: https://fly.io/apps/lawdata/monitoring

 1 desired, 1 placed, 0 healthy, 1 unhealthy
--> v0 failed - Failed due to unhealthy allocations - no stable job version to auto revert to and deploying as v1 

fly status
App
  Name     = lawdata          
  Owner    = personal         
  Version  = 1                
  Status   = running          
  Hostname = lawdata.fly.dev  
  Platform = nomad            

Deployment Status
  ID          = 4b604da9-1596-fc20-a14f-758bb4561c97         
  Version     = v1                                           
  Status      = running                                      
  Description = Deployment is running                        
  Instances   = 1 desired, 1 placed, 0 healthy, 0 unhealthy  

Instances
ID              PROCESS VERSION REGION  DESIRED STATUS  HEALTH CHECKS   RESTARTS        CREATED    
ae824a52        app     1 ⇡     sin     run     pending                 0               9m0s ago  
1736b69a        app     0       sin     stop    pending                 0               14m39s ago

fly logs -a lawdata
2023-01-11T08:46:27Z runner[1736b69a] sin [info]Starting instance
2023-01-11T08:46:29Z runner[1736b69a] sin [info]Configuring virtual machine
2023-01-11T08:46:29Z runner[1736b69a] sin [info]Pulling container image

fly ssh console             
# Error no instances found for lawdata

Please advise.

#2

It’s working now but it took about 30-45 minutes.

#3

To update, the same app is now having problems again:

ID              PROCESS VERSION REGION  DESIRED STATUS  HEALTH CHECKS   RESTARTS        CREATED    
34e284db        app     7 ⇡     sin     run     pending                 0               10m20s ago
30ebc8de        app     6       sin     stop    pending                 0               19m53s ago
3efda248        app     4       sin     stop    pending                 0               29m59s ago
16df93c8        app     4       sin     stop    pending                 0               37m7s ago 
993a7bd7        app     3       sin     stop    pending                 0               42m39s ago

With LOG_LEVEL=debug fly deploy --local-only, I get recurring logs of the following nature:

DEBUG <-- 200 https://api.fly.io/graphql (700.09ms)

{
  "data": {
    "app": {
      "deploymentStatus": {
        "id": "4bcfdb04-b302-d603-0bd8-a0f3b5e707f7",
        "inProgress": true,
        "status": "running",
        "successful": false,
        "description": "Deployment is running",
        "version": 7,
        "desiredCount": 1,
        "placedCount": 1,
        "healthyCount": 0,
        "unhealthyCount": 0,
        "allocations": []
      }
    }
  }
}

Finally resulting in the following error:

--> v7 failed - Failed due to unhealthy allocations - not rolling back to stable job version 7 as current job has same specification and deploying as v8

The only thing I changed recently was increasing the attached sqlite volume to 4GB from 3GB via fly vol extend. As an experiment, now trying increasing the memory to 2GB on shared-cpu from a previous 1GB.

1 Like
#4

We’re seeing some intermittent network-congestion issues on the Internet route between SIN and AWS us-east-1 (where we store container images), which we think is responsible for the deployment issues you’re seeing. We’re looking into routing workarounds to make this more reliable.

1 Like
#5

We deployed a change to our container registry to route around this network congestion, so deployments to sin should be more responsive for you moving forward. Let us know if you have any other issues.

1 Like