DFW - Deploying apps stuck in pending

1

Hi there,

We’ve had the GitHub action superfly/fly-actions@1.1 running as part of our CI pipeline for a while now. Today we’re seeing errors spinning up our DB:

pg create --region dfw --vm-size shared-cpu-1x --name pr146-092910e-beacon-db --volume-size 1 --organization beacon --initial-cluster-size 1

Creating postgres cluster pr146-092910e-beacon-db in organization beacon
Launching...⣾ WWWWWWWWWWWWWWW
Launching...⣽ WWWWWWWWWWWWWWW
Launching...⣻ WWWWWWWWWWWWWWW
Launching...⢿ WWWWWWWWWWWWWWW
Launching...⡿ WWWWWWWWWWWWWWW
Launching...⣟ WWWWWWWWWWWWWWW
Launching...⣯ WWWWWWWWWWWWWWW
Launching...⣷ WWWWWWWWWWWWWWW
Launching...⣾ WWWWWWWWWWWWWWW
Launching...⣽ WWWWWWWWWWWWWWW
Launching...⣻ WWWWWWWWWWWWWWW
Launching...⢿ WWWWWWWWWWWWWWW
Launching...⡿ WWWWWWWWWWWWWWW
Postgres cluster pr146-092910e-beacon-db created
  Username:    postgres
  Password:     *redacted*
  Hostname:    pr146-092910e-beacon-db.internal
  Proxy Port:  5432
  PG Port: 5433
Save your credentials in a secure place, you won't be able to see them again!

Monitoring Deployment

v0 is being deployed

This usually takes ~1 min but after multiple 15+ min runs our VMs seem to be stuck in pending:

fly status -a pr146-092910e-beacon-db
Update available 0.0.324 -> v0.0.325.
Run "fly version update" to upgrade.
App
  Name     = pr146-092910e-beacon-db          
  Owner    = beacon                           
  Version  = 0                                
  Status   = pending                          
  Hostname = pr146-092910e-beacon-db.fly.dev  

Deployment Status
  ID          = c81e1014-a85c-7d72-9749-ea7044e462fe         
  Version     = v0                                           
  Status      = running                                      
  Description = Deployment is running                        
  Instances   = 1 desired, 0 placed, 0 healthy, 0 unhealthy  

Instances
ID	PROCESS	VERSION	REGION	DESIRED	STATUS	HEALTH CHECKS	RESTARTS	CREATED

The other apps in this state are pr146-088bfff-beacon-db, and pr146-90858e8-beacon-db.

Do you have any guidance on how to proceed from here?

2

As a test I ran the same command manually. Miami seems fine:

fly pg create --region mia --vm-size shared-cpu-1x --name pr146-11112-beacon-db --volume-size 1 --organization beacon --initial-cluster-size 1
Update available 0.0.324 -> v0.0.325.
Run "fly version update" to upgrade.
Creating postgres cluster pr146-11112-beacon-db in organization beacon
Postgres cluster pr146-11112-beacon-db created
  Username:    postgres
  Password:   *redacted*
  Hostname:    pr146-11112-beacon-db.internal
  Proxy Port:  5432
  PG Port: 5433
Save your credentials in a secure place, you won't be able to see them again!

Monitoring Deployment

1 desired, 1 placed, 1 healthy, 0 unhealthy [health checks: 3 total, 3 passing]
--> v0 deployed successfully

Connect to postgres
Any app within the beacon organization can connect to postgres using the above credentials and the hostname "pr146-11112-beacon-db.internal."
For example: postgres://postgres:xxxx@pr146-11112-beacon-db.internal:5432

while dfw is still having problems:

fly pg create --region dfw --vm-size shared-cpu-1x --name pr146-11113-beacon-db --volume-size 1 --organization beacon --initial-cluster-size 1
Update available 0.0.324 -> v0.0.325.
Run "fly version update" to upgrade.
Creating postgres cluster pr146-11113-beacon-db in organization beacon
Postgres cluster pr146-11113-beacon-db created
  Username:    postgres
  Password:   *redacted*
  Hostname:    pr146-11113-beacon-db.internal
  Proxy Port:  5432
  PG Port: 5433
Save your credentials in a secure place, you won't be able to see them again!

Monitoring Deployment

v0 is being deployed
(...nothing else happens here)
3

For those looking, it seems like this is resolved as of this morning