deployment is hanging and not completing successfully

Deployment reliability has been the achilles heel at Fly.io, it’s getting to the point where I may move to another host. These past few days, I’ve not been able to deploy successfully with any kind of consistency at different data centers, the process always seems to hang at various points of the start up or the clean up process and the resulting app is in pending state.

1 Like

I don’t think deploy issues have been problematic for a while. I did notice more complaints this weekend for sure.

I noticed the app exposing our API was OOMing and I just scaled its memory. Hopefully that helps.

We’ll look into more detail tomorrow.

Can you give us more information about the issues you’ve been seeing?

@jerome Thank you for the quick response. The specific issue I’m experiencing is that deployments are lagging quite a bit. If you start a deploy and monitor the log, sometimes it can be more than half an hour before the action kicks into place (server starts, etc.).

I’ve definitely experienced other issues with deployment before, such as not being able to retrieve Docker images.

For what it’s worth, the latest deploys succeeded.

Deploys seem to be working fine on the backend, but the sync process that gets them into the DB we use to show y’all status is lagging. We’re working on it.

2 Likes

Is it still the issue? I’ve tried to deploy this morning and it just hanged (but deployed though).

--> You can detach the terminal anytime without stopping the deployment
==> Monitoring deployment
Logs: https://fly.io/apps/app/monitoring
Error 1 error occurred:
	* No deployment available to monitor
1 Like

Getting the same on my apps. All looks OK, but the GitHub job fails :sweat_smile:

1 Like

Happening here too:

➜ fly --config fly.prod.toml status
App
  Name     = prereview          
  Owner    = prereview          
  Version  = 1375               
  Status   = running            
  Hostname = prereview.fly.dev  
  Platform = nomad              

Instances
ID              PROCESS VERSION REGION  DESIRED STATUS  HEALTH CHECKS           RESTARTS        CREATED    
f8adbb16        app     1373    dc2     run     running 1 total, 1 passing      0               40m47s ago      
03e89815        app     1373    fra     run     running 1 total, 1 passing      0               40m47s ago      
543e29d9        app     1373    fra     run     running 1 total, 1 passing      0               49m34s ago      
e8da560b        app     1373    iad     run     running 1 total, 1 passing      0               49m34s ago      

➜ fly --config fly.prod.toml monitor
Error app prereview is not currently deploying a release. The build and release command must succeed before a release is deployed. The latest release version is 1375

➜ fly --config fly.prod.toml history
TYPE    STATUS          DESCRIPTION                     USER                            DATE                 
Release failed          Deploy image                    XXX                             22m43s ago              
Release failed          Deploy image                    XXX                             36m11s ago              
Release in_progress     Deploy image                    XXX                             54m42s ago  
Release succeeded       Deploy image                    XXX                             1h6m ago

The 1373 release seems to have deployed but has not been flagged as completed.

The two subsequent releases (1374 and 1375) failed with:

--> You can detach the terminal anytime without stopping the deployment
==> Monitoring deployment
Error 1 error occurred:
	* No deployment available to monitor

Have the same issue, can’t deploy a new version for more than 10 minutes:

No deployment available to monitor.

Good morning, thank you for all your work, not sure if this is a related issue, but i am getting the following errors:

deployment v1
Error failed creating volume: Timeout on CreateVolume.app

deployment v2
Error failed destroying volume: Timeout on DeleteVolume.app

Guessing this is

Any workaround ? I’m not able to ship my product updates today.

I referred to the documentation here to migrate the application to Apps V2 and have been able to deploy it successfully, so if anyone needs to deploy it urgently, this may be a solution.

2 Likes

This incident is marked as resolved but I still get “Failed due to unhealthy allocations” when trying to deploy (version known to be healthy). Already tried multiple times. Is anyone else getting the same?