Stuck in "deploying"

1

My app is stuck in deploying. It has been like that for 2 hours now. What can I do about this? I tried doing a new deployment but it didn’t help either.

App named lks in region ams (lagunen org).

From the logs it looks like the app was shutdown, but no new one was spawned.

2022-06-02T16:05:22Z app[a9a39ca8] ams [info]16:05:22.825 [info] CONNECTED TO Phoenix.LiveView.Socket in 47µs
2022-06-02T16:05:22Z app[a9a39ca8] ams [info]  Transport: :websocket
2022-06-02T16:05:22Z app[a9a39ca8] ams [info]  Serializer: Phoenix.Socket.V2.JSONSerializer
2022-06-02T16:05:22Z app[a9a39ca8] ams [info]  Parameters: %{"_csrf_token" => "STssezJ6RjEFVG0hAWM8Mh1aczJsFyQodhd2FB7nAmUN41PeBj5D-rJm", "_mounts" => "0", "_track_static" => %{"0" => "https://lks.fly.dev/assets/app-2aec0b0de850e4f480796558f11cbb39.css?vsn=d", "1" => "https://lks.fly.dev/assets/app-43225e7c9981857a04b1c349ea5eb174.js?vsn=d"}, "vsn" => "2.0.0"}
2022-06-02T16:10:30Z runner[a9a39ca8] ams [info]Shutting down virtual machine
2022-06-02T16:11:00Z app[a9a39ca8] ams [info]Sending signal SIGTERM to main child process w/ PID 523
2022-06-02T16:11:00Z app[a9a39ca8] ams [info]16:11:00.145 [notice] SIGTERM received - shutting down
2

hey there-- we had some issues with our orchestration cluster earlier which might be behind your recent troubles. You may want to look at fly status --all to see if there is any unusual behavior with your instances.

It does look like these logs are from a few hours ago; when was the last time you re-deployed? It’s possible that a re-deploy will succeed now. If not, you can gather more information about this by running LOG_LEVEL=debug fly deploy

3

Looks like this, which is not what I expect. There should be 1 instance running. Running a new deployment with the drug logs didn’t tell me much more.

==> Creating release
DEBUG --> POST https://api.fly.io/graphql {{"query":"mutation($input: DeployImageInput!) { deployImage(input: $input) { release { id version reason description deploymentStrategy user { id email name } evaluationId createdAt } releaseCommand { id command evaluationId } } }","variables":{"input":{"appId":"lks","image":"registry.fly.io/lks:deployment-1654195030","services":null,"definition":{"env":{"DATABASE_PATH":"/data/prod.db","PHX_HOST":"lks.fly.dev","PORT":"8080"},"experimental":{"allowed_public_ports":[],"auto_rollback":true},"kill_signal":"SIGTERM","kill_timeout":5,"mounts":{"destination":"/data","source":"data"},"processes":[],"services":[{"concurrency":{"hard_limit":25,"soft_limit":20,"type":"connections"},"http_checks":[],"internal_port":8080,"ports":[{"force_https":true,"handlers":["http"],"port":80},{"handlers":["tls","http"],"port":443}],"processes":["app"],"protocol":"tcp","script_checks":[],"tcp_checks":[{"grace_period":"1s","interval":"15s","restart_limit":0,"timeout":"2s"}]}]},"strategy":null}}}
}
DEBUG <-- 200 https://api.fly.io/graphql (1.04s) {"data":{"deployImage":{"release":{"id":"YgoYklRLL8Xo3fz7K9Jle02n","version":20,"reason":"","description":"Release","deploymentStrategy":"ROLLING","user":{"id":"KXJVBG8Zpm8aoT9G3","email":"hannes.nevalainen@me.com","name":"Hannes Nevalainen"},"evaluationId":"eb5ee755-7557-52c6-1976-4e3b6eea13d1","createdAt":"2022-06-02T18:38:05Z"},"releaseCommand":null}}}
--> release v20 created

--> You can detach the terminal anytime without stopping the deployment
==> Monitoring deployment
DEBUG --> POST https://api.fly.io/graphql {{"query":"query ($appName: String!, $deploymentId: ID!, $evaluationId: String!) { app(name: $appName) { deploymentStatus(id: $deploymentId, evaluationId: $evaluationId) { id inProgress status successful description version desiredCount placedCount healthyCount unhealthyCount allocations { id idShort status region desiredStatus version healthy failed canary restarts checks { status serviceName } } } } }","variables":{"appName":"lks","deploymentId":"","evaluationId":"eb5ee755-7557-52c6-1976-4e3b6eea13d1"}}
}
DEBUG <-- 200 https://api.fly.io/graphql (306.72ms) {"data":{"app":{"deploymentStatus":null}}}
DEBUG --> POST https://api.fly.io/graphql {{"query":"query ($appName: String!, $deploymentId: ID!, $evaluationId: String!) { app(name: $appName) { deploymentStatus(id: $deploymentId, evaluationId: $evaluationId) { id inProgress status successful description version desiredCount placedCount healthyCount unhealthyCount allocations { id idShort status region desiredStatus version healthy failed canary restarts checks { status serviceName } } } } }","variables":{"appName":"lks","deploymentId":"","evaluationId":"eb5ee755-7557-52c6-1976-4e3b6eea13d1"}}
}
DEBUG <-- 200 https://api.fly.io/graphql (204.47ms) {"data":{"app":{"deploymentStatus":{"id":"d7140186-9de8-0625-01ac-2a373a4386a4","inProgress":true,"status":"running","successful":false,"description":"Deployment is running","version":20,"desiredCount":1,"placedCount":0,"healthyCount":0,"unhealthyCount":0,"allocations":[]}}}}

v20 is being deployed
DEBUG --> POST https://api.fly.io/graphql {{"query":"query ($appName: String!, $deploymentId: ID!, $evaluationId: String!) { app(name: $appName) { deploymentStatus(id: $deploymentId, evaluationId: $evaluationId) { id inProgress status successful description version desiredCount placedCount healthyCount unhealthyCount allocations { id idShort status region desiredStatus version healthy failed canary restarts checks { status serviceName } } } } }","variables":{"appName":"lks","deploymentId":"d7140186-9de8-0625-01ac-2a373a4386a4","evaluationId":"eb5ee755-7557-52c6-1976-4e3b6eea13d1"}}
}
DEBUG <-- 200 https://api.fly.io/graphql (169.61ms) {"data":{"app":{"deploymentStatus":{"id":"d7140186-9de8-0625-01ac-2a373a4386a4","inProgress":true,"status":"running","successful":false,"description":"Deployment is running","version":20,"desiredCount":1,"placedCount":0,"healthyCount":0,"unhealthyCount":0,"allocations":[]}}}}
DEBUG --> POST https://api.fly.io/graphql {{"query":"query ($appName: String!, $deploymentId: ID!, $evaluationId: String!) { app(name: $appName) { deploymentStatus(id: $deploymentId, evaluationId: $evaluationId) { id inProgress status successful description version desiredCount placedCount healthyCount unhealthyCount allocations { id idShort status region desiredStatus version healthy failed canary restarts checks { status serviceName } } } } }","variables":{"appName":"lks","deploymentId":"d7140186-9de8-0625-01ac-2a373a4386a4","evaluationId":"eb5ee755-7557-52c6-1976-4e3b6eea13d1"}}
}
DEBUG <-- 200 https://api.fly.io/graphql (166.35ms) {"data":{"app":{"deploymentStatus":{"id":"d7140186-9de8-0625-01ac-2a373a4386a4","inProgress":true,"status":"running","successful":false,"description":"Deployment is running","version":20,"desiredCount":1,"placedCount":0,"healthyCount":0,"unhealthyCount":0,"allocations":[]}}}}
DEBUG --> POST https://api.fly.io/graphql {{"query":"query ($appName: String!, $deploymentId: ID!, $evaluationId: String!) { app(name: $appName) { deploymentStatus(id: $deploymentId, evaluationId: $evaluationId) { id inProgress status successful description version desiredCount placedCount healthyCount unhealthyCount allocations { id idShort status region desiredStatus version healthy failed canary restarts checks { status serviceName } } } } }","variables":{"appName":"lks","deploymentId":"d7140186-9de8-0625-01ac-2a373a4386a4","evaluationId":"eb5ee755-7557-52c6-1976-4e3b6eea13d1"}}
}
DEBUG <-- 200 https://api.fly.io/graphql (275.37ms) {"data":{"app":{"deploymentStatus":{"id":"d7140186-9de8-0625-01ac-2a373a4386a4","inProgress":true,"status":"running","successful":false,"description":"Deployment is running","version":20,"desiredCount":1,"placedCount":0,"healthyCount":0,"unhealthyCount":0,"allocations":[]}}}}
DEBUG --> POST https://api.fly.io/graphql {{"query":"query ($appName: String!, $deploymentId: ID!, $evaluationId: String!) { app(name: $appName) { deploymentStatus(id: $deploymentId, evaluationId: $evaluationId) { id inProgress status successful description version desiredCount placedCount healthyCount unhealthyCount allocations { id idShort status region desiredStatus version healthy failed canary restarts checks { status serviceName } } } } }","variables":{"appName":"lks","deploymentId":"d7140186-9de8-0625-01ac-2a373a4386a4","evaluationId":"eb5ee755-7557-52c6-1976-4e3b6eea13d1"}}
}
DEBUG <-- 200 https://api.fly.io/graphql (183.23ms) {"data":{"app":{"deploymentStatus":{"id":"d7140186-9de8-0625-01ac-2a373a4386a4","inProgress":true,"status":"running","successful":false,"description":"Deployment is running","version":20,"desiredCount":1,"placedCount":0,"healthyCount":0,"unhealthyCount":0,"allocations":[]}}}}
DEBUG --> POST https://api.fly.io/graphql {{"query":"query ($appName: String!, $deploymentId: ID!, $evaluationId: String!) { app(name: $appName) { deploymentStatus(id: $deploymentId, evaluationId: $evaluationId) { id inProgress status successful description version desiredCount placedCount healthyCount unhealthyCount allocations { id idShort status region desiredStatus version healthy failed canary restarts checks { status serviceName } } } } }","variables":{"appName":"lks","deploymentId":"d7140186-9de8-0625-01ac-2a373a4386a4","evaluationId":"eb5ee755-7557-52c6-1976-4e3b6eea13d1"}}

Then it just continues with the “monitoring”…

$ fly status --all
App
  Name     = lks          
  Owner    = lagunen      
  Version  = 20           
  Status   = pending      
  Hostname = lks.fly.dev  

Deployment Status
  ID          = d7140186-9de8-0625-01ac-2a373a4386a4         
  Version     = v20                                          
  Status      = running                                      
  Description = Deployment is running                        
  Instances   = 1 desired, 0 placed, 0 healthy, 0 unhealthy  

Instances
ID              PROCESS VERSION REGION  DESIRED STATUS          HEALTH CHECKS           RESTARTS        CREATED              
a9a39ca8        app     18      ams     stop    complete        1 total, 1 passing      0               3h38m ago           
a2b8a6d8        app     17      ams     stop    complete        1 total, 1 passing      0               2022-05-24T19:30:30Z
4

Hmm, now it did all the deployments in a quicks succession… now it is back online and the latest version deployed.

Did you guys poke something or did it magically fix itself?

5

yup! we were able to fix an issue with orchestration that was specific to your host just now. Thanks for letting us know!

as an aside, I might be misunderstanding your last post, but fly status --all will also show completed instances from the last week in addition to running ones

6

It says the version should be 20 but there are no instances running. I expected to see a row with version 20 and possibly 19 as well.

Great you were able to fix the problem and thanks for the quick action :slight_smile:

7

I’m still having deployment issues as well. Not sure if they’re related to the previous issue you mentioned. One of my deploys started but then stopped during dependency installation and is completely stuck now. Subsequent deploys get this far then hang:

==> Verifying app config
--> Verified app config
==> Building image
Remote builder fly-builder-broken-dust-8053 ready
==> Building image with Buildpacks
--> docker host: 20.10.12 linux x86_64

My remote builder seems to be stuck with a super high load average as well:

/ # uptime
 19:02:24 up 17 min,  0 users,  load average: 9.55, 8.20, 5.30

Any ideas?

8

I’m not yet sure if it’s related but you’ll probably want to rule out the impact of the builder app. In this case, I’d recommend destroying the builder with fly destroy fly-builder-broken-dust 8053 and re-deploying. In general, you can also test deploys by seeing if a deploy with --local-only succeeds.

1 Like
9

Thanks, that fixed it. I wasn’t sure what would happen if I destroyed my remote builder haha. Good to know.

1 Like