Deployment issue for one service

Hey Fly friends,

sorry to bother on Xmas eve. I’m rolling out a deployment of a critical fix nikola-sharder and the deployment is failing. The fix is trivial and just points to Nikola account creation to a healthy shard and it doesn’t appear that there’s an issue in my code.

I tried another service just now (nikola-healthchecker) and that other service did deploy. I’m a bit mystified as there were not material changes. Any ideas here? I tried doing a bunch of other things like selecting a different instance size and scaling up/down.

Thank you

–> Done Pushing Image
==> Optimizing Image
–> Done Optimizing Image
==> Creating Release
Release v112 created
Deploying to : nikola-sharder.fly.dev

Monitoring Deployment
You can detach the terminal anytime without stopping the deployment

2 desired, 1 placed, 0 healthy, 1 unhealthy
v112 failed - Failed due to unhealthy allocations - rolling back to job version 111
Failed Instances

==> Failure #1

Instance
ID = 4ff8b84a
Version = 112
Region = ewr
Desired = stop
Status = failed
Health Checks =
Restarts = 0
Created = 1s ago

Recent Events
TIMESTAMP TYPE MESSAGE
2020-12-25T00:31:20Z Received Task received by client
2020-12-25T00:31:20Z Task Setup Building Task Directory
2020-12-25T00:31:20Z Driver Failure rpc error: code = Unknown desc = could not create alloc logger: dial unix /var/run/vector/app_logs.sock: connect: resource temporarily unavailable
2020-12-25T00:31:20Z Not Restarting Error was unrecoverable
2020-12-25T00:31:20Z Killing Sent interrupt. Waiting 5s before force killing
2020-12-25T00:31:20Z Alloc Unhealthy Unhealthy because of failed task
2020-12-25T00:31:21Z Killing Sent interrupt. Waiting 5s before force killing

Recent Logs

2 desired, 2 placed, 2 healthy, 0 unhealthy
–> v113 deployed successfully
***v112 failed - Failed due to unhealthy allocations - rolling back to job version 111 and deploying as v113

david@Stargate TeslaMonitor %

I was able to work around this by changing my regions.

Dang, we’ll look into that. Seems like you were hitting a host that couldn’t build your VM for some reason.

Thank you. Not critical, as my workaround has held!