Deployment suddenly fails: Failed due to unhealthy allocations, but it's listening on [::]:8080

Questions / Help
1

My app used to deploy fine. But now I get this confusing error when deploying:

--> You can detach the terminal anytime without stopping the deployment
==> Monitoring deployment

 1 desired, 1 placed, 0 healthy, 1 unhealthy [restarts: 2] [health checks: 1 total]
Failed Instances

Failure #1

Instance
ID              PROCESS VERSION REGION  DESIRED STATUS  HEALTH CHECKS   RESTARTS        CREATED
768b9041        app     18      fra     run     running 1 total         2               44s ago

Recent Events
TIMESTAMP               TYPE            MESSAGE
2022-11-21T13:27:44Z    Received        Task received by client
2022-11-21T13:28:00Z    Task Setup      Building Task Directory
2022-11-21T13:28:07Z    Started         Task started by client
2022-11-21T13:28:11Z    Terminated      Exit Code: 1
2022-11-21T13:28:11Z    Restarting      Task restarting in 1.184125877s
2022-11-21T13:28:18Z    Started         Task started by client
2022-11-21T13:28:22Z    Terminated      Exit Code: 1
2022-11-21T13:28:22Z    Restarting      Task restarting in 1.244000057s
2022-11-21T13:28:31Z    Started         Task started by client

2022-11-21T13:28:16Z   [info]Starting instance
2022-11-21T13:28:17Z   [info]Configuring virtual machine
2022-11-21T13:28:17Z   [info]Pulling container image
2022-11-21T13:28:18Z   [info]Unpacking image
2022-11-21T13:28:18Z   [info]Preparing kernel init
2022-11-21T13:28:18Z   [info]Setting up volume 'myapp_data'
2022-11-21T13:28:18Z   [info]Opening encrypted volume
2022-11-21T13:28:18Z   [info]Configuring firecracker
2022-11-21T13:28:18Z   [info]Starting virtual machine
2022-11-21T13:28:18Z   [info]Starting init (commit: 81d5330)...
2022-11-21T13:28:18Z   [info]Mounting /dev/vdc at /app/data w/ uid: 0, gid: 0 and chmod 0755
2022-11-21T13:28:18Z   [info]Preparing to run: `./backend` as root
2022-11-21T13:28:18Z   [info]2022/11/21 13:28:18 listening on [fdaa:0:d0d3:a7b:8a:2:c0a9:2]:22 (DNS: [fdaa::3]:53)
2022-11-21T13:28:21Z   [info]Starting clean up.
2022-11-21T13:28:21Z   [info]Umounting /dev/vdc from /app/data
2022-11-21T13:28:28Z   [info]Starting instance
2022-11-21T13:28:30Z   [info]Configuring virtual machine
2022-11-21T13:28:30Z   [info]Pulling container image
2022-11-21T13:28:30Z   [info]Unpacking image
2022-11-21T13:28:30Z   [info]Preparing kernel init
2022-11-21T13:28:30Z   [info]Setting up volume 'myapp_data'
2022-11-21T13:28:30Z   [info]Opening encrypted volume
2022-11-21T13:28:30Z   [info]Configuring firecracker
2022-11-21T13:28:31Z   [info]Starting virtual machine
2022-11-21T13:28:31Z   [info]Starting init (commit: 81d5330)...
2022-11-21T13:28:31Z   [info]Mounting /dev/vdc at /app/data w/ uid: 0, gid: 0 and chmod 0755
2022-11-21T13:28:31Z   [info]Preparing to run: `./backend` as root
2022-11-21T13:28:31Z   [info]2022/11/21 13:28:31 listening on [fdaa:0:d0d3:a7b:8a:2:c0a9:2]:22 (DNS: [fdaa::3]:53)
2022-11-21T13:28:34Z   [info]Starting clean up.
2022-11-21T13:28:34Z   [info]Umounting /dev/vdc from /app/data
--> v18 failed - Failed due to unhealthy allocations - not rolling back to stable job version 18 as current job has same specification and deploying as v19

--> Troubleshooting guide at https://fly.io/docs/getting-started/troubleshooting/
Error abort

If I do flyctl vm status 2399e8c2 I get:

Instance
  ID            = 2399e8c2
  Process       = app
  Version       = 20
  Region        = fra
  Desired       = run
  Status        = failed
  Health Checks = 1 total
  Restarts      = 2
  Created       = 20m13s ago

Events
TIMESTAMP               TYPE            MESSAGE
2022-11-21T13:46:18Z    Received        Task received by client
2022-11-21T13:46:18Z    Task Setup      Building Task Directory
2022-11-21T13:46:23Z    Started         Task started by client
2022-11-21T13:46:27Z    Terminated      Exit Code: 1
2022-11-21T13:46:27Z    Restarting      Task restarting in 1.025743723s
2022-11-21T13:46:35Z    Started         Task started by client
2022-11-21T13:46:39Z    Terminated      Exit Code: 1
2022-11-21T13:46:39Z    Restarting      Task restarting in 1.217813815s
2022-11-21T13:46:48Z    Started         Task started by client
2022-11-21T13:46:52Z    Terminated      Exit Code: 1
2022-11-21T13:46:52Z    Not Restarting  Exceeded allowed attempts 2 in interval 5m0s and mode is "fail"
2022-11-21T13:46:52Z    Alloc Unhealthy Unhealthy because of failed task
2022-11-21T13:46:53Z    Killing         Sent interrupt. Waiting 5s before force killing

Checks
ID                                      SERVICE         STATE   OUTPUT
3df2415693844068640885b45074b954        tcp-8080        warning

Recent Logs

What could be the reason? My app is definitely listening on [::]:8080.
Could it be that something is wrong with the volume mounting?
I’d appreciate any hint for solving this :slight_smile:

2

Does anyone know how to solve this? :slight_smile:

3

Exit code 1 means the process is crashing. Unfortunately, it doesn’t seem like it’s logging anything so it’s hard to tell why.

4

@kurt Hm, but it says Failed due to unhealthy allocations, could it mean my fly.toml file contains a wrong configuration, for mounting a volume or something?