Deployments fail immediately due to unhealthy allocations

1

What it says in the title. Watching the logs shows no machine is spawned to even be unhealthy

2 Likes
2

new error output:

Recent Events
TIMESTAMP               TYPE            MESSAGE                                                                   
2023-02-25T23:51:28Z    Received        Task received by client                                                  
2023-02-25T23:51:28Z    Task Setup      Building Task Directory                                                  
2023-02-25T23:51:31Z    Driver Failure  rpc error: code = Unknown desc = could not fulfill runtime memory request
2023-02-25T23:51:31Z    Not Restarting  Error was unrecoverable                                                  
2023-02-25T23:51:31Z    Alloc Unhealthy Unhealthy because of failed task
3

Seems to only be an issue with nomad apps, i destroyed the app and recreated it with machines and all is working

1 Like
4

Also happening with my nomad apps.

Managed to deploy by, instead of using a ‘bluegreen’ strategy, choosing ‘immediate’:
fly deploy --remote-only --strategy immediate --config fly.toml

1 Like
5

Hi,

I am blocked on this since yesterday, not able to deploy my app for the same reason.

6

Adding --strategy immediate fixed it for me too, thank you.

7

--strategy immediate fixed it for me as well.

The issue is easily reproducible with Remix Run app:

  1. Install flyct and authenticate it
  2. Create a new Remix Run app npx create-remix@latest: choose “Just the basic”, “Fly.io”, “JavaScript”, agree to npm install
  3. cd into the app folder and initialize Fly.io flyctl launch: choose all defaults and agree to deploy
  4. Change literally any symbol in, say, app/routes/index.jsx of the app, try flyctl deploy and see failure
  5. Revert the change, try flyctl deploy and see success

I guess there is probably an even easier way to reproduce, it doesn’t seem to be tied to Remix, it’s just I’m sure this one is stable.

1 Like
8

I’m also unable to deploy since yesterday (last succesful deploy 15h28m ago), but I get no errors. It just fails on release with:

--> release v20 created

--> You can detach the terminal anytime without stopping the deployment
==> Release command detected: /app/bin/migrate

--> This release will not be available until the release command succeeds.
Error release command failed, deployment aborted

Adding --strategy immediate doesn’t change anything so it might be another cause. Any suggestions?

fly v0.0.470
Platform = nomad

9

Seeing the same on my postgres app, I can’t use the immediate strategy there, can’t restart it, can’t scale it… Truly disappointed with all the problems past weeks.

10

Waiting for the fix as I am blocked :frowning:

11

Similar issue here in ams. --strategy immediate works, but moving region also seems to work.

==> Creating release
--> release v79 created

--> You can detach the terminal anytime without stopping the deployment
==> Release command detected: php artisan migrate --force

--> This release will not be available until the release command succeeds.
Error release command failed, deployment aborted
12

Changing region (and creating a new db) worked for me.

13

Which region is working? I tried sin and it didn’t work.

14

ams is NOT working, but lhr and cdg are. lhr seems a bit slow.

15

Hi all, we’re working to provision more capacity in ams and are also adjusting resource allocation. Feel free to follow Fly.io Status - Capacity Issues in AMS region for updates.

2 Likes