Swap failing to allocate on deployment

I am building and pushing and image successfully in detached mode:

deployment-01GWG75AJG6QR9CBG2CMQ1SXQW: digest: sha256:a27bca86d3ecb16de91548aff3b09a6af9d29203131820fd01cbba1bec1ab8cf size: 2004
--> Pushing image done
image: registry.fly.io/<my app>:deployment-01GWG75AJG6QR9CBG2CMQ1SXQW
image size: 808 MB
==> Creating release
--> release v143 created

But that release never shows up in Activity (I can see up to v142) and I can see the code is actually not released to production by ssh-ing into the machine.

Deploys have been very unstable with Fly and itā€™s making me reconsider if I should use something else. Itā€™s a very critical feature for a developerā€™s workflow.

Edit: I see this after the release is created:

 2023-03-27T01:56:32.282 app[56b14333] lax [info] fallocate: fallocate failed: Text file busy

2023-03-27T01:56:33.240 app[56b14333] lax [info] Starting clean up. 

This comes from the entrypoint trying to assign swap I believe. It was generated with the dockerfile generator. So it just seems it dies and does not show the failure anywhere else.

Iā€™m not sure that has anything to do with fly. I did deploys on stage (nomad), prod (nomad), and then deployed roughly 100 times while testing out litefs on v2 apps to see how a migration would look.

Is this something new that youā€™re trying on an image?

Regarding activity, it doesnā€™t seem to show up for v2 apps consistently, which is definitely a fly thing.

Yeah I changed the dockerfile to allow swapping because the machine wasnā€™t able to run a Rails console. Used GitHub - rubys/dockerfile-rails: Provides a Rails generator to produce Dockerfiles and related files. from the docs.

This is the entrypoint generated for the dockerfile:

#!/bin/bash -e

if [ $UID -eq 0 ]; then
  # allocate swap space
  fallocate -l 512M /swapfile
  chmod 0600 /swapfile
  mkswap /swapfile
  echo 10 > /proc/sys/vm/swappiness
  swapon /swapfile
  echo 1 > /proc/sys/vm/overcommit_memory
  exec su rails $0 $@
fi

# If running the rails server then create or migrate existing database
if [ "${*}" == "./bin/rails server" ]; then
  ./bin/rails db:prepare
fi

exec "${@}"

That is a bit baffling as youā€™re in the if condition indicating this is running as root. The only explanation I can think of is that /swapfile already exists and is mounted?

As far as I read that should only happen at deploy time, which should launch a new container, but I can only speculate on how the flow is implemented.

Either way, it shouldnā€™t die on that statement so should probably make it

fallocate -l 512M /swapfile || echo "swap already assigned"

but Iā€™m not 100% sure that the swap is there. Iā€™ll try.

This version crashes on Rails:

#!/bin/bash -e

if [ $UID -eq 0 ]; then
  # allocate swap space
  if fallocate -l 512M /swapfile; then
    chmod 0600 /swapfile
    mkswap /swapfile
    echo 10 >/proc/sys/vm/swappiness
    swapon /swapfile
    echo 1 >/proc/sys/vm/overcommit_memory
  fi
  exec su rails $0 $@
fi

# If running the rails server then create or migrate existing database
if [ "${*}" == "./bin/rails server" ]; then
  ./bin/rails db:prepare
fi

exec "${@}"

Iā€™m tired, Iā€™ll try tomorrow.

1 Like

Donā€™t know if itā€™s related, but Iā€™m getting all deploys hanging on the final ā€œPlacingā€¦ā€ step. Seems to be happened to both my apps, but they both use the same Dockerfile so may well be something silly Iā€™m doing too :slight_smile:

same, getting proxy errors in logs

2023-03-27T08:53:35.720 proxy[f64442fc] sin [warn] Failed to proxy HTTP request (error: no known healthy instances found for route tcp/443. (hint: is your app shutdown? is there an ongoing deployment with a volume or using the 'immediate' strategy? if not, this could be a delayed state issue)). Retrying in 1000 ms (attempt 30)

2023-03-27T08:53:45.818 proxy[f64442fc] sin [warn] Failed to proxy HTTP request (error: no known healthy instances found for route tcp/443. (hint: is your app shutdown? is there an ongoing deployment with a volume or using the 'immediate' strategy? if not, this could be a delayed state issue)). Retrying in 1000 ms (attempt 40)

2023-03-27T08:53:55.967 proxy[f64442fc] sin [warn] Failed to proxy HTTP request (error: no known healthy instances found for route tcp/443. (hint: is your app shutdown? is there an ongoing deployment with a volume or using the 'immediate' strategy? if not, this could be a delayed state issue)). Retrying in 1000 ms (attempt 50)

I didnā€™t change anything in my docker file since last Thursday (when everything worked) and now I have the same problem as you.

seems to be a fly.io issue ?

see also Unreachable Worker Host - #2 by a4z

Didnā€™t change anything, works again this morning ĀÆ_(惄)_/ĀÆ

Yah, that looks like a nomad placement issue perhaps.

Itā€™s still really slow at that last step:

 1 desired, 1 placed, 0 healthy, 0 unhealthy

Hangs for ages.

Is there something Iā€™m doing thatā€™s causing that?

My app is offline and redeploy gets stuck.


solved.

Still failing on allocating the swap for me.

 fallocate: fallocate failed: Text file busy

Iā€™ve changed the title to be more specific to my issue.

I can also confirm that there is no swap on the machine:

# cat /proc/meminfo | grep -i swap
SwapCached:            0 kB
SwapTotal:             0 kB
SwapFree:              0 kB

At this point Iā€™m at a loss, donā€™t know if thereā€™s staff checking this forum or I have to go on the paid plan.

Why not start the container and try running fallocate as root with a shell?

Yeah already tried.