Swap failing to allocate on deployment

I am building and pushing and image successfully in detached mode:

deployment-01GWG75AJG6QR9CBG2CMQ1SXQW: digest: sha256:a27bca86d3ecb16de91548aff3b09a6af9d29203131820fd01cbba1bec1ab8cf size: 2004
--> Pushing image done
image: registry.fly.io/<my app>:deployment-01GWG75AJG6QR9CBG2CMQ1SXQW
image size: 808 MB
==> Creating release
--> release v143 created

But that release never shows up in Activity (I can see up to v142) and I can see the code is actually not released to production by ssh-ing into the machine.

Deploys have been very unstable with Fly and it’s making me reconsider if I should use something else. It’s a very critical feature for a developer’s workflow.

Edit: I see this after the release is created:

 2023-03-27T01:56:32.282 app[56b14333] lax [info] fallocate: fallocate failed: Text file busy

2023-03-27T01:56:33.240 app[56b14333] lax [info] Starting clean up. 

This comes from the entrypoint trying to assign swap I believe. It was generated with the dockerfile generator. So it just seems it dies and does not show the failure anywhere else.

I’m not sure that has anything to do with fly. I did deploys on stage (nomad), prod (nomad), and then deployed roughly 100 times while testing out litefs on v2 apps to see how a migration would look.

Is this something new that you’re trying on an image?

Regarding activity, it doesn’t seem to show up for v2 apps consistently, which is definitely a fly thing.

Yeah I changed the dockerfile to allow swapping because the machine wasn’t able to run a Rails console. Used GitHub - rubys/dockerfile-rails: Provides a Rails generator to produce Dockerfiles and related files. from the docs.

This is the entrypoint generated for the dockerfile:

#!/bin/bash -e

if [ $UID -eq 0 ]; then
  # allocate swap space
  fallocate -l 512M /swapfile
  chmod 0600 /swapfile
  mkswap /swapfile
  echo 10 > /proc/sys/vm/swappiness
  swapon /swapfile
  echo 1 > /proc/sys/vm/overcommit_memory
  exec su rails $0 $@
fi

# If running the rails server then create or migrate existing database
if [ "${*}" == "./bin/rails server" ]; then
  ./bin/rails db:prepare
fi

exec "${@}"

That is a bit baffling as you’re in the if condition indicating this is running as root. The only explanation I can think of is that /swapfile already exists and is mounted?

As far as I read that should only happen at deploy time, which should launch a new container, but I can only speculate on how the flow is implemented.

Either way, it shouldn’t die on that statement so should probably make it

fallocate -l 512M /swapfile || echo "swap already assigned"

but I’m not 100% sure that the swap is there. I’ll try.

This version crashes on Rails:

#!/bin/bash -e

if [ $UID -eq 0 ]; then
  # allocate swap space
  if fallocate -l 512M /swapfile; then
    chmod 0600 /swapfile
    mkswap /swapfile
    echo 10 >/proc/sys/vm/swappiness
    swapon /swapfile
    echo 1 >/proc/sys/vm/overcommit_memory
  fi
  exec su rails $0 $@
fi

# If running the rails server then create or migrate existing database
if [ "${*}" == "./bin/rails server" ]; then
  ./bin/rails db:prepare
fi

exec "${@}"

I’m tired, I’ll try tomorrow.

1 Like

Don’t know if it’s related, but I’m getting all deploys hanging on the final “Placing…” step. Seems to be happened to both my apps, but they both use the same Dockerfile so may well be something silly I’m doing too :slight_smile:

same, getting proxy errors in logs

2023-03-27T08:53:35.720 proxy[f64442fc] sin [warn] Failed to proxy HTTP request (error: no known healthy instances found for route tcp/443. (hint: is your app shutdown? is there an ongoing deployment with a volume or using the 'immediate' strategy? if not, this could be a delayed state issue)). Retrying in 1000 ms (attempt 30)

2023-03-27T08:53:45.818 proxy[f64442fc] sin [warn] Failed to proxy HTTP request (error: no known healthy instances found for route tcp/443. (hint: is your app shutdown? is there an ongoing deployment with a volume or using the 'immediate' strategy? if not, this could be a delayed state issue)). Retrying in 1000 ms (attempt 40)

2023-03-27T08:53:55.967 proxy[f64442fc] sin [warn] Failed to proxy HTTP request (error: no known healthy instances found for route tcp/443. (hint: is your app shutdown? is there an ongoing deployment with a volume or using the 'immediate' strategy? if not, this could be a delayed state issue)). Retrying in 1000 ms (attempt 50)

I didn’t change anything in my docker file since last Thursday (when everything worked) and now I have the same problem as you.

seems to be a fly.io issue ?

see also Unreachable Worker Host - #2 by a4z

Didn’t change anything, works again this morning ¯_(ツ)_/¯

Yah, that looks like a nomad placement issue perhaps.

It’s still really slow at that last step:

 1 desired, 1 placed, 0 healthy, 0 unhealthy

Hangs for ages.

Is there something I’m doing that’s causing that?

My app is offline and redeploy gets stuck.


solved.

Still failing on allocating the swap for me.

 fallocate: fallocate failed: Text file busy

I’ve changed the title to be more specific to my issue.

I can also confirm that there is no swap on the machine:

# cat /proc/meminfo | grep -i swap
SwapCached:            0 kB
SwapTotal:             0 kB
SwapFree:              0 kB

At this point I’m at a loss, don’t know if there’s staff checking this forum or I have to go on the paid plan.

Why not start the container and try running fallocate as root with a shell?

Yeah already tried.