What is the correct way to set swap for an app?

Elder · March 21, 2023, 1:20pm

I have tried the following in my dockerfile, but i got

cannot create /proc/sys/vm/swappiness: Read-only file system

RUN fallocate -l 1024M /swapfile && \
    chmod 0600 /swapfile && \
    mkswap /swapfile && \
    echo 10 > /proc/sys/vm/swappiness && \
    swapon /swapfile && \
    echo 1 > /proc/sys/vm/overcommit_memory

It’s for elixir/phoenix app

rubys · March 21, 2023, 1:38pm

You need that to run on each machine you deploy, not on the build machine.

The general approach is to change the ENTRYPOINT/CMD in your Dockerfile to run a script.

Create a script that runs the above commands and then the command that the ENTRYPOINT/CMD would have run. And then change your Dockerfile to run your script instead of your server.

Elder · March 21, 2023, 3:27pm

Thanks!

Any chance there is an example I could use? I couldn’t find any neither here on forum nor in docs

The generated dockerfile looks like this

...

COPY --from=builder --chown=nobody:root /app/_build/prod/rel/app ./

USER nobody

CMD ["/app/bin/server"]

# Appended by flyctl
ENV ECTO_IPV6 true
ENV ERL_AFLAGS "-proto_dist inet6_tcp"

rubys · March 21, 2023, 3:54pm

In your bin directory create a file named docker-entrypoint, with the following content:

#!/bin/bash -e

fallocate -l 1024M /swapfile
chmod 0600 /swapfile
mkswap /swapfile
echo 10 > /proc/sys/vm/swappiness
swapon /swapfile
echo 1 > /proc/sys/vm/overcommit_memory

/app/bin/server

Then in your Dockerfile change the CMD to

CMD ["/app/bin/docker-entrypoint"]

Elder · March 21, 2023, 5:33pm

@rubys does it need to run as root, not nobody?

I just tried it a got

2023-03-21T17:23:34Z app[91857293c4d6d8] ams [info]Preparing to run: `/app/bin/docker-entrypoint` as nobody
2023-03-21T17:23:34Z app[91857293c4d6d8] ams [info]Error: UnhandledIoError(Os { code: 13, kind: PermissionDenied, message: "Permission denied" })
2023-03-21T17:23:34Z app[91857293c4d6d8] ams [info][    0.159898] Kernel panic - not syncing: Attempted to kill init! exitcode=0x00000100
...
2023-03-21T17:23:34Z app[91857293c4d6d8] ams [info][    0.171596] Kernel Offset: disabled
2023-03-21T17:23:34Z app[91857293c4d6d8] ams [info][    0.171908] Rebooting in 1 seconds..
...
2023-03-21T17:23:36Z proxy[91857293c4d6d8] ams [info]Starting machine
2023-03-21T17:23:36Z proxy[91857293c4d6d8] ams [error]machines API returned an error: "machine ID 91857293c4d6d8 lease currently held by ..., expires at 2023-03-21T17:23:47Z"
...
2023-03-21T17:23:36Z app[91857293c4d6d8] ams [info]Starting init (commit: 08b4c2b)...
2023-03-21T17:23:36Z app[91857293c4d6d8] ams [info]Preparing to run: `/app/bin/docker-entrypoint` as nobody
2023-03-21T17:23:36Z app[91857293c4d6d8] ams [info]Error: UnhandledIoError(Os { code: 13, kind: PermissionDenied, message: "Permission denied" })
2023-03-21T17:23:36Z app[91857293c4d6d8] ams [info][    0.162307] Kernel panic - not syncing: Attempted to kill init! exitcode=0x00000100
...
2023-03-21T17:23:36Z app[91857293c4d6d8] ams [info][    0.172794] Kernel Offset: disabled
2023-03-21T17:23:36Z app[91857293c4d6d8] ams [info][    0.173149] Rebooting in 1 seconds..
2023-03-21T17:23:36Z proxy[91857293c4d6d8] ams [info]Starting machine
2023-03-21T17:23:36Z proxy[91857293c4d6d8] ams [error]machines API returned an error: "machine ID 91857293c4d6d8 lease currently held by ..., expires at 2023-03-21T17:23:47Z"
...
multiple times
...
2023-03-21T17:23:46Z proxy[91857293c4d6d8] ams [info]Starting machine
2023-03-21T17:23:46Z proxy[91857293c4d6d8] ams [error]machines API returned an error: "rate limit exceeded"
...
2023-03-21T17:26:01Z app[91857293c4d6d8] ams [warn]Virtual machine exited abruptly

It is still trying to launch it in a repeating cycle. That is weird, I’m not sure it works as it supposed to be. Why it tries to keep launching it instead of canceling the deployment. Also it took down one instance of the app (out of 2)

rubys · March 21, 2023, 5:40pm

Ah, yes. Remove the USER statement from your Dockerfile and try the following script:

#!/bin/bash -e

if [ $UID -eq 0 ]; then
  # allocate swap space
  fallocate -l 1024M /swapfile
  chmod 0600 /swapfile
  mkswap /swapfile
  echo 10 > /proc/sys/vm/swappiness
  swapon /swapfile
  echo 1 > /proc/sys/vm/overcommit_memory
  exec su nobody $0 $@
fi

exec /app/bin/server

This script when run as root will set up your swapfile and then rerun the script as nobody. On the second run it will start your server

Elder · March 21, 2023, 5:57pm

@rubys Similar issue again. Infinite deployment cycle. That doesn’t seem right. Why it never stops if it cannot launch it?

Deploying app with rolling strategy
  Waiting for 91857293c4d6d8 [app] to have state: started

2023-03-21T17:53:44Z runner[4d891d60a03778] ams [info]Pulling container image registry.fly.io/app:deployment-01GW2MW3YAD7D3FDTMPFB2BDJ7
2023-03-21T17:53:44Z runner[4d891d60a03778] ams [info]Unpacking image
2023-03-21T17:53:45Z runner[4d891d60a03778] ams [info]Configuring firecracker
2023-03-21T17:53:45Z app[4d891d60a03778] ams [info]Starting init (commit: 08b4c2b)...
2023-03-21T17:53:45Z app[4d891d60a03778] ams [info]Preparing to run: `/app/bin/migrate` as root
2023-03-21T17:53:45Z app[4d891d60a03778] ams [info]2023/03/21 17:53:45 listening on [fdaa:0:5884:a7b:10b:b6a9:1800:2]:22 (DNS: [fdaa::3]:53)
2023-03-21T17:53:47Z app[4d891d60a03778] ams [info]17:53:47.390 [info] Migrations already up
2023-03-21T17:53:48Z app[4d891d60a03778] ams [info]Starting clean up.
2023-03-21T17:53:49Z app[4d891d60a03778] ams [info][    4.120859] reboot: Restarting system
2023-03-21T17:53:50Z runner[4d891d60a03778] ams [info]machine restart policy set to 'no', not restarting
2023-03-21T17:53:52Z runner[91857293c4d6d8] ams [info]Pulling container image registry.fly.io/app:deployment-01GW2MW3YAD7D3FDTMPFB2BDJ7
2023-03-21T17:53:52Z runner[91857293c4d6d8] ams [info]Unpacking image
2023-03-21T17:53:52Z runner[91857293c4d6d8] ams [info]Configuring firecracker
2023-03-21T17:53:52Z app[91857293c4d6d8] ams [info]Sending signal SIGINT to main child process w/ PID 526
2023-03-21T17:53:52Z app[91857293c4d6d8] ams [info]BREAK: (a)bort (A)bort with dump (c)ontinue (p)roc info (i)nfo
2023-03-21T17:53:52Z app[91857293c4d6d8] ams [info]       (l)oaded (v)ersion (k)ill (D)b-tables (d)istribution
2023-03-21T17:53:52Z app[91857293c4d6d8] ams [info][os_mon] cpu supervisor port (cpu_sup): Erlang has closed
2023-03-21T17:53:52Z app[91857293c4d6d8] ams [info][os_mon] memory supervisor port (memsup): Erlang has closed
2023-03-21T17:53:53Z proxy[91857293c4d6d8] ams [error]machine is in a non-startable state: created
2023-03-21T17:53:53Z app[91857293c4d6d8] ams [info]Starting clean up.
2023-03-21T17:53:53Z proxy[91857293c4d6d8] ams [error]machine is in a non-startable state: created
2023-03-21T17:53:54Z app[91857293c4d6d8] ams [info][  191.401011] reboot: Restarting system
2023-03-21T17:53:55Z app[91857293c4d6d8] ams [info]Starting init (commit: 08b4c2b)...
2023-03-21T17:53:55Z app[91857293c4d6d8] ams [info]Preparing to run: `/app/bin/docker-entrypoint` as root
2023-03-21T17:53:55Z app[91857293c4d6d8] ams [info]Error: UnhandledIoError(Os { code: 13, kind: PermissionDenied, message: "Permission denie[    0.162085] Kernel panic - not syncing: Attempted to kill init! exitcode=0x00000100
2023-03-21T17:53:55Z app[91857293c4d6d8] ams [info][    0.163118] CPU: 0 PID: 1 Comm: init Not tainted 5.12.2 #1
2023-03-21T17:53:55Z app[91857293c4d6d8] ams [info][    0.163900] Call Trace:
2023-03-21T17:53:55Z app[91857293c4d6d8] ams [info][    0.164250]  show_stack+0x52/0x58
2023-03-21T17:53:55Z app[91857293c4d6d8] ams [info][    0.164748]  dump_stack+0x6b/0x86
2023-03-21T17:53:55Z app[91857293c4d6d8] ams [info][    0.165211]  panic+0xfb/0x2bc
2023-03-21T17:53:55Z app[91857293c4d6d8] ams [info][    0.165631]  do_exit.cold+0x60/0xb0
2023-03-21T17:53:55Z app[91857293c4d6d8] ams [info][    0.166125]  do_group_exit+0x3b/0xb0
2023-03-21T17:53:55Z app[91857293c4d6d8] ams [info][    0.166651]  __x64_sys_exit_group+0x18/0x20
2023-03-21T17:53:55Z app[91857293c4d6d8] ams [info][    0.167286]  do_syscall_64+0x38/0x50
2023-03-21T17:53:55Z app[91857293c4d6d8] ams [info][    0.167785]  entry_SYSCALL_64_after_hwframe+0x44/0xae
2023-03-21T17:53:55Z app[91857293c4d6d8] ams [info][    0.168471] RIP: 0033:0x7f194a141de3
2023-03-21T17:53:55Z app[91857293c4d6d8] ams [info][    0.168998] Code: eb ef 48 8b 76 28 e9 9b 03 00 00 64 48 8b 04 25 00 00 00 00 48 8b b0 b0 00 00 00 e9 af ff ff ff 48 63 ff b8 e7 00 00 00 0f 05 <ba> 3c 00 00 00 48 89 d0 0f 05 eb f9 90 ba 64 2f 00 00 c6 47 0e 00
2023-03-21T17:53:55Z app[91857293c4d6d8] ams [info][    0.171242] RSP: 002b:00007ffeccb3e3a8 EFLAGS: 00000246 ORIG_RAX: 00000000000000e7
2023-03-21T17:53:55Z app[91857293c4d6d8] ams [info][    0.171949] RAX: ffffffffffffffda RBX: 00007f1949f32b40 RCX: 00007f194a141de3
2023-03-21T17:53:55Z app[91857293c4d6d8] ams [info][    0.172587] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000001
2023-03-21T17:53:55Z app[91857293c4d6d8] ams [info][    0.173225] RBP: 0000000000000001 R08: 00007f194a23a818 R09: 0000000000000000
2023-03-21T17:53:55Z app[91857293c4d6d8] ams [info][    0.173886] R10: 0000000000000000 R11: 0000000000000246 R12: 00007ffeccb3e408
2023-03-21T17:53:55Z app[91857293c4d6d8] ams [info][    0.174514] R13: 00007ffeccb3e418 R14: 0000000000000000 R15: 0000000000000000
2023-03-21T17:53:55Z app[91857293c4d6d8] ams [info][    0.175259] Kernel Offset: disabled
2023-03-21T17:53:55Z app[91857293c4d6d8] ams [info][    0.175589] Rebooting in 1 seconds..
2023-03-21T17:53:56Z app[91857293c4d6d8] ams [warn]Virtual machine exited abruptly

rubys · March 21, 2023, 6:01pm

Ah, if you are developing on MacOS or Linux:

chmod +x bin/docker-entrypoint

If you are developing on Windows, add the following to your Dockerfile:

RUN chmod +x /app/bin/docker-entrypoint

Elder · March 21, 2023, 6:18pm

@rubys It now created the swap file but it seems it failed to start the app.

This account is currently not available does it refer to su nobody?

Why the deployment succeeded even with instance refused connection how health checks could have passed?

2023-03-21T18:12:10Z runner[91857293c4d6d8] ams [info]Pulling container image registry.fly.io/app:deployment-01GW2NXG216XTV8RVB5EJW2511
2023-03-21T18:12:11Z proxy[91857293c4d6d8] ams [error]machine is in a non-startable state: created
2023-03-21T18:12:11Z runner[91857293c4d6d8] ams [info]Unpacking image
2023-03-21T18:12:11Z runner[91857293c4d6d8] ams [info]Configuring firecracker
2023-03-21T18:12:12Z app[91857293c4d6d8] ams [info]Sending signal SIGINT to main child process w/ PID 527
2023-03-21T18:12:12Z app[91857293c4d6d8] ams [info]BREAK: (a)bort (A)bort with dump (c)ontinue (p)roc info (i)nfo
2023-03-21T18:12:12Z app[91857293c4d6d8] ams [info]       (l)oaded (v)ersion (k)ill (D)b-tables (d)istribution
2023-03-21T18:12:12Z app[91857293c4d6d8] ams [info][os_mon] cpu supervisor port (cpu_sup): Erlang has closed
2023-03-21T18:12:12Z app[91857293c4d6d8] ams [info][os_mon] memory supervisor port (memsup): Erlang has closed
2023-03-21T18:12:12Z app[91857293c4d6d8] ams [info]Starting clean up.
2023-03-21T18:12:13Z app[91857293c4d6d8] ams [info][  209.404093] reboot: Restarting system
2023-03-21T18:12:14Z app[91857293c4d6d8] ams [info]Starting init (commit: 08b4c2b)...
2023-03-21T18:12:14Z app[91857293c4d6d8] ams [info]Preparing to run: `/app/bin/docker-entrypoint` as root
2023-03-21T18:12:14Z app[91857293c4d6d8] ams [info]2023/03/21 18:12:14 listening on [fdaa:0:5884:a7b:10c:f2f8:3563:2]:22 (DNS: [fdaa::3]:53)
2023-03-21T18:12:15Z app[91857293c4d6d8] ams [info]Setting up swapspace version 1, size = 1024 MiB (1073737728 bytes)
2023-03-21T18:12:15Z app[91857293c4d6d8] ams [info]no label, UUID=711e360e-6b75-4e11-afae-e6cd4ffa974d
2023-03-21T18:12:15Z app[91857293c4d6d8] ams [info]This account is currently not available.
2023-03-21T18:12:15Z app[91857293c4d6d8] ams [info]Starting clean up.
2023-03-21T18:12:16Z app[5683d06a7904e8] ams [info]18:12:16.356 [warning] [libcluster:fly6pn] unable to connect to :"app@fdaa:0:5884:a7b:10c:f2f8:3563:2"
2023-03-21T18:12:16Z app[91857293c4d6d8] ams [info][    2.150045] reboot: Restarting system

...

2023-03-21T18:12:19Z proxy[91857293c4d6d8] ams [error]instance refused connection. is your app listening on 0.0.0.0:8080? make sure it is not only listening on 127.0.0.1 (hint: look at your startup logs, servers often print the address they are listening on)
...
2023-03-21T18:12:47Z proxy[91857293c4d6d8] ams [info]Starting machine
2023-03-21T18:12:47Z proxy[91857293c4d6d8] ams [error]machines API returned an error: "rate limit exceeded"
2023-03-21T18:12:48Z proxy[91857293c4d6d8] ams [info]Starting machine
2023-03-21T18:12:48Z proxy[91857293c4d6d8] ams [error]machines API returned an error: "rate limit exceeded"
2023-03-21T18:12:48Z proxy[91857293c4d6d8] ams [info]Starting machine
2023-03-21T18:12:48Z proxy[91857293c4d6d8] ams [error]machines API returned an error: "rate limit exceeded"
2023-03-21T18:12:49Z proxy[91857293c4d6d8] ams [info]Starting machine
2023-03-21T18:12:49Z proxy[91857293c4d6d8] ams [error]failed to change machine state: machine still active, refusing to start

rubys · March 21, 2023, 7:56pm

Yes, it looks like the user nobody doesn’t have a shell defined. Try replacing that line with:

exec su nobody -s /bin/bash -c "$0 $@"

Elder · March 26, 2023, 7:15pm

@rubys thanks a lot! it seems to be working now

I wish it was as simple as a setting in fly.toml.

Please consider adding this to the official Fly’s Phoenix dockerfile or documenting it in some other way, I’m probably is not the only one who could use it.

rubys · March 26, 2023, 10:42pm

Consider opening an issue on github to track this?

We support a number of frameworks, and I’ve got it down to a single command for Rails: Dockerfiles and fly.toml · Fly Docs . Each framework is a bit different.

system · April 2, 2023, 10:43pm

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
How to swap properly? "Out of memory: Killed process (node)" error on deploy postgres	3	674	April 5, 2023
Swap memory	14	5667	March 6, 2024
[Rails] Dockerfile generator with --swap option is not compatible with fly volumes Questions / Help	2	296	April 29, 2023
Django on fly.io Django	2	399	November 26, 2022
swapon failed: Invalid argument Questions / Help	5	766	April 28, 2024

What is the correct way to set swap for an app?

Related topics