Error after deploy, WARN could not unmount /rootfs: EINVAL: Invalid argument

Hello, I have a problem after deploying an application.

After successful deployment the application does not want to start. In the log there is a warning: WARN could not unmount /rootfs: EINVAL: Invalid argument . What all could be causing the error?

fly.toml

app = 'myapp'
primary_region = 'cdg'
swap_size_mb = 512


[build]
  dockerfile = './Dockerfile'

[http_service]
  internal_port = 8080
  force_https = true
  auto_stop_machines = true
  auto_start_machines = true
  min_machines_running = 0
  processes = ['app']

[[vm]]
  memory = '1gb'
  cpu_kind = 'shared'
  cpus = 1

log

.
.
.
2024-05-03T06:25:16.276 app[148e393c06e0e8] cdg [info] The known commands are:
2024-05-03T06:25:16.276 app[148e393c06e0e8] cdg [info] start Starts the system
2024-05-03T06:25:16.276 app[148e393c06e0e8] cdg [info] start_iex Starts the system with IEx attached
2024-05-03T06:25:16.276 app[148e393c06e0e8] cdg [info] daemon Starts the system as a daemon
2024-05-03T06:25:16.276 app[148e393c06e0e8] cdg [info] daemon_iex Starts the system as a daemon with IEx attached
2024-05-03T06:25:16.276 app[148e393c06e0e8] cdg [info] eval "EXPR" Executes the given expression on a new, non-booted system
2024-05-03T06:25:16.276 app[148e393c06e0e8] cdg [info] rpc "EXPR" Executes the given expression remotely on the running system
2024-05-03T06:25:16.276 app[148e393c06e0e8] cdg [info] remote Connects to the running system via a remote shell
2024-05-03T06:25:16.276 app[148e393c06e0e8] cdg [info] restart Restarts the running system via a remote command
2024-05-03T06:25:16.276 app[148e393c06e0e8] cdg [info] stop Stops the running system via a remote command
2024-05-03T06:25:16.276 app[148e393c06e0e8] cdg [info] pid Prints the operating system PID of the running system via a remote command
2024-05-03T06:25:16.276 app[148e393c06e0e8] cdg [info] version Prints the release name and version to be booted
2024-05-03T06:25:17.248 app[148e393c06e0e8] cdg [info] INFO Main child exited normally with code: 0
2024-05-03T06:25:17.260 app[148e393c06e0e8] cdg [info] INFO Starting clean up.
2024-05-03T06:25:17.270 proxy[148e393c06e0e8] cdg [info] machine started in 1.515921645s
2024-05-03T06:25:17.270 app[148e393c06e0e8] cdg [info] WARN could not unmount /rootfs: EINVAL: Invalid argument
2024-05-03T06:25:17.271 app[148e393c06e0e8] cdg [info] [ 1.385233] reboot: Restarting system
2024-05-03T06:25:18.543 runner[148e393c06e0e8] cdg [info] machine exited with exit code 0, not restarting
.
.
.

deploy

.
.
.
--> Building image done
==> Pushing image to fly
The push refers to repository [registry.fly.io/myapp]
d1371471e36e: Layer already exists
7a6e9172753b: Layer already exists
3258b070e14b: Layer already exists
4ef79693d025: Layer already exists
281510ba10f9: Layer already exists
ba5a5fe43301: Layer already exists
deployment-01HWYHG747WXXX: digest: sha256:ed80bb8a0babf8459f4c7XXX size: 1577
--> Pushing image done
image: registry.fly.io/myapp:deployment-01HWY...
image size: 755 MB

Watch your deployment at https://fly.io/apps/myapp/monitoring

-------
Updating existing machines in 'myapp' with rolling strategy

-------
 ✔ [1/2] Machine 148e393c06e0e8 [app] update succeeded
 ✔ [2/2] Machine e7843ed7f96283 [app] update succeeded
.
.
.

The app has only been running locally for me so far, I’m trying to revive an old project after someone else.

Versions in Dockerfile

.
.
.
ARG ELIXIR_VERSION=1.11.2
ARG OTP_VERSION=23.2.1
ARG DEBIAN_VERSION=bullseye-20210902-slim
.
.
.

I originally thought adding swap_size_mb would help, but I still have a problem.

I’ve come across a few Topics about this bug, but haven’t managed to resolve it yet. Thanks for any ideas and help. :slight_smile:

What I found:

https://community.fly.io/t/cant-deploy-smoke-checks-fail-warn-could-not-unmount-rootfs-einval-invalid-argument/19464/1

1 Like

So I just had this with a new deploy, and turns out I didn’t have enough memory (was a Rails app, 256mb wasn’t enough). After scaling that all works fine :slight_smile:

1 Like

By any chance do you also have a swap file or swapon declaration in your fly.rake file? I had a similar issue last week where an app that was working for over a year suddenly got stuck in a restart loop with this same error. It ended up being fixed by removing the optional declaration in the rake file and adding it to the toml (which it looks like you already have).

1 Like

Thanks guys, unfortunately it didn’t help me, I still have the same problem.
Since this app never ran on Flyi, I’m wondering what all is capable of causing this error. Could it be anything in fly.toml or other configuration files? Any SECRET not set or something similar? I want to know how widely to look for the error. :slight_smile:

I alsoI also tried following this tutorial from the beginning, but I got the same error.

I have just started getting this error as well… I am using remix run.

This error is a red herring. You can ignore it.

It looks like your app is exiting immediately, after printing a help text. This makes me think that your CMD in your Dockerfile is not configured properly. I think you need to add start add the end, for example:

CMD ["/app/bin/sandwiches", "start"]
1 Like

I am getting the same error.
Here are my logs:

2024-05-04T13:53:03.060 app[d8dd67ea24e748] lhr [info] [ 0.046093] Spectre V2 : WARNING: Unprivileged eBPF is enabled with eIBRS on, data leaks possible via Spectre v2 BHB attacks!
2024-05-04T13:53:03.120 app[d8dd67ea24e748] lhr [info] [ 0.066030] PCI: Fatal: No config space access function found
2024-05-04T13:53:03.318 app[d8dd67ea24e748] lhr [info] INFO Starting init (commit: c1e2693b)…
2024-05-04T13:53:03.391 app[d8dd67ea24e748] lhr [info] INFO Preparing to run: /app/docker-entrypoint npm run start as root
2024-05-04T13:53:03.397 app[d8dd67ea24e748] lhr [info] INFO [fly api proxy] listening at /.fly/api
2024-05-04T13:53:03.405 app[d8dd67ea24e748] lhr [info] 2024/05/04 13:53:03 INFO SSH listening listen_address=[fdaa:1:b9f5:a7b:1b:3192:f82e:2]:22 dns_server=[fdaa::3]:53
2024-05-04T13:53:03.407 app[d8dd67ea24e748] lhr [info] /bin/bash: - : invalid option
2024-05-04T13:53:04.031 runner[d8dd67ea24e748] lhr [info] Machine started in 1.14s
2024-05-04T13:53:04.398 app[d8dd67ea24e748] lhr [info] INFO Main child exited normally with code: 1
2024-05-04T13:53:04.414 app[d8dd67ea24e748] lhr [info] INFO Starting clean up.
2024-05-04T13:53:04.415 app[d8dd67ea24e748] lhr [info] WARN could not unmount /rootfs: EINVAL: Invalid argument
2024-05-04T13:53:04.416 app[d8dd67ea24e748] lhr [info] [ 1.360391] reboot: Restarting system
2024-05-04T13:53:04.526 runner[d8dd67ea24e748] lhr [info] machine did not have a restart policy, defaulting to restart
2024-05-04T13:53:24.057 app[d8dd67ea24e748] lhr [info] [ 0.041393] Spectre V2 : WARNING: Unprivileged eBPF is enabled with eIBRS on, data leaks possible via Spectre v2 BHB attacks!
2024-05-04T13:53:24.116 app[d8dd67ea24e748] lhr [info] [ 0.061420] PCI: Fatal: No config space access function found
2024-05-04T13:53:24.306 app[d8dd67ea24e748] lhr [info] INFO Starting init (commit: c1e2693b)…
2024-05-04T13:53:24.379 app[d8dd67ea24e748] lhr [info] INFO Preparing to run: /app/docker-entrypoint npm run start as root
2024-05-04T13:53:24.386 app[d8dd67ea24e748] lhr [info] INFO [fly api proxy] listening at /.fly/api
2024-05-04T13:53:24.394 app[d8dd67ea24e748] lhr [info] 2024/05/04 13:53:24 INFO SSH listening listen_address=[fdaa:1:b9f5:a7b:1b:3192:f82e:2]:22 dns_server=[fdaa::3]:53
2024-05-04T13:53:24.394 app[d8dd67ea24e748] lhr [info] /bin/bash: - : invalid option
2024-05-04T13:53:24.400 runner[d8dd67ea24e748] lhr [info] Machine started in 496ms
2024-05-04T13:53:25.386 app[d8dd67ea24e748] lhr [info] INFO Main child exited normally with code: 1
2024-05-04T13:53:25.402 app[d8dd67ea24e748] lhr [info] INFO Starting clean up.
2024-05-04T13:53:25.404 app[d8dd67ea24e748] lhr [info] WARN could not unmount /rootfs: EINVAL: Invalid argument
2024-05-04T13:53:25.404 app[d8dd67ea24e748] lhr [info] [ 1.348495] reboot: Restarting system
2024-05-04T13:53:25.514 runner[d8dd67ea24e748] lhr [info] machine has reached its max restart count (10)

Here is my Docker file:

syntax = docker/dockerfile:1
Adjust NODE_VERSION as desired
ARG NODE_VERSION=18.14.2
FROM node:${NODE_VERSION}-slim as base
LABEL fly_launch_runtime=“Remix/Prisma”
Remix/Prisma app lives here
WORKDIR /app
Set production environment
ENV NODE_ENV=production
Throw-away build stage to reduce size of final image
FROM base as build
Install packages needed to build node modules
RUN apt-get update -qq &&
apt-get install -y python-is-python3 pkg-config build-essential openssl
Install node modules
COPY --link package.json package-lock.json .
RUN npm install --production=false
Generate Prisma Client
COPY --link prisma .
RUN npx prisma generate
Copy application code
COPY --link . .
Build application
RUN npm run build
Remove development dependencies
RUN npm prune --production
Final stage for app image
FROM base
Copy built application
COPY --from=build /app /app
Entrypoint prepares the database.
ENTRYPOINT [“/app/docker-entrypoint”]

Start the server by default, this can be overwritten at runtime
CMD [ “npm”, “run”, “start” ]

Here is my fly.toml:

app = “appname”
kill_signal = “SIGINT”
kill_timeout = 15
primary_region = “lhr”
processes =

[build]

[env]
PORT = “8080”

[experimental]
auto_rollback = true

[[services]]
http_checks =
internal_port = 8080
processes = [“app”]
protocol = “tcp”
script_checks =
[services.concurrency]
hard_limit = 25
soft_limit = 20
type = “connections”

[[services.ports]]
force_https = true
handlers = [“http”]
port = 80

[[services.ports]]
handlers = [“tls”, “http”]
port = 443

[[services.tcp_checks]]
grace_period = “1s”
interval = “15s”
restart_limit = 0
timeout = “2s”

[[vm]]
size = ‘shared-cpu-2x’
memory = ‘2gb’
cpu_kind = ‘shared’
cpus = 2

Thanks for the advice, I had also misplaced other things in the config and still troubled by errors, but I have moved on from this problem. :slight_smile:

This is your real error. Maybe there’s a syntax error in docker-entrypoint? Can you share that.

Docker file below:

syntax = docker/dockerfile:1

Adjust NODE_VERSION as desired
ARG NODE_VERSION=18.14.2
FROM node:${NODE_VERSION}-slim as base

LABEL fly_launch_runtime=“Remix/Prisma”

Remix/Prisma app lives here
WORKDIR /app

Set production environment
ENV NODE_ENV=production

Throw-away build stage to reduce size of final image
FROM base as build

Install packages needed to build node modules
RUN apt-get update -qq &&
apt-get install -y python-is-python3 pkg-config build-essential openssl

Install node modules
COPY --link package.json package-lock.json .
RUN npm install --production=false

enerate Prisma Client
COPY --link prisma .
RUN npx prisma generate

Copy application code
COPY --link . .

Build application
RUN npm run build

Remove development dependencies
RUN npm prune --production

Final stage for app image
FROM base

Copy built application
COPY --from=build /app /app

Entrypoint prepares the database.
ENTRYPOINT [“/app/docker-entrypoint”]

Start the server by default, this can be overwritten at runtime
CMD [ “npm”, “run”, “start” ]

I seem to have solved it by taking ouit these two lines altogether. Not sure if they are needed or not?

I get this same error. I also use Remix Run
WARN could not unmount /rootfs: EINVAL: Invalid argument

I didn’t change anything, and only changed some components. Never changed my Dockerfile or fly.toml file.

Was this solved?