Error after deploy, WARN could not unmount /rootfs: EINVAL: Invalid argument

janjakubek · May 3, 2024, 7:28am

Hello, I have a problem after deploying an application.

After successful deployment the application does not want to start. In the log there is a warning: WARN could not unmount /rootfs: EINVAL: Invalid argument . What all could be causing the error?

fly.toml

app = 'myapp'
primary_region = 'cdg'
swap_size_mb = 512


[build]
  dockerfile = './Dockerfile'

[http_service]
  internal_port = 8080
  force_https = true
  auto_stop_machines = true
  auto_start_machines = true
  min_machines_running = 0
  processes = ['app']

[[vm]]
  memory = '1gb'
  cpu_kind = 'shared'
  cpus = 1

log

.
.
.
2024-05-03T06:25:16.276 app[148e393c06e0e8] cdg [info] The known commands are:
2024-05-03T06:25:16.276 app[148e393c06e0e8] cdg [info] start Starts the system
2024-05-03T06:25:16.276 app[148e393c06e0e8] cdg [info] start_iex Starts the system with IEx attached
2024-05-03T06:25:16.276 app[148e393c06e0e8] cdg [info] daemon Starts the system as a daemon
2024-05-03T06:25:16.276 app[148e393c06e0e8] cdg [info] daemon_iex Starts the system as a daemon with IEx attached
2024-05-03T06:25:16.276 app[148e393c06e0e8] cdg [info] eval "EXPR" Executes the given expression on a new, non-booted system
2024-05-03T06:25:16.276 app[148e393c06e0e8] cdg [info] rpc "EXPR" Executes the given expression remotely on the running system
2024-05-03T06:25:16.276 app[148e393c06e0e8] cdg [info] remote Connects to the running system via a remote shell
2024-05-03T06:25:16.276 app[148e393c06e0e8] cdg [info] restart Restarts the running system via a remote command
2024-05-03T06:25:16.276 app[148e393c06e0e8] cdg [info] stop Stops the running system via a remote command
2024-05-03T06:25:16.276 app[148e393c06e0e8] cdg [info] pid Prints the operating system PID of the running system via a remote command
2024-05-03T06:25:16.276 app[148e393c06e0e8] cdg [info] version Prints the release name and version to be booted
2024-05-03T06:25:17.248 app[148e393c06e0e8] cdg [info] INFO Main child exited normally with code: 0
2024-05-03T06:25:17.260 app[148e393c06e0e8] cdg [info] INFO Starting clean up.
2024-05-03T06:25:17.270 proxy[148e393c06e0e8] cdg [info] machine started in 1.515921645s
2024-05-03T06:25:17.270 app[148e393c06e0e8] cdg [info] WARN could not unmount /rootfs: EINVAL: Invalid argument
2024-05-03T06:25:17.271 app[148e393c06e0e8] cdg [info] [ 1.385233] reboot: Restarting system
2024-05-03T06:25:18.543 runner[148e393c06e0e8] cdg [info] machine exited with exit code 0, not restarting
.
.
.

deploy

.
.
.
--> Building image done
==> Pushing image to fly
The push refers to repository [registry.fly.io/myapp]
d1371471e36e: Layer already exists
7a6e9172753b: Layer already exists
3258b070e14b: Layer already exists
4ef79693d025: Layer already exists
281510ba10f9: Layer already exists
ba5a5fe43301: Layer already exists
deployment-01HWYHG747WXXX: digest: sha256:ed80bb8a0babf8459f4c7XXX size: 1577
--> Pushing image done
image: registry.fly.io/myapp:deployment-01HWY...
image size: 755 MB

Watch your deployment at https://fly.io/apps/myapp/monitoring

-------
Updating existing machines in 'myapp' with rolling strategy

-------
 ✔ [1/2] Machine 148e393c06e0e8 [app] update succeeded
 ✔ [2/2] Machine e7843ed7f96283 [app] update succeeded
.
.
.

The app has only been running locally for me so far, I’m trying to revive an old project after someone else.

Versions in Dockerfile

.
.
.
ARG ELIXIR_VERSION=1.11.2
ARG OTP_VERSION=23.2.1
ARG DEBIAN_VERSION=bullseye-20210902-slim
.
.
.

I originally thought adding swap_size_mb would help, but I still have a problem.

I’ve come across a few Topics about this bug, but haven’t managed to resolve it yet. Thanks for any ideas and help.

What I found:

https://community.fly.io/t/cant-deploy-smoke-checks-fail-warn-could-not-unmount-rootfs-einval-invalid-argument/19464/1

nbgoodall · May 3, 2024, 8:42am

So I just had this with a new deploy, and turns out I didn’t have enough memory (was a Rails app, 256mb wasn’t enough). After scaling that all works fine

jsphc · May 3, 2024, 11:22am

By any chance do you also have a swap file or swapon declaration in your fly.rake file? I had a similar issue last week where an app that was working for over a year suddenly got stuck in a restart loop with this same error. It ended up being fixed by removing the optional declaration in the rake file and adding it to the toml (which it looks like you already have).

janjakubek · May 3, 2024, 12:05pm

Thanks guys, unfortunately it didn’t help me, I still have the same problem.
Since this app never ran on Flyi, I’m wondering what all is capable of causing this error. Could it be anything in fly.toml or other configuration files? Any SECRET not set or something similar? I want to know how widely to look for the error.

I alsoI also tried following this tutorial from the beginning, but I got the same error.

tomasmillar · May 3, 2024, 10:13pm

I have just started getting this error as well… I am using remix run.

ben-io · May 3, 2024, 10:26pm

This error is a red herring. You can ignore it.

janjakubek:

2024-05-03T06:25:16.276 app[148e393c06e0e8] cdg [info] The known commands are:
2024-05-03T06:25:16.276 app[148e393c06e0e8] cdg [info] start Starts the system
2024-05-03T06:25:16.276 app[148e393c06e0e8] cdg [info] start_iex Starts the system with IEx attached
2024-05-03T06:25:16.276 app[148e393c06e0e8] cdg [info] daemon Starts the system as a daemon
2024-05-03T06:25:16.276 app[148e393c06e0e8] cdg [info] daemon_iex Starts the system as a daemon with IEx attached
2024-05-03T06:25:16.276 app[148e393c06e0e8] cdg [info] eval "EXPR" Executes the given expression on a new, non-booted system
2024-05-03T06:25:16.276 app[148e393c06e0e8] cdg [info] rpc "EXPR" Executes the given expression remotely on the running system
2024-05-03T06:25:16.276 app[148e393c06e0e8] cdg [info] remote Connects to the running system via a remote shell
2024-05-03T06:25:16.276 app[148e393c06e0e8] cdg [info] restart Restarts the running system via a remote command
2024-05-03T06:25:16.276 app[148e393c06e0e8] cdg [info] stop Stops the running system via a remote command
2024-05-03T06:25:16.276 app[148e393c06e0e8] cdg [info] pid Prints the operating system PID of the running system via a remote command
2024-05-03T06:25:16.276 app[148e393c06e0e8] cdg [info] version Prints the release name and version to be booted
2024-05-03T06:25:17.248 app[148e393c06e0e8] cdg [info] INFO Main child exited normally with code: 0

It looks like your app is exiting immediately, after printing a help text. This makes me think that your CMD in your Dockerfile is not configured properly. I think you need to add start add the end, for example:

CMD ["/app/bin/sandwiches", "start"]

tomasmillar · May 4, 2024, 2:06pm

I am getting the same error.
Here are my logs:

2024-05-04T13:53:03.060 app[d8dd67ea24e748] lhr [info] [ 0.046093] Spectre V2 : WARNING: Unprivileged eBPF is enabled with eIBRS on, data leaks possible via Spectre v2 BHB attacks!
2024-05-04T13:53:03.120 app[d8dd67ea24e748] lhr [info] [ 0.066030] PCI: Fatal: No config space access function found
2024-05-04T13:53:03.318 app[d8dd67ea24e748] lhr [info] INFO Starting init (commit: c1e2693b)…
2024-05-04T13:53:03.391 app[d8dd67ea24e748] lhr [info] INFO Preparing to run: /app/docker-entrypoint npm run start as root
2024-05-04T13:53:03.397 app[d8dd67ea24e748] lhr [info] INFO [fly api proxy] listening at /.fly/api
2024-05-04T13:53:03.405 app[d8dd67ea24e748] lhr [info] 2024/05/04 13:53:03 INFO SSH listening listen_address=[fdaa:1:b9f5:a7b:1b:3192:f82e:2]:22 dns_server=[fdaa::3]:53
2024-05-04T13:53:03.407 app[d8dd67ea24e748] lhr [info] /bin/bash: - : invalid option
2024-05-04T13:53:04.031 runner[d8dd67ea24e748] lhr [info] Machine started in 1.14s
2024-05-04T13:53:04.398 app[d8dd67ea24e748] lhr [info] INFO Main child exited normally with code: 1
2024-05-04T13:53:04.414 app[d8dd67ea24e748] lhr [info] INFO Starting clean up.
2024-05-04T13:53:04.415 app[d8dd67ea24e748] lhr [info] WARN could not unmount /rootfs: EINVAL: Invalid argument
2024-05-04T13:53:04.416 app[d8dd67ea24e748] lhr [info] [ 1.360391] reboot: Restarting system
2024-05-04T13:53:04.526 runner[d8dd67ea24e748] lhr [info] machine did not have a restart policy, defaulting to restart
2024-05-04T13:53:24.057 app[d8dd67ea24e748] lhr [info] [ 0.041393] Spectre V2 : WARNING: Unprivileged eBPF is enabled with eIBRS on, data leaks possible via Spectre v2 BHB attacks!
2024-05-04T13:53:24.116 app[d8dd67ea24e748] lhr [info] [ 0.061420] PCI: Fatal: No config space access function found
2024-05-04T13:53:24.306 app[d8dd67ea24e748] lhr [info] INFO Starting init (commit: c1e2693b)…
2024-05-04T13:53:24.379 app[d8dd67ea24e748] lhr [info] INFO Preparing to run: /app/docker-entrypoint npm run start as root
2024-05-04T13:53:24.386 app[d8dd67ea24e748] lhr [info] INFO [fly api proxy] listening at /.fly/api
2024-05-04T13:53:24.394 app[d8dd67ea24e748] lhr [info] 2024/05/04 13:53:24 INFO SSH listening listen_address=[fdaa:1:b9f5:a7b:1b:3192:f82e:2]:22 dns_server=[fdaa::3]:53
2024-05-04T13:53:24.394 app[d8dd67ea24e748] lhr [info] /bin/bash: - : invalid option
2024-05-04T13:53:24.400 runner[d8dd67ea24e748] lhr [info] Machine started in 496ms
2024-05-04T13:53:25.386 app[d8dd67ea24e748] lhr [info] INFO Main child exited normally with code: 1
2024-05-04T13:53:25.402 app[d8dd67ea24e748] lhr [info] INFO Starting clean up.
2024-05-04T13:53:25.404 app[d8dd67ea24e748] lhr [info] WARN could not unmount /rootfs: EINVAL: Invalid argument
2024-05-04T13:53:25.404 app[d8dd67ea24e748] lhr [info] [ 1.348495] reboot: Restarting system
2024-05-04T13:53:25.514 runner[d8dd67ea24e748] lhr [info] machine has reached its max restart count (10)

Here is my Docker file:

syntax = docker/dockerfile:1
Adjust NODE_VERSION as desired
ARG NODE_VERSION=18.14.2
FROM node:${NODE_VERSION}-slim as base
LABEL fly_launch_runtime=“Remix/Prisma”
Remix/Prisma app lives here
WORKDIR /app
Set production environment
ENV NODE_ENV=production
Throw-away build stage to reduce size of final image
FROM base as build
Install packages needed to build node modules
RUN apt-get update -qq &&
apt-get install -y python-is-python3 pkg-config build-essential openssl
Install node modules
COPY --link package.json package-lock.json .
RUN npm install --production=false
Generate Prisma Client
COPY --link prisma .
RUN npx prisma generate
Copy application code
COPY --link . .
Build application
RUN npm run build
Remove development dependencies
RUN npm prune --production
Final stage for app image
FROM base
Copy built application
COPY --from=build /app /app
Entrypoint prepares the database.
ENTRYPOINT [“/app/docker-entrypoint”]

Start the server by default, this can be overwritten at runtime
CMD [ “npm”, “run”, “start” ]

Here is my fly.toml:

app = “appname”
kill_signal = “SIGINT”
kill_timeout = 15
primary_region = “lhr”
processes =

[build]

[env]
PORT = “8080”

[experimental]
auto_rollback = true

[[services]]
http_checks =
internal_port = 8080
processes = [“app”]
protocol = “tcp”
script_checks =
[services.concurrency]
hard_limit = 25
soft_limit = 20
type = “connections”

[[services.ports]]
force_https = true
handlers = [“http”]
port = 80

[[services.ports]]
handlers = [“tls”, “http”]
port = 443

[[services.tcp_checks]]
grace_period = “1s”
interval = “15s”
restart_limit = 0
timeout = “2s”

[[vm]]
size = ‘shared-cpu-2x’
memory = ‘2gb’
cpu_kind = ‘shared’
cpus = 2

janjakubek · May 6, 2024, 7:29am

Thanks for the advice, I had also misplaced other things in the config and still troubled by errors, but I have moved on from this problem.

ben-io · May 6, 2024, 6:37pm

This is your real error. Maybe there’s a syntax error in docker-entrypoint? Can you share that.

tomasmillar · May 7, 2024, 1:25pm

Docker file below:

syntax = docker/dockerfile:1

Adjust NODE_VERSION as desired
ARG NODE_VERSION=18.14.2
FROM node:${NODE_VERSION}-slim as base

LABEL fly_launch_runtime=“Remix/Prisma”

Remix/Prisma app lives here
WORKDIR /app

Set production environment
ENV NODE_ENV=production

Throw-away build stage to reduce size of final image
FROM base as build

Install packages needed to build node modules
RUN apt-get update -qq &&
apt-get install -y python-is-python3 pkg-config build-essential openssl

Install node modules
COPY --link package.json package-lock.json .
RUN npm install --production=false

enerate Prisma Client
COPY --link prisma .
RUN npx prisma generate

Copy application code
COPY --link . .

Build application
RUN npm run build

Remove development dependencies
RUN npm prune --production

Final stage for app image
FROM base

Copy built application
COPY --from=build /app /app

Entrypoint prepares the database.
ENTRYPOINT [“/app/docker-entrypoint”]

Start the server by default, this can be overwritten at runtime
CMD [ “npm”, “run”, “start” ]

tomasmillar · May 7, 2024, 1:36pm

I seem to have solved it by taking ouit these two lines altogether. Not sure if they are needed or not?

Marcus100 · May 10, 2024, 2:36am

I get this same error. I also use Remix Run
WARN could not unmount /rootfs: EINVAL: Invalid argument

I didn’t change anything, and only changed some components. Never changed my Dockerfile or fly.toml file.

Was this solved?

peterldowns · June 6, 2024, 6:07pm

Following up, I see this warning line when my machines are auto-scaled down. Everything is working perfectly as far as I can tell — my app runs just fine, when i stop making requests it scales to zero — but I just want to know:

is this a problem I should be worried about?
is there anything I can do to stop the warning from being logged?

Here are some example logs from my application as it gets scaled to zero:

2024-06-06T17:59:45Z proxy[080e710a1124d8] ewr [info]App panoctocon has excess capacity, auto stopping machine 080e710a1124d8. 0 out of 1 machines left running (region=ewr, process group=app)
2024-06-06T17:59:45Z app[080e710a1124d8] ewr [info] INFO Sending signal SIGINT to main child process w/ PID 323
2024-06-06T17:59:45Z app[080e710a1124d8] ewr [info]INFO shutting down signal=interrupt state=cancel
2024-06-06T17:59:45Z app[080e710a1124d8] ewr [info]INFO done state=complete
2024-06-06T17:59:45Z app[080e710a1124d8] ewr [info] INFO Main child exited normally with code: 0
2024-06-06T17:59:45Z app[080e710a1124d8] ewr [info] INFO Starting clean up.
2024-06-06T17:59:45Z app[080e710a1124d8] ewr [info] WARN could not unmount /rootfs: EINVAL: Invalid argument
2024-06-06T17:59:45Z app[080e710a1124d8] ewr [info][  386.897196] reboot: Restarting system

If it helps, I’m using a container based on alpine where I just copy in some html/css/js and a golang server binary. Happy to share with someone from the Fly team if it would help debug.

ben-io · June 6, 2024, 6:14pm

You can ignore that warning.

peterldowns · June 10, 2024, 8:47pm

That’s great to hear — thanks!

vishalvibes · September 29, 2024, 5:13am

@ben-io I am also getting this error, should I also ignore this?

FROM python:3.12-slim

ENV PYTHONUNBUFFERED=1 \
    PYTHONDONTWRITEBYTECODE=1

WORKDIR /app

RUN pip install poetry

COPY . .

RUN poetry install --no-root

CMD ["poetry", "run", "uvicorn", "app.main:app", "--host", "0.0.0.0", "--port", "8000"]

Topic		Replies	Views
Problems deploy my first django app - could not unmount /rootfs: EINVAL Django postgres , machines	2	263	June 2, 2024
Getting WARN could not unmount /rootfs: EINVAL: Invalid argument Build debugging machines	0	43	September 29, 2024
App machines are not restarting after recent deployment	2	139	May 9, 2024
Fly Vapor Server Restarting Questions / Help	5	54	August 14, 2024
Deployment issue	4	616	November 13, 2020

Error after deploy, WARN could not unmount /rootfs: EINVAL: Invalid argument

Related topics