Using "image": No such file or directory

Questions / Help
1

I’m trying to deploy the app at GitHub - jsha/unboundtest: Web service to test DNS resolution against Unbound with config similar to Let's Encrypt, which I build with a custom Dockerfile (in that repo). When I build from that Dockerfile and locally run:

docker run -p 1232 -it j4cob/unboundtest

It works properly. However, when I run: docker push j4cob/unboundtest:latest and flyctl deploy I get an error:

flyctl deploy
Deploying multi-dns
==> Validating app configuration
--> Validating app configuration done
Services
TCP 80/443 ⇢ 1232
==> Creating build context
--> Creating build context done
==> Building image with Docker
Step 1/11 : FROM ubuntu:hirsute
 ---> 58fb7079c94b
Step 2/11 : ENV DEBIAN_FRONTEND noninteractive
 ---> Using cache
 ---> 10333af7699b
Step 3/11 : RUN apt update && apt -y install unbound golang-go
 ---> Using cache
 ---> 35866951c83f
Step 4/11 : COPY . /unboundtest
 ---> 951ddff57571
Step 5/11 : WORKDIR /unboundtest
 ---> Running in 61a7822b1d90
 ---> 3f217d61d2c9
Step 6/11 : RUN GOBIN=/usr/bin go install ./
 ---> Running in 032f029da0d1
go: downloading github.com/miekg/dns v1.1.41
go: downloading golang.org/x/sys v0.0.0-20210303074136-134d130e1a04
go: downloading golang.org/x/net v0.0.0-20210226172049-e18ecbb05110
 ---> 0d5888b3cfb3
Step 7/11 : RUN mkdir -p /var/run/unboundtest
 ---> Running in 3a8d67efac43
 ---> f310073443ae
Step 8/11 : COPY index.html root.key unbound.conf /var/run/unboundtest/
 ---> 667655b05585
Step 9/11 : WORKDIR /var/run/unboundtest
 ---> Running in 39505a24926a
 ---> 8d7d4a68a920
Step 10/11 : EXPOSE 1232
 ---> Running in b4f5ae67a71b
 ---> b2c02fb696a0
Step 11/11 : CMD /usr/bin/unboundtest
 ---> Running in efa4812a829b
 ---> 0a564a917cbb
got a message  {  <nil>    0 0 <nil>  0xc00029a3d8}
Successfully built 0a564a917cbb
Successfully tagged registry.fly.io/multi-dns:deployment-1617741363
--> Building image done
==> Pushing image to fly
The push refers to repository [registry.fly.io/multi-dns]
b24971c82641: Pushed
4de9f7fc1814: Pushed
6e937022410f: Pushed
299b3a803a8b: Pushed
cef0b5acd521: Layer already exists
5f5822f0bce9: Layer already exists
c6012a8ebf34: Layer already exists
9dd73792a95c: Layer already exists
deployment-1617741363: digest: sha256:94e3545ddbd97ff6fe3879102feec920e85ce05e84713408807b72b05711287d size: 1992
--> Pushing image done
Image: registry.fly.io/multi-dns:deployment-1617741363
Image size: 863 MB
==> Creating release
Release v0 created
Deploying to multi-dns.fly.dev
Monitoring Deployment
You can detach the terminal anytime without stopping the deployment

1 desired, 1 placed, 0 healthy, 1 unhealthy
v3 failed - Failed due to unhealthy allocations - no stable job version to auto revert to
***v3 failed - Failed due to unhealthy allocations - no stable job version to auto revert to and deploying as v4

Troubleshooting guide at https://fly.io/docs/getting-started/troubleshooting/

My fly.toml contains:

[build]
  image = "j4cob/unboundtest:latest"

Here’s the output of flyctl logs:

2021-04-06T01:02:19.231Z bec617ce sea [info] Starting instance
2021-04-06T01:02:19.269Z bec617ce sea [info] Configuring virtual machine
2021-04-06T01:02:19.271Z bec617ce sea [info] Pulling container image
2021-04-06T01:02:26.845Z bec617ce sea [info] Unpacking image
2021-04-06T01:02:39.728Z bec617ce sea [info] Preparing kernel init
2021-04-06T01:02:40.138Z bec617ce sea [info] Configuring firecracker
2021-04-06T01:02:40.229Z bec617ce sea [info] Starting virtual machine
2021-04-06T01:02:40.407Z bec617ce sea [info] Starting init (commit: 0512da4)...
2021-04-06T01:02:40.427Z bec617ce sea [info] Running: `/bin/sh -c /usr/bin/unboundtest` as root
2021-04-06T01:02:40.431Z bec617ce sea [info] [    0.079226] Kernel panic - not syncing: Attempted to kill init! exitcode=0x00000100
2021-04-06T01:02:40.431Z bec617ce sea [info] Error: UnhandledIoError(Os { code: 2, kind: NotFound, message: "No such file or directory" })
2021-04-06T01:02:40.432Z bec617ce sea [info] [    0.080662]  dump_stack+0x5e/0x74
2021-04-06T01:02:40.432Z bec617ce sea [info] [    0.080406] Call Trace:
2021-04-06T01:02:40.432Z bec617ce sea [info] [    0.079895] CPU: 0 PID: 1 Comm: init Not tainted 5.9.15 #1
2021-04-06T01:02:40.433Z bec617ce sea [info] [    0.080991]  panic+0xf3/0x2a1
2021-04-06T01:02:40.433Z bec617ce sea [info] [    0.081616]  ? __vm_munmap+0x93/0xd0
2021-04-06T01:02:40.433Z bec617ce sea [info] [    0.081288]  do_exit+0x94d/0xb30
2021-04-06T01:02:40.434Z bec617ce sea [info] [    0.082304]  __x64_sys_exit_group+0x13/0x20
2021-04-06T01:02:40.434Z bec617ce sea [info] [    0.082712]  do_syscall_64+0x37/0x50
2021-04-06T01:02:40.434Z bec617ce sea [info] [    0.081968]  do_group_exit+0x3e/0xa0
2021-04-06T01:02:40.435Z bec617ce sea [info] [    0.083038]  entry_SYSCALL_64_after_hwframe+0x44/0xa9
2021-04-06T01:02:40.435Z bec617ce sea [info] [    0.083508] RIP: 0033:0x6c791b
2021-04-06T01:02:40.437Z bec617ce sea [info] [    0.083798] Code: eb ef 48 8b 76 28 e9 70 05 00 00 64 48 8b 04 25 00 00 00 00 48 8b b0 b0 00 00 00 e9 af ff ff ff 48 63 ff b8 e7 00 00 00 0f 05 <ba> 3c 00 00 00 48 89 d0 0f 05 eb f9 66 0f 1f 84 00 00 00 00 00 ba

In particular these seem like the relevant lines:

2021-04-06T01:02:40.427Z bec617ce sea [info] Running: `/bin/sh -c /usr/bin/unboundtest` as root
2021-04-06T01:02:40.431Z bec617ce sea [info] [    0.079226] Kernel panic - not syncing: Attempted to kill init! exitcode=0x00000100
2021-04-06T01:02:40.431Z bec617ce sea [info] Error: UnhandledIoError(Os { code: 2, kind: NotFound, message: "No such file or directory" })
2

Looks like your build process is picking up your Dockerfile and using that instead of the specified image. This sounds like a bug on our end.

Does j4cob/unboundtest and your Dockerfile result in the same image? If so, this should work as expected.

I wonder if /bin/sh is the binary not being found. That would be surprising. I’ll do some testing with your repo and try to reproduce.

1 Like
3

Yes, the j4cob/unboundtest on Dockerhub is built from the Dockerfile in the linked repo, and I built + pushed it with the current contents of the repo.

4

There’s an issue where the image in fly.toml isn’t always picked up. We’ll get that fixed!

1 Like
5

Huh, this is interesting. If I move the Dockerfile away I get:

$ flyctl deploy 
Deploying multi-dns
==> Validating app configuration
--> Validating app configuration done
Services
TCP 80/443 ⇢ 1232
Error app does not have a Dockerfile or buildpacks configured. See https://fly.io/docs/reference/configuration/#the-build-section

Maybe my fly.toml is misconfigured?

6

Ugh yes, this is a bug.

You should be able to run flyctl deploy -i j4cob/unboundtest as a workaround.

1 Like
7

Confirmed: this works to deploy the image. I still get the “no such file or directory” error, though.

One thing that’s strange to me: The error says /bin/sh -c unboundtest, but the CMD in my Dockerfile uses the fully-qualified path:

CMD /usr/bin/unboundtest
8

The command is expanded to this in the image config:
["/bin/sh", "-c", "#(nop) ", "CMD [\"/bin/sh\" \"-c\" \"unboundtest\"]"]

Does the image work locally?

9

The image does work locally. And I can confirm that running docker image inspect j4cob/unboundtest gives me:

...
        "ContainerConfig": {
...
            "Cmd": [
                "/bin/sh",
                "-c",
                "#(nop) ",
                "CMD [\"/bin/sh\" \"-c\" \"unboundtest\"]"
            ],
        "Config": {
...
            "Cmd": [
                "/bin/sh",
                "-c",
                "unboundtest"
            ],
...

I’m not sure what the difference between those sections is, or why neither of these uses the fully-qualified path (/usr/bin/unbound).

10

I see that there are three forms for the CMD directive: Dockerfile reference | Docker Docs

I’ll try the preferred form.

11

I was just going to suggest that. This is the first time a CMD issue has come up like this but I can’t tell what’s different for your image.

12

Now I get:

2021-04-06T23:18:23.306Z 1d03d6fa sea [info] Running: `/usr/bin/unboundtest` as root
2021-04-06T23:18:23.308Z 1d03d6fa sea [info] Error: UnhandledIoError(Os { code: 2, kind: NotFound, message: "No such file or directory" })

Which is an improvement (right command) but still an error. :slight_smile:

13

That’s really strange. We’re doing some testing with your image to see what’s up. Are you still using j4cob/unboundtest:latest, or has your fly.toml changed at all?

1 Like
14

fyi flyctl is correctly picking up image in fly.toml now

1 Like
15

We’re still working on this. This image is behaving very strangely on our setup. We’ll keep you updated on our progress, but we will figure it out soon!

1 Like
16

Thanks for the updates! I’ll be curious to hear what it is. :slight_smile:

17

Hello,

I’m getting the same error but I specify the image in the Dockefile:

FROM redis:alpine

# Install the Doppler CLI
RUN (curl -Ls https://cli.doppler.com/install.sh || wget -qO- https://cli.doppler.com/install.sh) | sh

ADD start-redis-server.sh /usr/bin/
RUN chmod +x /usr/bin/start-redis-server.sh

ENTRYPOINT ["doppler", "run", "--"]
CMD ["start-redis-server.sh"]

Everything else matches the stuff in the repo here: GitHub - fly-apps/redis: Launch a Redis server on Fly

20

@nahtnam this is probably a bug in our handling of things like -- in shell commands.

You might be able to work around this by creating a entrypoint.sh file:

exec doppler run -- "$@"

You’ll have to add it to your container and mark it executable like we’re doing with the start-redis-server script.

Then change the Dockerfile to:

ENTRYPOINT entrypoint.sh
21

@kurt Hmm, then I get the error doppler not found even though I installed it. Seems that doppler only works if I run it from Dockerfile. Maybe I need to add it to the path

22

@nahtnam is the doppler binary statically linked? If it relies on glibc, that’s not available in alpine-based Docker images. If you can, does this work in a local Docker?

@jsha I fixed this. Your image should now work as-is.

We “overmount” /run on boot as a tmpfs. Docker allows you to write to /var/run and /run and puts that in the rootfs somehow.

Now if we detect there’s already a file descriptor for /run, we won’t overmount.

1 Like