Container (non-root) user can't write to /dev/stdout or /dev/stderr

8

Could this be an issue with Fly’s init? To collect and summarise the known information:

  • The code in the public init-snapshot repo clearly tries to set the ownership of the pipes to the app user (credit to @ignoramus for the find).

  • However the pipes are still owned by root (as demonstrated by @Alex21).
    Minimal Dockerfile combining the relevant commands:

    FROM ubuntu:latest
USER www-data
CMD whoami; ls -l /dev/stdout /proc/self/fd/1; stat -L /proc/self/fd/1; echo "test" >>/dev/stdout

    Logs:

    www-data
lrwxrwxrwx 1 root     root     15 Jan 27 19:00 /dev/stdout -> /proc/self/fd/1
l-wx------ 1 www-data www-data 64 Jan 27 19:00 /proc/self/fd/1 -> pipe:[4403]
  File: /proc/self/fd/1
  Size: 0          Blocks: 0          IO Block: 4096   fifo
Device: ch/12d     Inode: 4403        Links: 1
Access: (0600/prw-------)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2023-01-27 19:00:15.069441815 +0000
Modify: 2023-01-27 19:00:15.085441814 +0000
Change: 2023-01-27 19:00:15.085441814 +0000
 Birth: -
/bin/sh: 1: cannot create /dev/stdout: Permission denied

    The problem is with the ownership of pipe:[4403].
    The line “Access: (0600/prw-------) Uid: ( 0/ root) Gid: ( 0/ root)”
    should be “Access: (0600/prw-------) Uid: ( 33/www-data) Gid: ( 33/www-data)”.

    (Aside: stat -L /proc/1/fd/* shows that all the pipes of PID 1 are also owned by root.)

I don’t know why the pipes have the wrong ownership.

However I can explain a small part of the weirdness, specifically why writing to standard out succeeds but redirecting to /dev/stdout fails (as pointed out by @Alex21):

The kernel does not perform permission checks on file descriptors inherited by a process, but it does perform permission checks when a process tries to re-open those file descriptors using a path such as /dev/stdout or /proc/self/fd/1.

Here is a demonstration using regular files instead of pipes. (Edit: explanation below)

# whoami
root
# touch /tmp/out; ls -l /tmp/out
-rw-r--r-- 1 root root 0 Jan 27 19:03 /tmp/out
# cmd='whoami; ls -l /dev/stdout /proc/self/fd/1; echo "test" >>/dev/stdout'
# su -s /bin/sh -c "$cmd" www-data >/tmp/out
sh: 1: cannot create /dev/stdout: Permission denied
# cat /tmp/out
www-data
lrwxrwxrwx 1 root     root     15 Jan 27 19:02 /dev/stdout -> /proc/self/fd/1
l-wx------ 1 www-data www-data 64 Jan 27 19:03 /proc/self/fd/1 -> /tmp/out

Maybe this is why this issue has been so elusive – most apps just log to standard output/error, so they work even though the pipes have the wrong ownership. It’s only when an app tries to use /dev/stdout that the wrong ownership becomes apparent.

Temporary workaround:

Inspired by Alex21’s excellent cat trick, here is a more transparent workaround – set ENTRYPOINT in the Dockerfile (or in fly.toml) as follows:

ENTRYPOINT ["/bin/sh", "-c", "\"$@\" 2>&1 | cat", "/bin/sh"]

And here is a more elaborate version that keeps stdout and stderr separate:

ENTRYPOINT ["/bin/sh", "-c", "mkfifo /tmp/stdout /tmp/stderr; cat /tmp/stdout >&1 & cat /tmp/stderr >&2 & exec \"$@\" >/tmp/stdout 2>/tmp/stderr", "/bin/sh"]

(The named pipes in the second version could be removed/avoided using some file descriptor acrobatics. The second version only spawns two long-lived processes, both cat, which I think is the bare minimum if stdout and stderr are to be separate. The first version only spawns one cat process but its sh process is also long-lived. If the image had an ENTRYPOINT that needs to be preserved, it should be manually inserted before the \"$@\".)

1 Like