How to run docker *inside* a sprite

General
1

This post is a follow up to the great thread: How to get Docker running on Sprites?

Disclaimer: I do understand the post below. I checked it. But (obviously :slight_smile: I didn’t write it. It’s written by Claude-AI. But we also tested this setup. I am using this as we speak.

Remark by the “human-in-the-loop” old-school dev: Yes this feels like docker inside docker inside docker inside an unbelievable awesome genius fly io custom docker without docker in side a docker … :slight_smile:

So yes it’s maybe an unusual, complex setup. But some people might want to run their “golden-tarball” (how a docker-image was called 20 years ago) before going to PROD…

Running containers (not just docker ps) inside a Sprite

Installing Docker and starting dockerd by hand works, as shown above. But on my Sprite (Ubuntu 26.04, kernel 6.12-fly),
the first docker run failed in two ways:

  1. cgroups: runc create failed: unable to apply cgroup configuration: cannot enter cgroupv2 "/sys/fs/cgroup/docker" with domain controllers -- it is in an invalid state. Inside the Sprite, /sys/fs/cgroup is a namespaced
    threaded cgroup that already contains the Sprite’s own processes, so no child cgroup can get the memory/io
    controllers. Moving processes into a leaf cgroup (the usual docker-in-docker trick) fails with
    Operation not supported.
  2. Storage: dmesg shows overlayfs: maximum fs stacking depth exceeded. The Sprite’s / is already an overlay.
    Docker 29 defaults to the containerd image store, which lives in /var/lib/containerd on that overlay.

What fixes both:

  • Sprites already mount a dedicated ext4 volume at /var/lib/docker (check with findmnt /var/lib/docker).
    Use the classic overlay2 driver so everything lives on that volume.
  • Run containers with crun with its cgroup manager disabled.

Setup

sudo apt-get update
sudo apt-get install -y docker.io docker-compose-v2 crun
sudo usermod -aG docker "$(whoami)"     # $USER is empty in a Sprite shell

/etc/docker/daemon.json:

{
  "features": { "containerd-snapshotter": false },
  "storage-driver": "overlay2",
  "default-runtime": "crun-nocg",
  "runtimes": { "crun-nocg": { "path": "/usr/bin/crun", "runtimeArgs": ["--cgroup-manager=disabled"] } },
  "bridge": "none",
  "iptables": false,
  "ip6tables": false,
  "live-restore": true,
  "log-driver": "local",
  "log-opts": { "max-size": "10m", "max-file": "3" }
}

bridge: none and iptables: false are optional: I only use --network host, so Docker doesn’t need to touch
iptables or the forwarding sysctl (which is off on a Sprite).

There’s no systemd, so run dockerd as a Sprite service. It then starts at boot and restarts if it crashes:

sprite-env services create docker --cmd sudo --args /usr/bin/dockerd
sudo docker info | grep -E 'Storage Driver|Backing Filesystem|Default Runtime'
#  Storage Driver: overlay2 / Backing Filesystem: extfs / Default Runtime: crun-nocg

If a web server in front needs the containers, let it depend on that service, e.g.
sprite-env services create web --cmd /usr/sbin/nginx --needs docker --http-port 8080.

Test

sudo docker run --rm --network host hello-world

A locked-down container works too: --read-only --tmpfs /tmp --cap-drop ALL --security-opt no-new-privileges:true --user 1001:1001. Inside it, /proc/self/status shows CapEff: 0, NoNewPrivs: 1, Seccomp: 2. Outbound IPv6
works from a --network host container. --restart unless-stopped works too: I killed the container’s process and
Docker started it again.

Caveats

  • No resource limits. With cgroups disabled, --memory / --pids-limit are accepted but not enforced.
    Limit the app itself, e.g. a fixed -Xmx for the JVM.
  • Checkpoints only capture the Sprite’s writable overlay. /var/lib/docker is a separate volume, so don’t rely
    on a checkpoint restore to roll back images; pull them again.
  • --network host only. Without a bridge, there’s no port publishing (-p).
  • Cold wake: [result pending]
  • Docker Hub’s anonymous pull limits apply as usual.

Tested 2026-11-04 with Docker 29.1.3 (Ubuntu docker.io), containerd 2.2.2 and crun 1.21.