fly.io /init docs part 2

[anatoli_devops]

The continuation of fly.io /init override documentation (I didn’t open computer for 7 days and the topic got closed).

The Pilot init process is the one you get if you specify you want to use containers. Container processes are run as pid 1.

@rubys how can I specify I want container? I already have this in my fly.toml

[build]
  image = 'lscr.io/linuxserver/wireguard'

What business or software problem do you have?

@halfer the s6-overlay based image fails to run, because its /init process wants PID 1 (s6-overlay-suexec: fatal: can only run as pid 1). I need to find a secure workaround that won’t require maintaining and updating my own image. To validate the security, I need documentation on fly /init process interaction to understand how it works, what it does, and what attack vectors I need to take care of.

[halfer]

For your first question, read this. I think that container control hasn’t landed in the TOML file format yet, so an API call with a JSON blob is required. I assume this will be added in the future. (I am guessing that containers will run inside a Firecracker VM, so Fly’s PID1 still runs, but then it runs Docker-in-Firecracker, which lets you have your own PID1 process).

For your second question, I think you’re focussing too much on the mechanics, and not enough on the problem. Containers would probably solve this, so that’s worth a go. Also, you could run s6 as a separate small machine (i.e. not use s6-overlay). Finally, if you want s6 to run in the same machine as your main process (e.g. a web server) then use S6 to start up other things. S6 doesn’t need to be PID1 to do that.

[mayailurus]

@halfer

I think that container control hasn’t landed in the TOML file format yet, so an API call with a JSON blob is required.

It does work in fly.toml, actually, although there are some rough edges that might cause a person to conclude otherwise, .

Here is a full example of a user process running as PID 1…

app = "multiverse"
primary_region = "ewr"
swap_size_mb = 512

machine_config = "solo-config.json"

[build]
  image = "debian:bookworm"

And the JSON file:

{
  "containers": [
    {"name": "c1",
     "image": "debian:bookworm",
     "cmd": ["sleep", "inf"]}
  ]
}

Then…

$ fly app create multiverse
$ fly config validate --strict
$ fly deploy --ha=false --depot=false
$ fly ssh console
# od -c /proc/1/cmdline 
0000000   s   l   e   e   p  \0   i   n   f  \0
0000012

I.e., PID 1 is the cmd from solo-config.json.

---

Aside: The build.image knob is mandatory, but it’s not clear what it really changes at run time…

[halfer]

Oh, super. I looked for a container key in the docs, and came up with nowt. It’s that pesky machine_config I was looking for!

[anatoli_devops]

The only page that talks about containers is the one below. It also says “requires using Pilot as the init system”, but there is again, no page explaining /init and Pilot.

Multi-container Machines

I just guess it is just because everything is experimental. Including the necessity to have one more config file.

[anatoli_devops]

The solution with machine_config works fine. Not sure why container name is needed, or the duplicate build:image section. Maybe it will be removed when the feature is less experimental.

[mayailurus]

@anatoli_devops

or the duplicate build:image section.

Ah… I overstated this one above, actually. (Sorry, guys!) Re-reading it, @rubys did say in the announcement thread that there is an intentional mechanism to replace one container’s image. So, it’s the containers.image key in the JSON portion that’s the duplicate, which was the opposite of my early intuition.

app = "multiverse"
primary_region = "ewr"
swap_size_mb = 512

machine_config = '''{"containers": [{"name": "c1", "cmd": ["sleep", "inf"]}]}'''

[build]
  image = "debian:bookworm"

(Although I do belatedly see the logic, .)

This is still a little complicated, in the cases where a person just wants PID 1 and not multiple containers, but I imagine that will indeed get planed out as time goes on…

[rubys]

I re-retired a few months back (but remain an enthusiastic fly.io user), so any planning out will be done by others unless a case can be made that it directly benefits my usage. And I’m not qualified to authoritatively evaluate the security implications of using s6 in this context.

On a more constructive note, if the intended usage of s6 is merely a means to an end, there may be other ways to accomplish the same goals. For my purposes, I developed (with the help of Claude) navigator which can be used on fly.io, on hetzner with kamal, and your own hardware. It does what I need and more, and I’m willing to collaborate with others.

Getting started is as easy as a two line modification to your Dockerfile and adding a configuration file.

[mayailurus]

@rubys

I re-retired a few months back (but remain an enthusiastic fly.io user), so any planning out will be done by others […]

Yeah, that’s totally fair. I didn’t mean to put anyone in particular on the spot, in any way in the slightest. Thanks for all you did for this!