Multi-container Machines: sidecar container execution & fly deploy image replacement workflow

Questions / Help
,
1

Hi everyone, I’m testing the new Multi-container Machines Multi-container Machines · Fly Docs feature using machine_config in fly.json to run an app container alongside a cloudflare/mesh:latest sidecar.

I’m encountering two main issues regarding sidecar execution and image deployment workflow:

──────

1. Sidecar (cloudflare/mesh:latest) process execution & SSH inspection

  • Behavior: The logs only show [info] cloudflare-mesh container started, but no application logs are outputted by the sidecar process.
  • SSH Inspection: When SSHing into the machine specifying the container (fly ssh console --app=cf-proxy --container=cloudflare-mesh --pty -C "busybox sh"), I cannot locate the container’s filesystem or its default /entrypoint binary.
  • What is the expected way to inspect an individual container’s rootfs when SSHing into a multi-container machine?
    ──────

2. fly deploy image resolution & container replacement workflow

  • If I specify "image": "registry.fly.io/cf-proxy:latest" in containers.json, deployment fails because the :latest tag is not found
    in the registry.
  • Currently, the only workaround is:
    1. Trigger a build/deploy to generate an image tag.
    2. Copy the resulting image tag (e.g. registry.fly.io/cf-proxy:deployment-01M0V1AYKRAMBF8S4PAXA2FVRZ).
    3. Paste it directly into containers.json.
    4. Run fly deploy again.

What is the correct way of using the latest build of the referenced Dockerfile ?
──────

Configuration Files

fly.json

{
  "$schema": "https://www.schemastore.org/fly.json",
  "app": "cf-proxy",
  "build": { "dockerfile": "Dockerfile" },
  "deploy": { "strategy": "rolling" },
  "experimental": {},
  "machine_config": "crates/cf-proxy/containers.json",
  "primary_region": "iad"
}

crates/cf-proxy/containers.json

{
  "$schema": "../../../../lib/shared/schema/machine-config.schema.json",
  "containers": [
    {
      "env": { "SRCNAT_ENABLED": "true" },
      "image": "cloudflare/mesh:latest",
      "cmd": [],
      "name": "cloudflare-mesh",
      "restart": { "policy": "no" },
      "secrets": [{ "env_var": "MESH_NODE_TOKEN", "name": "MESH_NODE_TOKEN" }]
    },
    {
      "depends_on": [{ "name": "cloudflare-mesh" }],
      "env": { "PORT": "8080" },
      "image": "registry.fly.io/cf-proxy:deployment-01M0V1AYKRAMBF8S4PAXA2FVRZ",
      "name": "cf-proxy",
      "restart": { "policy": "no" },
      "secrets": [{ "env_var": "MESH_NODE_TOKEN", "name": "MESH_NODE_TOKEN" }]
    }
  ],
  "guest": {
    "cpu_kind": "shared",
    "kernel_args": [
      "net.ipv4.ip_forward=1",
      "net.ipv6.conf.all.forwarding=1",
      "net.ipv6.conf.default.forwarding=1"
    ],
    "cpus": 1,
    "memory_mb": 512,
    "persist_rootfs": "always"
  }
}

(I can provide the Dockerfile as well if needed).

Any insights into the expected config for Pilot multi-container lifecycle and fly deploy container mapping would be greatly appreciated!

2

Hi… The multi-container Machines are fun, :red_rock_dwelling:, but have some rough edges still, and builds are definitely one of those rough edges.

The initial design stab was to have fly deploy automatically substitute the Dockerfile build for the image of exactly one container. Specifically, the container named app would be automatically overridden, or, if there was none, the first in the containers array.

I.e., probably what you want to do is rename cf-proxy to app.

-    "name": "cf-proxy",
+    "name": "app",

(In containers.json.)

(Also, this is the literal string "app" and not a meta-variable.)

You can use fly m status -d to verify that each container has the expected image.

fly ssh console --container=<name> works for me, although I use an older version of flyctl (for weird reasons). (They were tweaking flyctl’s handling of containers recently, too, so things might really have changed.)

My guess is that the cloudflare-mesh container’s image was replaced (due to being first on the list) with the Dockerfile build that you intended for cf-proxy, which definitely would have been disorienting when viewed from within the SSH session…

Hope this helps!

3

thank you !! That solved.