Fly app secrets missing from multi-container Compose Machine

Hey there,

I’m trying to deploy a multi-container application (zitadel idP) using the Compose support in fly.toml:

app = "silent-meadow-4821"
primary_region = "fra"

[build.compose]
  file = "compose.fly.yml"

[env]
  AUDIENCE = "https://silent-meadow-4821.fly.dev"
  # other non-secret variables...

The deployment contains several containers (ZITADEL, ZITADEL Login, a proxy, and a small Alpine test container).

I’m seeing an issue where normal variables from [env] are available at runtime, but secrets configured with fly secrets are missing entirely.

Secrets

The application has secrets such as:

ZITADEL_DATABASE_POSTGRES_DSN
ZITADEL_MASTERKEY
ZITADEL_FIRSTINSTANCE_ORG_HUMAN_PASSWORD
SYSTEM_USER_PRIVATE_KEY
ZITADEL_SYSTEMAPIUSERS

fly secrets list -a silent-meadow-4821 reports the secrets as deployed.

Minimal test container

To rule out ZITADEL itself, I added this container to compose.fly.yml:

services:
  secrets-test:
    image: alpine:3.22
    restart: "no"
    command:
      - /bin/sh
      - -c
      - |
        for name in \
          ZITADEL_DATABASE_POSTGRES_DSN \
          ZITADEL_MASTERKEY \
          ZITADEL_FIRSTINSTANCE_ORG_HUMAN_PASSWORD \
          SYSTEM_USER_PRIVATE_KEY \
          ZITADEL_SYSTEMAPIUSERS \
          AUDIENCE
        do
          value="$(printenv "$name" 2>/dev/null || true)"
          if [ -n "$value" ]; then
            printf 'SECRET_CHECK %s=present length=%s\n' "$name" "${#value}"
          else
            printf 'SECRET_CHECK %s=missing\n' "$name"
          fi
        done

The result is:

SECRET_CHECK ZITADEL_DATABASE_POSTGRES_DSN=missing
SECRET_CHECK ZITADEL_MASTERKEY=missing
SECRET_CHECK ZITADEL_FIRSTINSTANCE_ORG_HUMAN_PASSWORD=missing
SECRET_CHECK SYSTEM_USER_PRIVATE_KEY=missing
SECRET_CHECK ZITADEL_SYSTEMAPIUSERS=missing
SECRET_CHECK AUDIENCE=present length=37

AUDIENCE comes from [env] in fly.toml, so regular environment variables are reaching the Machine/container correctly.

Tested outside the individual containers

I also connected directly to the underlying Machine:

fly ssh console -a silent-meadow-4821 --no-container

Then:

printenv AUDIENCE

returns:

https://silent-meadow-4821.fly.dev

while:

printenv ZITADEL_DATABASE_POSTGRES_DSN

returns nothing.

So this doesn’t appear to be specific to the ZITADEL container or the Alpine test container. The secret is already absent from the underlying Machine runtime environment.

Generated Machine configuration

The generated Machine configuration contains the expected normal environment variables:

"env": {
  "AUDIENCE": "https://silent-meadow-4821.fly.dev",
  "ZITADEL_EXTERNALDOMAIN": "silent-meadow-4821.fly.dev",
  ...
}

and the Compose services are represented under:

"containers": [
  {
    "name": "zitadel-login",
    ...
  },
  {
    "name": "secrets-test",
    "image": "alpine:3.22",
    ...
  },
  {
    "name": "zitadel-init",
    ...
  },
  {
    "name": "zitadel-setup",
    ...
  },
  {
    "name": "zitadel-api",
    ...
  }
]

I don’t see ignore_app_secrets: true in the Machine configuration.

The Machine metadata reports:

"fly_flyctl_version": "0.4.102",
"fly_platform_version": "v2"

Expected behavior

My understanding from the Fly multi-container documentation is that secrets configured using fly secrets are global and should be available to all containers in a multi-container Machine.

I would therefore expect:

fly ssh console -a silent-meadow-4821 --no-container
printenv ZITADEL_DATABASE_POSTGRES_DSN

to return the configured secret, just as AUDIENCE is available.

Instead, [env] variables are present while all fly secrets variables are absent.

Question

Is there anything additional required to make app-level fly secrets available when using:

[build.compose]
file = "compose.fly.yml"

with a multi-container Machine?

Or could this be an issue with secret injection for Compose-based multi-container deployments?

I’m happy to provide the full sanitized fly.toml, compose.fly.yml, Machine configuration, secret deployment status, or additional diagnostics if useful.

Hi… Last I checked, there was indeed a deviation from the documented behavior when it comes to secrets in multi-container Machines. This older thread might help:

https://community.fly.io/t/flyctl-secrets-appear-to-be-totally-ignored-when-using-docker-compose/27658

Generally speaking, the alternative machine_config style has fewer rough edges (although still not zero), if you can use that instead…

Hey,

thanks for pin pointing the existing thread! i will have a look into it asap.

kind regards