Secrets Not Injecting as Env Vars for Multi-container `machine_config` Setup

1

TL;DR: I can’t get secrets that I set to be available in running containers as environment variables, even after the machine in the app is redeployed.

I’ve checked multiple existing help articles on this, and they’re all saying the same things that I’d tried (thread, thread).

I have the secrets set:

$ fly config env
Secrets
 NAME                    │ DIGEST
 TELEGRAM_WEBHOOK_URL    │ c8d6e7dc8446536d
 TELEGRAM_WEBHOOK_SECRET │ d32f87ccee2633e5

Environment Variables
 NAME │ VALUE

$ fly secrets list
 NAME                    │ DIGEST           │ STATUS
 TELEGRAM_WEBHOOK_URL    │ c8d6e7dc8446536d │ Deployed
 TELEGRAM_WEBHOOK_SECRET │ d32f87ccee2633e5 │ Deployed

Yet when SSH’d into the machine at runtime and viewing, the variables aren’t there:

FLY_PRIVATE_IP=redacted
FLY_MACHINE_ID=redacted
npm_config_install_links=false
PYTHONUNBUFFERED=1
FLY_ALLOC_ID=redacted
FLY_VM_MEMORY_MB=2048
PWD=/
HERMES_WEB_DIST=/opt/hermes/hermes_cli/web_dist
FLY_APP_NAME=redacted
HOME=/root
PLAYWRIGHT_BROWSERS_PATH=/opt/hermes/.playwright
FLY_PROCESS_GROUP=app
FLY_MACHINE_VERSION=redacted
FLY_REGION=lax
SHLVL=1
PRIMARY_REGION=lax
HERMES_TUI_DIR=/opt/hermes/ui-tui
FLY_IMAGE_REF=docker-hub-mirror.fly.io/nousresearch/hermes-agent:latest
PATH=/opt/hermes/bin:/opt/hermes/.venv/bin:/opt/data/.local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
HERMES_HOME=/opt/data
_=/usr/bin/env

I’ve tried referencing all the documentation I can (here, here, here), but everything I’ve tried fails to work for me.

My fly.toml looks like:

app = "redacted"
primary_region = "lax"

[build]
  image = "nousresearch/hermes-agent:latest"

[[services]]
  internal_port = 8443
  protocol = "tcp"
  auto_stop_machines = "suspend"
  auto_start_machines = true

  [[services.ports]]
    handlers = ["tls", "http"]
    port = 443

[[mounts]]
  source = "data"
  destination = "/opt/data"
  scheduled_snapshots = false

[[vm]]
  memory = "2gb"
  cpus = 1

Just very confused on where I’m going wrong here; I’m trying to move my Hermes agent from a cloud VM on Hetzner to Fly to take advantage of Fly’s scale-to-zero. Followed Hermes’ Telegram Webhook mode docs as well as Fly’s own docs on Hermes, but have had to modify a bit as Fly’s docs don’t account for auto-stop/auto-start (auto-suspend in my case). Any help would be greatly appreciated here. Thanks!

2

Hm… The fly.toml in the official docs that you linked to includes a machine_config, which has a non-obvious filtering side effect when it comes to secrets, if I remember correctly.

Perhaps your own config specified that originally and it stuck (loosely speaking) to the Machine?

I would try creating a simpler, fresh VM at this point and see if it picks them up…

$ fly secrets set CANARY=chirp --stage
$ fly m run debian:trixie --shell
# printenv CANARY
chirp
# exit

[If that still fails, then repeat with LOG_LEVEL=debug set in your local shell (on your development laptop/desktop). That will allow you to see the min_secrets_version, etc.]

3

Very interesting, thanks for the quick reply.

I tried an adapted form of your reply and it worked just fine:

fly m run nousresearch/hermes-agent:latest --shell --memory=2048mb

and the env vars were present.

:white_check_mark: Solution

After more finicking, I found that it indeed was something with the config.

You were spot on, the tutorial I followed has a machine_config, but I was trying to resolve some issues by going without. I had to reinstate the machine_config to be able to run a specific process as PID 1 though.

Eventually I found the container config reference docs for how to write the secrets section. It’s not super clear from the docs, but it’s required to have a secrets section to pull in your app’s secrets.

fly.toml:

app = "redacted"
primary_region = "lax"
machine_config = "machine_config.json"

[build]
  image = "nousresearch/hermes-agent:latest"

[[services]]
  internal_port = 8443
  protocol = "tcp"
  auto_stop_machines = "suspend"
  auto_start_machines = true

  [[services.ports]]
    handlers = ["tls", "http"]
    port = 443

[[mounts]]
  source = "data"
  destination = "/opt/data"
  scheduled_snapshots = false

[[vm]]
  memory = "2gb"
  cpus = 1

where machine_config.json is:

{
  "containers": [
    {
      "name": "hermes",
      "image": "nousresearch/hermes-agent:latest",
      "cmd": ["gateway", "run"],
      "secrets": [
        {"env_var": "TELEGRAM_WEBHOOK_SECRET"},
        {"env_var": "TELEGRAM_WEBHOOK_URL"}
      ]
    }
  ]
}

worked great, and the env vars were available at runtime.

Thanks for your debugging help @mayailurus!