Fly.toml config not being used

Hi,

This might be a dumb question, but I can’t work out what I’ve done wrong. I have tried deleting/creating new apps a few times, but keep having the same problem.

In short: how do I change the config of an app? In my case I am trying to add a mount, but none of the fields in the fly.toml file seem to be used if I make changes.

fly config validate tells me the config is valid (I assume this is referring to the local fly.config)
fly config display displays a generic config that does not contain my local changes to fly.toml
fly deploy deploys successfully, but it clearly isn’t using my fly.toml

In case it helps:

  • This is a Phoenix app, but pre-1.6.0. I’ve made changes that get it running, but I can’t add a mount to the config.
  • I originally created the app using fly launch, but I since deleted that and tried creating using fly apps create in case that was the problem.

I think I’ve read the docs, and they don’t seem to specify how to force changes.

Thanks
Zander

Can you paste your fly.toml here? Also, did you create any volumes before setting the mounts?

Sure, here’s the fly.toml:

# fly.toml file generated for trips on 2021-12-31T15:49:38Z

app = "dark-breeze-4631f"

kill_signal = "SIGTERM"
kill_timeout = 5
processes = []

[env]
  PHX_HOST = "dark-breeze-4631.fly.dev"
  PORT = "8080"

[experimental]
  allowed_public_ports = []
  auto_rollback = true

[mounts]
  destination = "/database"
  source = "trips"

[[services]]
  http_checks = []
  internal_port = 8080
  processes = ["app"]
  protocol = "tcp"
  script_checks = []

  [services.concurrency]
    hard_limit = 25
    soft_limit = 20
    type = "connections"

  [[services.ports]]
    handlers = ["http"]
    port = 80

  [[services.ports]]
    handlers = ["tls", "http"]
    port = 443

And my volume is created. I can’t recall exactly what point I created it, but it’s not possible to create a volume before an app, so I guess it must have been after?

ID                   NAME  SIZE REGION ZONE ATTACHED VM CREATED AT
vol_18l524y7jkp47zmp trips 1GB  lhr    34c3             20 minutes ago

I also found this Github issue, which seems similar but was closed 6 months ago:

Thanks. Which verison of flyctl are you using?

Can you post the output of fly volumes list and fly regions list, and the full output of your deploy command? If you have a mounts section, deployment should fail if it cannot find a suitable volume.

Sure, here you go:

~/Projects/trips ❯❯❯ flyctl version                                                                                                                                        ✘ 1
flyctl v0.0.273 darwin/arm64 Commit: 05b5691 BuildDate: 2021-12-31T14:29:07Z
~/Projects/trips ❯❯❯ flyctl volumes list
fID                   NAME  SIZE REGION ZONE ATTACHED VM CREATED AT
vol_18l524y7jkp47zmp trips 1GB  lhr    34c3             32 minutes ago
~/Projects/trips ❯❯❯ flyctl regions list
Region Pool:
lhr
Backup Region:

And here is the config that the app seems to be using (not what is in fly.toml):

~/Projects/trips ❯❯❯ flyctl config display
{
    "env": {},
    "experimental": {
        "allowed_public_ports": [],
        "auto_rollback": true,
        "cmd": [],
        "entrypoint": [],
        "exec": [],
        "private_network": true
    },
    "kill_signal": "SIGINT",
    "kill_timeout": 5,
    "processes": [],
    "services": [
        {
            "concurrency": {
                "hard_limit": 25,
                "soft_limit": 20,
                "type": "connections"
            },
            "http_checks": [],
            "internal_port": 8080,
            "ports": [
                {
                    "handlers": [
                        "http"
                    ],
                    "port": 80
                },
                {
                    "handlers": [
                        "tls",
                        "http"
                    ],
                    "port": 443
                }
            ],
            "processes": [
                "app"
            ],
            "protocol": "tcp",
            "script_checks": [],
            "tcp_checks": [
                {
                    "grace_period": "1s",
                    "interval": "15s",
                    "restart_limit": 0,
                    "timeout": "2s"
                }
            ]
        }
    ]
}

Thanks. Can you paste the output of fly deploy?

I just tried it a few minutes ago, so I have it here. This time I tried passing in the config file (although the name matches the default so I don’t think it did anything:

~/Projects/trips ❯❯❯ fly deploy -c fly.toml
==> Verifying app config
--> Verified app config
==> Building image
Remote builder fly-builder-summer-brook-7121 ready
==> Creating build context
--> Creating build context done
==> Building image with Docker
--> docker host: 20.10.12 linux x86_64
[+] Building 223.6s (0/1)
[+] Building 4.7s (30/30) FINISHED
 => [internal] load remote build context                                                                                                                                   0.0s
 => copy /context /                                                                                                                                                        2.3s
 => [internal] load metadata for docker.io/library/debian:bullseye-20210902-slim                                                                                           0.9s
 => [internal] load metadata for docker.io/hexpm/elixir:1.13.1-erlang-24.2-debian-bullseye-20210902-slim                                                                   0.9s
 => [builder  1/18] FROM docker.io/hexpm/elixir:1.13.1-erlang-24.2-debian-bullseye-20210902-slim@sha256:ba2e50ad59370fab4a63a3751dc867ee45074e5f5201d8b612706d07e6e3e8ba   0.0s
 => [stage-1 1/7] FROM docker.io/library/debian:bullseye-20210902-slim@sha256:e3ed4be20c22a1358020358331d177aa2860632f25b21681d79204ace20455a6                             0.0s
 => CACHED [stage-1 2/7] RUN apt-get update -y && apt-get install -y libstdc++6 openssl libncurses5 locales sqlite3 htop     && apt-get clean && rm -f /var/lib/apt/lists  0.0s
 => CACHED [stage-1 3/7] RUN sed -i '/en_US.UTF-8/s/^# //g' /etc/locale.gen && locale-gen                                                                                  0.0s
 => CACHED [stage-1 4/7] WORKDIR /app                                                                                                                                      0.0s
 => CACHED [builder  2/18] RUN apt-get update -y && apt-get install -y build-essential git npm     && apt-get clean && rm -f /var/lib/apt/lists/*_*                        0.0s
 => CACHED [builder  3/18] WORKDIR /app                                                                                                                                    0.0s
 => CACHED [builder  4/18] RUN mix local.hex --force &&     mix local.rebar --force                                                                                        0.0s
 => CACHED [builder  5/18] COPY mix.exs mix.lock ./                                                                                                                        0.0s
 => CACHED [builder  6/18] RUN mix deps.get --only prod                                                                                                                    0.0s
 => CACHED [builder  7/18] RUN mkdir config                                                                                                                                0.0s
 => CACHED [builder  8/18] COPY config/config.exs config/prod.exs config/                                                                                                  0.0s
 => CACHED [builder  9/18] RUN mix deps.compile                                                                                                                            0.0s
 => CACHED [builder 10/18] COPY priv priv                                                                                                                                  0.0s
 => CACHED [builder 11/18] COPY assets assets                                                                                                                              0.0s
 => CACHED [builder 12/18] RUN cd assets && npm install && ./node_modules/brunch/bin/brunch build -p                                                                       0.0s
 => CACHED [builder 13/18] RUN mix phx.digest                                                                                                                              0.0s
 => CACHED [builder 14/18] COPY lib lib                                                                                                                                    0.0s
 => CACHED [builder 15/18] RUN mix compile                                                                                                                                 0.0s
 => CACHED [builder 16/18] COPY config/runtime.exs config/                                                                                                                 0.0s
 => CACHED [builder 17/18] COPY rel rel                                                                                                                                    0.0s
 => CACHED [builder 18/18] RUN mix release                                                                                                                                 0.0s
 => CACHED [stage-1 5/7] COPY --from=builder /app/_build/prod/rel ./                                                                                                       0.0s
 => CACHED [stage-1 6/7] COPY trips_dump.sql trips_dump.sql                                                                                                                0.0s
 => CACHED [stage-1 7/7] RUN set -eux;     ln -nfs /app/$(basename *)/bin/$(basename *) /app/entry                                                                         0.0s
 => exporting to image                                                                                                                                                     0.0s
 => => exporting layers                                                                                                                                                    0.0s
 => => writing image sha256:3ccdfb991c1b0a30bf2c39340650a7aa7817b493facb9542f92b902a25fd003d                                                                               0.0s
 => => naming to registry.fly.io/dark-breeze-4631:deployment-1640967122                                                                                                    0.0s
--> Building image done
==> Pushing image to fly
The push refers to repository [registry.fly.io/dark-breeze-4631]
975dbf6bf485: Layer already exists
5892b6a0d2ce: Layer already exists
c5637069476c: Layer already exists
85dfabae1706: Layer already exists
8308665c3924: Layer already exists
7229b49946ed: Layer already exists
ba5a5fe43301: Layer already exists
deployment-1640967122: digest: sha256:3472e1d20c3b1e5731dfcc40784ef7a0015fcc473d4f30b649389ffe323e36f5 size: 1786
--> Pushing image done
image: registry.fly.io/dark-breeze-4631:deployment-1640967122
image size: 133 MB
==> Creating release
--> release v2 created

--> You can detach the terminal anytime without stopping the deployment
==> Monitoring deployment

 1 desired, 1 placed, 1 healthy, 0 unhealthy
--> v2 deployed successfully


I’ve been noticing the same thing this morning. and possibly last night with env vars. neither the env vars nor health checks from my config file are showing up when I run fly config --verbose --config fly/web.toml display after a deploy. In my situation I am using no volumes nor the mounts option.

Thanks. Can you try without specifying -c fly.toml?

Done, doesn’t seem to have been any change. Thanks for looking into this.

~/Projects/trips ❯❯❯ fly deploy
==> Verifying app config
--> Verified app config
==> Building image
Remote builder fly-builder-summer-brook-7121 ready
==> Creating build context
--> Creating build context done
==> Building image with Docker
--> docker host: 20.10.12 linux x86_64
[+] Building 226.6s (0/1)
[+] Building 1.0s (30/30) FINISHED
 => CACHED [internal] load remote build context                                                                                                                            0.0s
 => CACHED copy /context /                                                                                                                                                 0.0s
 => [internal] load metadata for docker.io/hexpm/elixir:1.13.1-erlang-24.2-debian-bullseye-20210902-slim                                                                   0.9s
 => [internal] load metadata for docker.io/library/debian:bullseye-20210902-slim                                                                                           0.9s
 => [builder  1/18] FROM docker.io/hexpm/elixir:1.13.1-erlang-24.2-debian-bullseye-20210902-slim@sha256:ba2e50ad59370fab4a63a3751dc867ee45074e5f5201d8b612706d07e6e3e8ba   0.0s
 => [stage-1 1/7] FROM docker.io/library/debian:bullseye-20210902-slim@sha256:e3ed4be20c22a1358020358331d177aa2860632f25b21681d79204ace20455a6                             0.0s
 => CACHED [stage-1 2/7] RUN apt-get update -y && apt-get install -y libstdc++6 openssl libncurses5 locales sqlite3 htop     && apt-get clean && rm -f /var/lib/apt/lists  0.0s
 => CACHED [stage-1 3/7] RUN sed -i '/en_US.UTF-8/s/^# //g' /etc/locale.gen && locale-gen                                                                                  0.0s
 => CACHED [stage-1 4/7] WORKDIR /app                                                                                                                                      0.0s
 => CACHED [builder  2/18] RUN apt-get update -y && apt-get install -y build-essential git npm     && apt-get clean && rm -f /var/lib/apt/lists/*_*                        0.0s
 => CACHED [builder  3/18] WORKDIR /app                                                                                                                                    0.0s
 => CACHED [builder  4/18] RUN mix local.hex --force &&     mix local.rebar --force                                                                                        0.0s
 => CACHED [builder  5/18] COPY mix.exs mix.lock ./                                                                                                                        0.0s
 => CACHED [builder  6/18] RUN mix deps.get --only prod                                                                                                                    0.0s
 => CACHED [builder  7/18] RUN mkdir config                                                                                                                                0.0s
 => CACHED [builder  8/18] COPY config/config.exs config/prod.exs config/                                                                                                  0.0s
 => CACHED [builder  9/18] RUN mix deps.compile                                                                                                                            0.0s
 => CACHED [builder 10/18] COPY priv priv                                                                                                                                  0.0s
 => CACHED [builder 11/18] COPY assets assets                                                                                                                              0.0s
 => CACHED [builder 12/18] RUN cd assets && npm install && ./node_modules/brunch/bin/brunch build -p                                                                       0.0s
 => CACHED [builder 13/18] RUN mix phx.digest                                                                                                                              0.0s
 => CACHED [builder 14/18] COPY lib lib                                                                                                                                    0.0s
 => CACHED [builder 15/18] RUN mix compile                                                                                                                                 0.0s
 => CACHED [builder 16/18] COPY config/runtime.exs config/                                                                                                                 0.0s
 => CACHED [builder 17/18] COPY rel rel                                                                                                                                    0.0s
 => CACHED [builder 18/18] RUN mix release                                                                                                                                 0.0s
 => CACHED [stage-1 5/7] COPY --from=builder /app/_build/prod/rel ./                                                                                                       0.0s
 => CACHED [stage-1 6/7] COPY trips_dump.sql trips_dump.sql                                                                                                                0.0s
 => CACHED [stage-1 7/7] RUN set -eux;     ln -nfs /app/$(basename *)/bin/$(basename *) /app/entry                                                                         0.0s
 => exporting to image                                                                                                                                                     0.0s
 => => exporting layers                                                                                                                                                    0.0s
 => => writing image sha256:3ccdfb991c1b0a30bf2c39340650a7aa7817b493facb9542f92b902a25fd003d                                                                               0.0s
 => => naming to registry.fly.io/dark-breeze-4631:deployment-1640969068                                                                                                    0.0s
--> Building image done
==> Pushing image to fly
The push refers to repository [registry.fly.io/dark-breeze-4631]
975dbf6bf485: Layer already exists
5892b6a0d2ce: Layer already exists
c5637069476c: Layer already exists
85dfabae1706: Layer already exists
8308665c3924: Layer already exists
7229b49946ed: Layer already exists
ba5a5fe43301: Layer already exists
deployment-1640969068: digest: sha256:3472e1d20c3b1e5731dfcc40784ef7a0015fcc473d4f30b649389ffe323e36f5 size: 1786
--> Pushing image done
image: registry.fly.io/dark-breeze-4631:deployment-1640969068
image size: 133 MB
==> Creating release
--> release v3 created

--> You can detach the terminal anytime without stopping the deployment
==> Monitoring deployment

 1 desired, 1 placed, 1 healthy, 0 unhealthy
--> v3 deployed successfully

I was able to reproduce. Can you try with the newest release, 0.0.274?

Unfortunately this doesn’t seem to have changed it for me. I upgraded and deployed, but the mount is still not attached, and the config hasn’t changed.

~/Projects/trips ❯❯❯ fly version update
/opt/homebrew/bin/zsh -c
Running automatic update [brew upgrade flyctl]
==> Auto-updated Homebrew!
Updated 2 taps (superfly/tap and homebrew/core).
==> Updated Formulae
Updated 3 formulae.

==> Upgrading 1 outdated package:
superfly/tap/flyctl 0.0.273 -> 0.0.274
==> Downloading https://github.com/superfly/flyctl/releases/download/v0.0.274/flyctl_0.0.274_macOS_arm64.tar.gz
==> Downloading from https://objects.githubusercontent.com/github-production-release-asset-2e65be/199060308/9a0d91c6-e10f-463c-a824-d94e39aed22b?X-Amz-Algorithm=AWS4-
######################################################################## 100.0%
==> Upgrading superfly/tap/flyctl
  0.0.273 -> 0.0.274

==> Caveats
zsh completions have been installed to:
  /opt/homebrew/share/zsh/site-functions
==> Summary
🍺  /opt/homebrew/Cellar/flyctl/0.0.274: 7 files, 39.8MB, built in 2 seconds
==> Running `brew cleanup flyctl`...
Disable this behaviour by setting HOMEBREW_NO_INSTALL_CLEANUP.
Hide these hints with HOMEBREW_NO_ENV_HINTS (see `man brew`).
Removing: /opt/homebrew/Cellar/flyctl/0.0.273... (7 files, 39.8MB)
Removing: /Users/zander/Library/Caches/Homebrew/flyctl--0.0.273.tar.gz... (18.0MB)
~/Projects/trips ❯❯❯ fly deploy
==> verifying app config ...
--> verified app config.
==> Building image
Remote builder fly-builder-summer-brook-7121 ready
==> Creating build context
--> Creating build context done
==> Building image with Docker
--> docker host: 20.10.12 linux x86_64
[+] Building 225.8s (0/1)
[+] Building 5.2s (30/30) FINISHED
 => [internal] load remote build context                                                                                                                         0.0s
 => copy /context /                                                                                                                                              2.3s
 => [internal] load metadata for docker.io/library/debian:bullseye-20210902-slim                                                                                 0.9s
 => [internal] load metadata for docker.io/hexpm/elixir:1.13.1-erlang-24.2-debian-bullseye-20210902-slim                                                         0.9s
 => [builder  1/18] FROM docker.io/hexpm/elixir:1.13.1-erlang-24.2-debian-bullseye-20210902-slim@sha256:ba2e50ad59370fab4a63a3751dc867ee45074e5f5201d8b612706d0  0.0s
 => [stage-1 1/7] FROM docker.io/library/debian:bullseye-20210902-slim@sha256:e3ed4be20c22a1358020358331d177aa2860632f25b21681d79204ace20455a6                   0.0s
 => CACHED [stage-1 2/7] RUN apt-get update -y && apt-get install -y libstdc++6 openssl libncurses5 locales sqlite3 htop     && apt-get clean && rm -f /var/lib  0.0s
 => CACHED [stage-1 3/7] RUN sed -i '/en_US.UTF-8/s/^# //g' /etc/locale.gen && locale-gen                                                                        0.0s
 => CACHED [stage-1 4/7] WORKDIR /app                                                                                                                            0.0s
 => CACHED [builder  2/18] RUN apt-get update -y && apt-get install -y build-essential git npm     && apt-get clean && rm -f /var/lib/apt/lists/*_*              0.0s
 => CACHED [builder  3/18] WORKDIR /app                                                                                                                          0.0s
 => CACHED [builder  4/18] RUN mix local.hex --force &&     mix local.rebar --force                                                                              0.0s
 => CACHED [builder  5/18] COPY mix.exs mix.lock ./                                                                                                              0.0s
 => CACHED [builder  6/18] RUN mix deps.get --only prod                                                                                                          0.0s
 => CACHED [builder  7/18] RUN mkdir config                                                                                                                      0.0s
 => CACHED [builder  8/18] COPY config/config.exs config/prod.exs config/                                                                                        0.0s
 => CACHED [builder  9/18] RUN mix deps.compile                                                                                                                  0.0s
 => CACHED [builder 10/18] COPY priv priv                                                                                                                        0.0s
 => CACHED [builder 11/18] COPY assets assets                                                                                                                    0.0s
 => CACHED [builder 12/18] RUN cd assets && npm install && ./node_modules/brunch/bin/brunch build -p                                                             0.0s
 => CACHED [builder 13/18] RUN mix phx.digest                                                                                                                    0.0s
 => CACHED [builder 14/18] COPY lib lib                                                                                                                          0.0s
 => CACHED [builder 15/18] RUN mix compile                                                                                                                       0.0s
 => CACHED [builder 16/18] COPY config/runtime.exs config/                                                                                                       0.0s
 => CACHED [builder 17/18] COPY rel rel                                                                                                                          0.0s
 => CACHED [builder 18/18] RUN mix release                                                                                                                       0.0s
 => CACHED [stage-1 5/7] COPY --from=builder /app/_build/prod/rel ./                                                                                             0.0s
 => [stage-1 6/7] COPY trips_dump.sql trips_dump.sql                                                                                                             0.2s
 => [stage-1 7/7] RUN set -eux;     ln -nfs /app/$(basename *)/bin/$(basename *) /app/entry                                                                      0.3s
 => exporting to image                                                                                                                                           0.0s
 => => exporting layers                                                                                                                                          0.0s
 => => writing image sha256:2f28134c15a4a426559efc0ffb713c7342f50aa2375c1af9f75422068a962ca3                                                                     0.0s
 => => naming to registry.fly.io/dark-breeze-4631:deployment-1640971128                                                                                          0.0s
--> Building image done
==> Pushing image to fly
The push refers to repository [registry.fly.io/dark-breeze-4631]
d9955cecdf76: Pushed
9f55ab2b5cdd: Pushed
c5637069476c: Layer already exists
85dfabae1706: Layer already exists
8308665c3924: Layer already exists
7229b49946ed: Layer already exists
ba5a5fe43301: Layer already exists
deployment-1640971128: digest: sha256:ff6b9d7ab724355b2b4e1ebc897d1fefef63f1459c6422809b94be293acf7791 size: 1786
--> Pushing image done
image: registry.fly.io/dark-breeze-4631:deployment-1640971128
image size: 133 MB
==> Creating release
--> release v4 created

--> You can detach the terminal anytime without stopping the deployment
==> Monitoring deployment

 1 desired, 1 placed, 1 healthy, 0 unhealthy [health checks: 1 total, 1 passing]
--> v4 deployed successfully

~/Projects/trips ❯❯❯ fly volumes list
ID                   NAME  SIZE REGION ZONE ATTACHED VM CREATED AT
vol_18l524y7jkp47zmp trips 1GB  lhr    34c3             1 hour ago
~/Projects/trips ❯❯❯ fly volumes list
ID                   NAME  SIZE REGION ZONE ATTACHED VM CREATED AT
vol_18l524y7jkp47zmp trips 1GB  lhr    34c3             1 hour ago
~/Projects/trips ❯❯❯ fly config display
{
    "env": {},
    "experimental": {
        "allowed_public_ports": [],
        "auto_rollback": true,
        "cmd": [],
        "entrypoint": [],
        "exec": [],
        "private_network": true
    },
    "kill_signal": "SIGINT",
    "kill_timeout": 5,
    "processes": [],
    "services": [
        {
            "concurrency": {
                "hard_limit": 25,
                "soft_limit": 20,
                "type": "connections"
            },
            "http_checks": [],
            "internal_port": 8080,
            "ports": [
                {
                    "handlers": [
                        "http"
                    ],
                    "port": 80
                },
                {
                    "handlers": [
                        "tls",
                        "http"
                    ],
                    "port": 443
                }
            ],
            "processes": [
                "app"
            ],
            "protocol": "tcp",
            "script_checks": [],
            "tcp_checks": [
                {
                    "grace_period": "1s",
                    "interval": "15s",
                    "restart_limit": 0,
                    "timeout": "2s"
                }
            ]
        }
    ]
}

OK, I think we’ve definitively fixed this. The bug was only present for those who did not have a build section in their fly.toml.

A new release is coming out now. Once you see it here, upgrade and try again.

Thanks for alerting us to this!

1 Like

Excellent, I can confirm it’s working for me now. The volume is attached and the configs match. Thanks so much for the quick response, I think your company is great.

1 Like