How to deploy app and secrets together?

[dcyoung]

We’re looking to streamline our secrets mgmt. Currently our CI/CD pipeline handles deployment of the app using flyctl, but we have no means of guaranteeing an app and secrets are deployed together.

In previous setups, i’ve handled this via sealed secrets and config as code… where manifests reference the secrets for their deployments, but not sure how to solve w/ fly.

As a step toward keeping applications + env configuration in lock step, is there a way to “set” secrets in a “deploy” CLI cmd, allowing the caller to maintain the versioning logic but guarantee the app and secrets are deployed together or not at all?

[kwaw]

Does fly secrets set SECRET=value not work for your use case?

[dcyoung]

As i understand it, running a deploy and a secrets set command separately will yield separate deployments for the code and the secrets. Setting secrets first without triggering a redeploy is still destructive to the previous secret values in case a subsequent deployment of new code fails.

Consider the scenario where [app_v0, secrets_v0] are both deployed. The user wants to deploy [app_v1, secrets_v1] together, because either combination of [app_v0, secrets_v1] or [app_v1, secrets_v0] would NOT be a viable deployment.

How can [app_v1, secrets_v1] be deployed TOGETHER, with rollback to [app_v0, secrets_v0] in the event of a deployment failure.

Is there a way to do this with fly secrets set?

[greg]

Hi,

Perhaps there is a way but personally I don’t know of one.

I had two ideas which aren’t exactly what you are after but perhaps get a bit closer.

1. Use fly deploy --env NAME=VALUE. That reads like you can set run-time values at the same time as the deploy but they wouldn’t be encrypted and so wouldn’t be secret. If your values have to be kept secret, that wouldn’t be the solution.

2. First run fly deploy --build-only. That would build e.g [app_v1] but not deploy it. At this point you would still have [app_v0, secrets_v0] deployed. But have [app_v1, secrets_v0] ready. Then you’d run fly secrets set NAME=VALUE. Running that command updates the secrets and triggers a new release (as new secrets are only pulled in when a VM starts). In theory … that should release the latest built image. Which would be … app_v1. And so you would then get [app_v1, secrets_v1] deployed.

From flyctl deploy · Fly Docs

I haven’t tried either but can’t hurt to try it on a test app and see what happens.

[fideloper-fly]

I think you want the --stage flag when setting secrets (docs)

The --stage flag:

--stage           Set secrets but skip deployment for machine apps

This is for “machine apps” which are our “appsv2”. If you run fly status on your app, you’ll see if it’s “nomad” based or “machines” based. You want machines.

So, in your pipeline, you could do something like this pseudo code:

foreach(secrets as key, value):
    fly secrets set "key=value" --stage
endforeach


fly deploy

API Calls instead?

The other thing you can do is call the equivalent GraphQL API call directly instead of using the CLI (this is the same as the --stage flag, the secret gets set but no deployment is kicked off).

curl -H "Authorization: Bearer $FLY_API_TOKEN" \
    -H "Content-Type: application/json" \
    -H "Accept: application/json" \
    https://api.fly.io/graphql \
    -d '{
  "query": "mutation($input: SetSecretsInput!) { setSecrets(input: $input) { release { id version reason description user { id email name } evaluationId createdAt } } }",
  "variables": {
    "input": {
      "appId": "my-app-name",
      "secrets": {
        "key": "SOME_SECRET_KEY",
        "value": "some-value"
      }
    }
  }
}'

To use that API call yourself, set:

1. FLY_API_TOKEN
2. appId (the name of your app)
3. The key/value pair(s) of secrets.

I believe you can set multiple secrets by using an array of "secrets" (but in my testing, setting a single secret also worked).

[dcyoung]

the --stage flag is interesting. Any documentation on how it behaves? Will restarts of the existing deployment still use the older secrets (pre staging the new secrets).

CI/CD scenario, w/ existing deployment of [app_v0, secrets_v0]

1. stage secrets_v1
2. attempt deploy app_v1
3. deployment fails, [app_v0, secrets_v0] remain deployed
4. sometime later, app_v0 is restarted… will it use secrets_v0 or the newly staged secrets_v1??

[fideloper-fly]

Restarts will not use new secrets, only cases where a new VM is created.

Cases where new VM’s are created:

1. Deployments (create new VMs, destroy last ones)
2. Updating a machine via Machines API (destroy old machine, create new machine).

Machine VMs are immutable, so staged secrets don’t apply to machines that are just in a “stopped” state.