What is the correct workflow for utilising secrets during deployment?

In our ruby codebases, we use a shared ruby gem that is hosted on Github Packages. This gem is not open source, so we need a secret to pull it from Github packages when we deploy our app. Luckily, this is a frequently demanded use case so Github has completely integrated Rubygems and we do not have to change anything within our codebases.

To authenticate to Github packages, we use a url that contains the username and token needed for authentication as the Gem source in our Gemfile.

source ENV["ENV_VAR_NAME"] do
  gem "GEM_NAME"
end

Now it is not clear to me how to access this value during the deployment process in fly.io, whilst keeping the value secret. I’ve found that ENV vars and secrets are not available during build/deploy. So I’m positive I’ve missed something in the docs, but after hours of searching I still haven’t found how Fly solves this.

Does anybody know this?

tl;dr: How to store secrets to be used during deployment?

Hi @catadesk

For setting environments variables for use during deployment and not during runtime, you can use the --env option for fly deploy.
EDIT: this is actually for runtime not deployment.

Here’s the docs related to fly deploy.

Are you using a Dockerfile? Could you post it here?

The deploy --env option does set runtime environment variables. It works just like the fly.toml [env] section. You’d be better off using the --build-arg flag. Docker build args are passed to your build but not available at runtime.

1 Like

No dockerfile, this is a Ruby/Rails app which uses Fly’s default buildpack. So there is no Dockerfile involved.

I also figured out that placing the token in the url is not the ideal solution, even if it’s a secret. Bundle doesn’t work well with that it seems.

I’ve been trying all kinds of things to get the the auth token to the bundle process during deployment. Now looking into Github Actions, no success so far. Any ideas? Anybody using Github Packages with non public Ruby gems and Fly.io?

You can pass build args from environment variables during the deploy: fly deploy --build-arg MY_BUILD_SECRET=$MY_BUILD_SECRET

Will that do what you want?

Ok first of all, thanks for all the suggestions! The rapid response of the community really helped me get an idea of the situation.

I do have a working solution for now, which is referring to the git repo instead of the Github Packages rubygem. Now we can use the package. The secret still leaks to Gemfile.lock, but we can limit that risk by using a token with limited read capabilities.

Will update when we have a better solution. Also still interested to hear from other github/ruby/fly users!

Similar issue with Next.js. During deploy, we build parts of the site that be static. To do so, we rely on a production database connection (outside of fly). Leaking a production string anywhere in the build process or committing a secret to our repo is not an option, so is there a preferred path here?

Sending build-args seems not ideal, as Fly’s own documentation warn against the use of this option for secrets since they’re baked into the image.

Another option is to build locally with the Buildkit build secrets feature, then push the image to the Fly registry.

Something like:

fly auth docker
DOCKER_BUILDKIT=1 docker build -t registry.fly.io/myapp:tag --secrets mykey=value
docker push registry.fly.io/myapp:tag
fly deploy -i registry.fly.io/myapp:tag

Using these secrets in Dockerfile is a bit annoying, as they have to be mounted at a filesystem path. But maybe worth a try! We’d love to support this in fly deploy directly someday.

Also check out the linked article for a similar use case for SSH-based authentication to things like private repositories.

1 Like

Thank you! I hope you do consider this for the future. One of the things I love about fly.io is how low-friction it feels. This suddenly feels like some friction.

Even so, appreciate the fast response and I’ll go this route once I understand how this works. I’ve been having to avoid local builds because of issues between Docker and M1 chips. :frowning:

I’m not sure if I should be doing this, but I am doing this and it works, in case this reply helps others.

Wanting to do --remote-only builds and wanting to keep secrets as secret as possible, I added this to my Dockerfile:

ARG MONGODB_URI
ENV MONGODB_URI=$MONGODB_URI

Then on deploy: fly deploy --remote-only --build-arg MONGODB_URI=etc. Docker will be passed my key value as an argument. ARG will make it available as a variable. And finally ENV will make it available to my application as build-time.

Phew!

We only recommend against this since it means the URI is stored along with the image. If the image ends up on a local machine or another repo, that value can be leaked. But if that doesn’t concern you, it’s probably fine for now!

3 Likes

Ok, thank you! That makes sense. I’ll rework this. :slight_smile:

the build secrets consideration add an extra layer that is quite confusing.
But can confirm your solution did worked out !

Am I wrong to think there is no need to define fly.io secrets via flyctl secrets set ... when using docker ?

Command :
flyctl deploy --remote-only --build-arg MY_SECRET=${{ secrets.MY_SECRET }} ...

Dockerfile :

FROM xyz

ARG MY_SECRET
...

ENV MY_SECRET=$MY_SECRET
...

Kind of. build_args aren’t exactly trustable custodians for your super secret secrets (ref). Fly’s secrets infrastructure is backed by a software security module, which (on-paper at least) has stricter guarantees and stringent access controls than ENVs set in docker images (ref).

1 Like