Environment variables set by flyctl secrets

I have been trying to use flyctl secrets instead of using .env file for Firebase connection parameters. I am using Docker to deploy the Node application and are using Nginx. It works when including the .env file in the repository, but using “flyctl secrets import” does not. I have accessed the machine using “flyctl ssh console ” and the secrets are visible as environmental variables but the application does not work as I get some Firebase connections errors. Is there a container on the VM that I am not seeing? where the application is located? If so, does it not get the environment variables from the secrets?

Secrets should work pretty much exactly like environment variables.*

A shot in the dark: Does your .env file contain quotes? And did you run flyctl secrets import directly on it?

flyctl secrets import does’t recognize normal quoting syntax (ref, code), so if your .env file has a line such as AUTH_TOKEN="abc" and you run flyctl secrets import < .env then the secret AUTH_TOKEN will be set to "abc" (literally, with the double quotes; which is bad). You can check if this happened by opening an SSH connection and running env, there shouldn’t be any quotes in the output.

*Secrets are not available at build time (notes), but I don’t think that’s the issue here.

1 Like

Thanks for the response, Tom!
I have tried with both AUTH_TOKEN=“abc” and AUTH_TOKEN=abc where both tries failed. I am trying to prevent using an env file and are using the flyctl secrets import -a command to set the secrets in the terminal without the use of an env file. This should be done before deployment.

Checked the VM by using the ssh console command on both tries, both had env variables listed as “abc” and abc depending the set secret. So it looks like the env variables are correct. Note that I am on windows and are using the CLI. Do you have any other suggestions?

You definitely don’t want quotes in the value of the variable. I’m surprised AUTH_TOKEN=abc failed. Another thing to watch out for is Windows line endings; make sure you are using a recent flyctl version (check using flyctl version, it should be >= v0.0.475 so as to include this commit).

I suggest you run the following command in an SSH shell so you can see exactly what the env var is set to:

node -e 'require("dotenv").config(); console.log(JSON.stringify(process.env.YOUR_ENV_VAR));'

(This uses JSON.stringify to display special characters. So the output here will be surrounded by double quotes, which is fine. What you don’t want is quotes inside of that; e.g. "\"abc\"" would be bad.)

Try running this with just the secrets (without the .env file), then try it with your .env file after unsetting the secrets. The output should be identical.

The reason for unsetting the secrets is that dotenv doesn’t override existing environment variables by default, so if the secret is set then the value in the .env file should be ignored. If it isn’t ignored, that might indicate that something is wrong, e.g. the variable might have the wrong name.

The full Firebase error message might be also useful for debugging (make sure to remove any sensitive info if you post it).

1 Like

Thanks for the reply!

Tried your command but it did not work. Not sure if you meant locally or on the VM, but on the VM node was not recognized and locally I had trouble with dotenv with the reference error of “dotenv is not defined”.

Here is how I am trying to implement the secrets (the original values is not displayed). I am marking EOF with CTRL+Z, but I have tried manually entering each secrets with flyctl secrets set NAME=VALUE:

Here is the Dockerfile:
image

In the Fly.toml file I am changing the default internal port from 8080 to 80 to make Nginx work. The error I am getting in the console of the deployed application is “caught FirebaseError: Firebase Storage: No default bucket found. Did you set the ‘storageBucket’ property when initializing the app? (storage/no-default-bucket)”.

I meant on the VM. I don’t know how you can be deploying a Node app if Node is not installed…

The screenshots all look fine (the digest confirms there are no line ending issues). Here is what I would do:

  1. Get back to a working deploy with just the .env file (may need to unset the secrets).
  2. Replace the sensitive values in the .env file with placeholders, and add some code to log the environment variables. (You must have some JavaScript code somewhere that reads those environment variables; you can simply insert console.log(JSON.stringify(process.env, Object.keys(process.env).sort())); there, and the output should appear in flyctl logs.)
  3. Remove the .env file, set the secrets, redeploy, and compare the output.

The error message is interesting, it might suggest that the environment variable REACT_APP_FIREBASE_STORAGE_BUCKET is missing, or that your initialization code is wrong. I’m guessing it’s something similar to this, may be helpful if you can post it.

1 Like

Followed your steps and it looks like it works when including the .env file. Although I was using dummy values for the Firebase connection parameters the application was working. Deleting the .env file and using secrets instead resulted in the Firebase storage bucket error. The console output was correct when using the .env file but did not output anything when using secrets. Maybe it has something to do with how the application gets the environment variables?

I think nginx is to blame: https://nginx.org/en/docs/ngx_core_module.html#env

By default, nginx removes all environment variables inherited from its parent process except the TZ variable. This directive allows preserving some of the inherited variables, changing their values, or creating new environment variables.

Adding some nginx config directives of the form env REACT_APP_FIREBASE_API_KEY; should hopefully propagate the environment variables and fix the issue.

[Edit: I was wrong, see Environment variables set by flyctl secrets - #16 by fideloper-fly]

1 Like

Not sure how to implement those directives. Here is the config file I am currently using:

image

Are you thinking of specifying the env variables with its value in that file? Something like this:
environment {
REACT_APP_FIREBASE_API_KEY=123
}

Or is there a way of defining the name of the env variables which it retrieves the values from?

Tested by adding the parameters as env in the config that is included, but got the error message “nginx: [emerg] “env” directive is not allowed here in /etc/nginx/conf.d/default.conf:2” in the fly console. It looks like the env is only valid in the main context and not what is to be included.

Add them at the top level, e.g.

env REACT_APP_FIREBASE_API_KEY;
env REACT_APP_FIREBASE_AUTH_DOMAIN;
...

http {
    server {
        ...
    }
}

No need to set the values in the config file. The version shown will inherit the values from the environment variables (Fly secrets).

Yes, tried it but did not seem to work. Tested by adding the parameters as env in the config that is included, but got the error message “nginx: [emerg] “env” directive is not allowed here in /etc/nginx/conf.d/default.conf:2” in the fly console. It looks like the env is only valid in the main context and not what is to be included.

Indeed, the env directives should go in /etc/nginx/nginx.conf rather than /etc/nginx/conf.d/default.conf.

The docs suggest copying /etc/nginx/nginx.conf from the container and using that as a starting template for a custom config file.

Changed the Dockerfile from /etc/nginx/conf.d/default.conf to /etc/nginx/nginx.conf but got this error message “nginx: [emerg] “server” directive is not allowed here in /etc/nginx/nginx.conf:9”.

Not sure what you meant by copying the /etc/nginx/nginx.conf from the container. I apologize for all the questions as I am not that familiar with this stuff.

No worries.
The file /etc/nginx/nginx.conf is the main config file, and it includes /etc/nginx/conf.d/default.conf.
As per your original setup, the file nginx.conf in your source directory is copied into the image as /etc/nginx/conf.d/default.conf (the names are confusing, I know).
To add the env directives, you should also overwrite /etc/nginx/nginx.conf.
So create a second config file, say nginx-main.conf, and add a line COPY nginx-main.conf /etc/nginx/nginx.conf to your Dockerfile.
The contents of nginx-main.conf should be the same as version of /etc/nginx/nginx.conf that exists in a clean container, except you’ll add the env directives.
So grab a clean copy of /etc/nginx/nginx.conf, e.g. by SSH-ing into a VM and using cat (hint: make sure your Dockerfile isn’t overwriting that file, by checking that it contains http { ... }). Then paste it into nginx-main.conf and add the env directives.

Believe I did it correctly. Went to the VM by SSH-ing and copied /etc/nginx/nginx.conf (it included the http {…}. Added a new file in the root folder called nginx-main.conf and added the env directives.

Changed the Dockerfile to COPY the new file:

image

Deleted the application and made a new including the secrets but still got the same error as before, “caught FirebaseError: Firebase Storage: No default bucket found.”. It still seems like it did not get the environment variables.

Hello!

I’m not sure, since I don’t do much React / Firebase - What code is interacting with firebase? Is it:

  1. Frontend React code?
  2. or Backend NodeJS api code (the API that react is calling)?

If it’s :one: then it’s likely an issue of secrets not being available by default at build-time during a deployment. Frontend code isn’t going to be able to read env vars set in the VM when run on a client’s web browser. Here are docs on making them available at build-time, so the front-end has the opportunity to write the values somewhere it can read.

If it’s :two: then something weird is going on, but I don’t think it’s related to Nginx

The Nginx config shown so far is just serving static sites, not hosting any API backend, so I suspect the issue is :one:

1 Like

Yeah, it looks like it’s React code running in the web browser; my comments on nginx were misplaced (apparently it doesn’t spawn CGI processes, unlike Apache (ref); sorry about that wild goose chase).

Since these variables will be publicly exposed to clients (which is okay for Firebase), I’m not sure if there is much point in using Fly’s secrets to manage them. It would probably be easier to stick with a .env file (or .env.production if you want to keep it outside of version control). Another idea is to switch to build args, which you can set from fly.toml (Dockerfile ref, fly.toml ref).

Thanks guys! I will just stick with including a .env file for the build then as the Firebase connection parameters is not as sensitive as I thought.

Tried using build secrets as follow:

RUN --mount=type=secret,id=API_KEY
API_KEY=“$(cat /run/secrets/API_KEY)”

and deployed the application using fly deploy --build-secret API_KEY=example

This resulted in the previous errors, no secrets where added and the application seemed to not find any variables for the Firebase connection. Any thoughts would be appreciated.

The secret is only available for the duration of the RUN command, so make sure you are making it available at the right point. In your case it’s npm run build, so your Dockerfile should have something like this:

RUN --mount=type=secret,id=API_KEY \
    API_KEY="$(cat /run/secrets/API_KEY)" \
    npm run build

(Aside: The example of multiple commands in the docs is incorrect. They use RUN --mount=... MY_SUPER_SECRET=... some_command && more_commands_maybe but due to shell syntax rules that only makes the secret available for some_command. The way to also make it available for more_commands_maybe is to use RUN --mount=... export MY_SUPER_SECRET=... && some_command && more_commands_maybe.)