We are looking to port an application over from Heroku and are testing it on Fly.io.
When our application boots up, it prints the ENVs (redacting secret values) to the console. The values that are being printed during boot up are NOT matching what we loaded in via the fly secrets set command. That is concerning.
What is REALLY concerning is that the values I see appear to match values from our Heroku setup on production. How is that possible? Was there an automagical reading of those values and then setting them?
It sounds like you might be on to something except that I did not make any connections to Fly from Heroku. Both Heroku and Fly are connected to Github, but those are the only connections I am aware of.
Locally I have ‘remotes’ to each location… is there a chance that Fly does something there as part of a quick setup routine?
Cheers
@dusty We are trying to evaluate Fly.io and this issue is keeping us from actually testing your system. I just tried creating a brand new project and yet it still pulls in ENVs from somewhere else. I know it is doing this because our build script prints the ENVs on runtime startup. This is really concerning to us.
I removed all remotes (except Github) from my local repo before setting up the Fly project.
I removed all .env files and .sh scripts that Fly might have been looking at
There are multiple ENVs printed that were NOT set by me using flyctl secrets
ENVs that are set somewhere else do NOT show up in our secrets list
ENVs that are set somewhere else are NOT overridable
I have tried to set WORKERS in our secrets list but that is still shown as 28 (which I have no idea where it is getting that value from)
Fly Launch. I have 0 connection to Heroku that I can see. I even moved my git repo to a clean directory on my computer to ensure there were no git remote dependencies.
R
After trying several other PaaS options, I feel like Fly might be a great choice for us despite this mysterious ENV issue. As such, I have returned to try again and am still seeing the same issue. I do see the ENVs as I set them when I run fly ssh console and then env from the box. However, when my system boots, it consoles out all of the ENVs that are set and those values are not consistent with what I set. That scares me… partially because they are not the same but also because there are ENVs there that I did not set (but are definitely from our project). Additionally, I can’t override WORKERS which is set at 28, despite what I see running env on the box.
Could we setup a Zoom call with a tech to get to the bottom of this?
Does your app use anything like direnv and have something like an .envrc? Perhaps a framework that picks an environment specific configuration file that it writes to its env? Do you deploy your app with a custom Dockerfile or did you deploy with one of the pre-made launchers? Any other info you can share about your app like language, framework, etc.?
If you see a difference when you SSH in, then it seems to me that something is happening from inside your app code. Let me expand a bit. Environment variables on Fly VMs come from a couple places:
then anything that’s been set in your own Dockerfile, or part of your built image
The variables you see when SSHing in are the environment variables Fly’s systems have set at VM boot and we don’t do anything interesting or different for SSH versus booting your app process etc.
I think the only way you’d see a difference between env via SSH and from your app logs is because your app process set it. If your app set superfluous env variables during its boot (via some library or something along those lines), then they’d be set for your app but not for any other running processes on your VM (like an SSH session).
Hey! Did you mean for flyctl ping - #4 by mobilevet to be a response to this thread by chance? It looked like you responded by email and the forum maybe misrouted your response.
If so - yes very possible. The Heroku Buildpacks confuse me quite a bit, honestly. If you’re comfortable with it, I would consider getting a basic Dockerfile setup for your project, in part because the Heroku buildpacks are super slow to deploy because they’re huge images that try to serve everyone rather than your projects’ needs. nodejs.org has a pretty good write-up for how to dockerize a Node web-app here.
In reference to not finding a node process; I did find some of the source for the Node part of the buildpack here I think and I found these variables that would make me think your node binary is somewhere in $HOME/.heroku/node/bin/.
Ha, yea, I think I sent this via email reply and it looks like the streams were crossed.
Thanks for the info, I will take a look at the docs and see if I can run node from the directory you mentioned. That would be the easiest solution. Unfortunately I have deployed two more versions this morning and the flyctl ssh console command is no longer completing successfully for me. It did on version 6 from yesterday, but immediately after deploying this morning I am no longer able to connect.
Initially it said Error host unavailable at top1.nearest.of.server-dev.internal: host was not found in DNS
Now it tries to connect but says operation timed out
I am deploying with the Heroku build pack, heroku/buildpacks:20.
We do have a .env file locally but that is not included in the repo so it should NOT be on the machine. However, that said, I think the fly deploy is picking it up. I swear I tried deploying after deleting this file locally but when I just changed a value and deployed it updated. Interesting that secrets will override that but if a value is NOT in secrets it will pick it up from the .env.
So, that mystery is solved. Thank you for the help (really thought I had checked that… )
Finally mystery is the WORKERS=28. I am NOT setting that and in fact I set secrets WORKERS=1 and that is confirmed when I ssh in and run env. My guess is that is tied to the concurrency setup of the app somehow? My toml file specifies a hard=25 and soft=20 limit, but neither of those are 28. Additionally, I understood that to be the limit for when it starts to load balance… not a setting of the WORKERS.
Hi @mobilevet, I’m afraid I don’t have insight into your WORKERS mystery, but I noticed you keep referring to your repo. I’m not sure if this helps, but to be clear, you don’t need git at all to deploy to Fly. If you want to make sure a file that’s present in your working directory doesn’t get included in the Docker build context, you’d use a .dockerignore file.
When I login via SSH and look at the ENV I see WORKERS=1.
When my system starts up, it prints out the config values that are loaded in and that is what shows WORKERS=28.
It is also quite easy to confirm the value as we use Node clustering and we see 28 instances of the server load (hence our RAM footprint).
@Brad@jphenow That all seems to add up. What I find interesting is that we don’t experience that behavior on Heroku and we clearly use their Node build pack there.
Regardless, I think we have solved the mystery. Thanks for tracking it all down.