Secrets are not replicated across regions

Questions / Help
1

Currently, my app is deployed in CDG. When I want to deploy a new version, it sticks on AMS (no matter how many times I run the deployment command).

The issue is that secrets are not replicated across regions when deploying so the app fails everytime because it can find the environment variables it needs to run. So I’m stuck with an obsolete version of my app because the deployment always happen in AMS instead of my current environment in CDG.

I’m a bit lost here. It seems to me a major functionality is missing ? (having my env vars imported through the fly.toml or anything else is not option; secrets are what they are : secrets)

2

For the region, I believe Fly tries to intelligently pick the initial region based on a combination of request location and capacity. Like if I deploy, from the UK, I tend to get lhr but if that’s not available I may get ams. So I’m guessing you are finding similar, getting ams.

You can see what region pool is available to use:

$ flyctl regions list
Region Pool: 
lhr
Backup Region: 
ams
cdg

If you only want cdg, make sure that is the only one listed: flyctl regions backup

As regards secrets, hmm … I don’t think secrets are per region. Mine aren’t. They are injected at runtime when the vm boots. So unless something has gone wrong, when your vm starts, in region X, it should pull in the secrets from Fly’s vault. Not sure why yours is not.

3

You can also use the flag --now to force the deploy to go through even if it fails, which is sometimes useful for debugging stuff like this.

4

Thanks for your replies.

Indeed, the backup regions should not be the problem. I lately destroy and recreating my app and now I cannot access my env variables anywhere, even if they are created and listed in the “secrets” command.

I’m starting to lose my mind since I passed all the afternoon on this deployment that should be completed before tomorrow.

Maybe the issue is that I’m not patient enough and that I suspend/resume VMs at first so I can deploy my env variables and then really deploy the app. Maybe it messes up with inner configuration of Fly. I don’t know.

5

You can force cdg or another region overriding backup regions: fly regions backup cdg.

Try setting a secret with the app running. It sounds like releases are conflicting and possibly rolling back. fly status will show you instances with version numbers, the same versions will have the same environment variables (including secrets).

6

Thanks Kurt, I understand better what could be happening.

In fact, my app need env vars to be set up to start. Without that, the health check don’t pass. If env vars are set after the first deploy, it could explain why everything fails and rolls back constantly.

7

Ok. Still no luck.

So here’s what I do to create my app :

❯ fly launch
An existing fly.toml file was found for app tms-edi
? Would you like to copy its configuration to the new app? Yes
Creating app in /home/aurelien/Dev/tms-edi
Scanning source code
Detected a Dockerfile app
? App Name (leave blank to use an auto-generated name): tms-edi
Automatically selected personal organization: John Doe
? Select region: fra (Frankfurt, Germany)
Created app tms-edi in organization personal
Wrote config file fly.toml
? Would you like to setup a Postgresql database now? No
? Would you like to deploy now? No
Your app is ready. Deploy with `flyctl deploy`

Then I set the env vars :

flyctl --app tms-edi secrets set TMS_EDI_MAPSTIMEV2_BASE_PATH='...' TMS_EDI_MAPSTIMEV2_TOKEN=... TMS_EDI_DATAGOUV_BASE_PATH=... TMS_EDI_CONFIGURATION='...' TMS_EDI_AWS_ACCESS_KEY_ID='...' TMS_EDI_AWS_SECRET_ACCESS_KEY='...'

Secrets are staged for the first deployment

(I can provide the Docker building output if needed)

❯ fly deploy --now

Here’s the running PHP app :

2022-02-13T18:48:09Z app[494dc726] fra [info]PHP Fatal error:  Uncaught InvalidArgumentException: 'TMS_EDI_CONFIGURATION' env var should be defined in /app/src/config.php:47
2022-02-13T18:48:09Z app[494dc726] fra [info]Stack trace:
2022-02-13T18:48:09Z app[494dc726] fra [info]#0 /app/src/index.php(13): require_once()
2022-02-13T18:48:09Z app[494dc726] fra [info]#1 [internal function]: {closure}()
2022-02-13T18:48:09Z app[494dc726] fra [info]#2 /app/vendor/amphp/amp/lib/Coroutine.php(67): Generator->current()

And the secrets :

❯ flyctl --app tms-edi secrets list
NAME                          DIGEST                           DATE
TMS_EDI_AWS_ACCESS_KEY_ID     afc72e350c59ae74339f4e0b9b742e62 1m55s ago
TMS_EDI_AWS_SECRET_ACCESS_KEY 917fdfa90a0d667dad49127b2c9f2a87 1m55s ago
TMS_EDI_CONFIGURATION         92ff01c495bf7f11aa93ee644d0e17c7 1m55s ago
TMS_EDI_DATAGOUV_BASE_PATH    193000c61af6b6e51a3413f2dade9b38 1m55s ago
TMS_EDI_MAPSTIMEV2_BASE_PATH  9f345351b280f78e2df9ebeebc56b2ca 1m55s ago
TMS_EDI_MAPSTIMEV2_TOKEN      579e2af51a9287d548e17ec58b1452b9 1m54s ago

So, everything is in place. But it never works. Do I miss something so the env vars can be passed to Docker container ?

8

I forgot :

Instances
ID              PROCESS VERSION REGION  DESIRED STATUS  HEALTH CHECKS           RESTARTS  CREATED
494dc726        app     0       fra     run     running 1 total, 1 critical     0         7m50s ago

It seems this is the v0. But the previous fly deploy --now command tried to deploy the v2 :

deployment-1644777713: digest: sha256:2319122d8e36d99899408c4186ef8681d42201da03b4df067b674855fd82868d size: 4296
--> Pushing image done
image: registry.fly.io/tms-edi:deployment-1644777713
image size: 152 MB
==> Creating release
--> release v2 created

--> You can detach the terminal anytime without stopping the deployment
==> Monitoring deployment

 1 desired, 1 placed, 0 healthy, 0 unhealthy [health checks: 1 total, 1 critical]
9

Okay. I finally understood what the problem was. It came from S6 (GitHub - just-containers/s6-overlay: s6 overlay for containers (includes execline, s6-linux-utils & a custom init)) that I use to handle services in my container. If you don’t pass S6_KEEP_ENV=1 in the env vars, it drops everything else…

Furthermore, S6 does not seem to respond for the health checks and if the PHP service fails it restarts it indefinitely.

I removed all this overlay to test a bit, and all is working as expected.

Thansk for your time, anyway :wink:

1 Like