v2 migration replaced app secret

[tinypilot]

Fly auto-migrated my app to v2 this week, and it broke me in an unexpected way.

I’ve resolved the issue, but I’m sharing in case it helps others or in case the Fly team wants to investigate.

I believe the issue was caused by having conflicting environment variable values between fly secrets and the [env] section of my fly.toml.

The app was gatekeeper.

The app had a secret I created at some point called SHOPIFY_SHOP_NAME, but I realized that the value wasn’t really secret, so I moved it to the env section of my fly.toml and assigned it the value tinypilot. That was probably a year ago.

After Fly auto-migrated my app to v2, everything worked fine. SHOPIFY_SHOP_NAME still had a value of tinypilot.

We deployed the next day from CircleCI with fly deploy, and that changed the value of SHOPIFY_SHOP_NAME from tinypilot (correct) to YOUR-SHOPIFY-STORE-NAME (incorrect). I’m not sure where it got the value, but I suspect that I initially assigned fly secrets set SHOPIFY_SHOP_NAME=YOUR-SHOPIFY-STORE-NAME and then corrected it by overriding the value in my fly.toml.

When I realized what happened, I ran fly secrets unset SHOPIFY_SHOP_NAME and my app picked up the correct value of tinypilot again.

What it looks like happened is that in v1, the fly.toml took precedence over fly secrets, but in v2, fly secrets takes precedence, but I’m not entirely sure that’s what happened.

If anyone from the Fly team wants to peek into the account and share insights, you’re welcome to. Again, there’s no outstanding issue on my end, but I wanted to share in case it’s useful.

[nina]

Oof, that’s not awesome. Thanks for sharing all the details and how you fixed it. We’ve seen a version of this before and I thought we fixed it. We’ll take a look and make sure it doesn’t affect future migrations.

[nina]

Not to get too deep in the weeds but we think this is happening for a handful of apps where the setting for the “latest” job we get back from Nomad don’t match the current job. We’ve identified a few apps that fit that bill so we’ll be working with them separately on migrations - most likely they will need to migrate themselves with our help.

Thanks for writing this up, it’s super helpful!