Issue with secrets?

Seems to be some issues with secrets at the moment. Unless it is just me.

e.g for an example:

$ flyctl --app name secrets set AWS_DEFAULT_REGION="eu-west-2"
Error no change detected in secrets
$ flyctl --app name secrets set AWS_DEFAULT_REGION="madeup"
Error no change detected in secrets
$ flyctl --app name secrets set AWS_DEFAULT_REGION="hmm"
Error An unknown error occured.

… it doesn’t seem to spot changes in secrets. Unsetting a secret leaves it set too. Hmm.

It would really help if the console had some UI way to set/remove secrets. Manually typing them in the CLI in huge commands isn’t ideal.

Thanks.

We had a temporary issue with writing to Vault (where we store secrets) that was the likely cause of this. Can you give it another try?

Ah.

I still get the error.

$ flyctl --app name secrets set AWS_DEFAULT_REGION="this-value-is-different"
Error no change detected in secrets
$ flyctl --app name secrets set AWS_DEFAULT_REGION="and-this-value-is-different"
Error no change detected in secrets

To confuse things, when I list the secrets, it seems to have spotted the change. I don’t know the value it now thinks it is because it just shows a digest, naturally. So I guess I would have to add some kind of debug line in the app to get it to output to console/logs and wait for them to come in.

However … if the value had changed … it should trigger a new deployment, I would think. Which it hasn’t. So not sure what is happening.

$ flyctl --app name secrets list
NAME                  DIGEST                           DATE      
AWS_DEFAULT_REGION    500e3c268d448f7036ab9a0d3b... 8s ago

Oh, interesting.

I tried it again, and sure enough, got 'no change detected in secrets.

But when I listed the secrets, the digest had changed. So it has realised there is a new value.

But not triggered a deployment. Does that only happen if a secret is added/removed? In which case perhaps that is correct behaviour. Apart from the error message.

… although … if the value of the secret gets ‘baked in’ to the instance when it is deployed, that would not be correct behaviour. As it would be left with the old value of the secret. Hmm.

On the other hand, if the value of the secret is read-in at runtime, then a changed value would not require a new deployment. Not sure how things work at your end, naturally, what the expected flow is. So far I’ve seen a new deployment happen each time but maybe that was just because of the sequence happened to add/remove a secret.

Will you run flyctl releases and see if there are any releases for your secret changes? It should trigger a deploy when you update one.

Ah … good point.

Yes, it is showing a release there. So it is deploying by the looks of that. Just in the background, presumably. So I guess the error message is disconnecting from the deployment process. Since previously when I changed a secret, I saw a message saying deploying a new version and it does its thing and lists the instance, healthy etc. That must be happening but just in the background by the looks of the releases page as it does take about two minutes to release which would match the timings.

So maybe that error isn’t so much of an issue if it actually is detecting a change in the value (based on the digest) and is triggering a deploy (based on releases) after all.

That seems to be what’s happening. We’ll look into that bug, it should definitely show you deployment status when you set one.

Ok, no problem.

It may be connected to this, but on the ‘secrets list’ page, the DATE column does not show the time a few seconds ago for that secret.

But that may be correct. I’m not sure if the “DATE” column is showing the time the secret was created or the time it was last modified. If it was the time it was created, that is correct. If it is the time it was last modified, it’s wrong. Since it should be showing the date today if so.

AWS_DEFAULT_REGION 95988846d3320cc180241b63161... 2020-09-08T19:40:16Z

Is it possible you reset that secret the value it was on Sept 8th? We version secrets internally, if you set a value you’ve used before it just uses the older version. It’s not well exposed through our CLI though.

Ah … !

You are right. I hadn’t realised that was the case. Yep, I have been cycling through AWS regions today trying eu-west-1,2,3 … is there even a 3? Don’t know. Anyway, yep, if I set it as eu-west-2, the date drops back to September. If I try eu-west-3, which I’ve never typed before, sure enough the DATE changes to 5s ago. So it is essentially time that value was first seen.

That explains that.

I still get Error no change detected in secrets

But that doesn’t matter if it’s doing the right things behind the scenes :slight_smile: