Can secrets be dynamically managed after boot

I have the need to rotate secrets on an hourly basis. It does not appear this can be done with fly secrets (which I assume are being managed by vault). Is there any other service that fly offers in this regard? I really don’t want to run my own vault if I can avoid it. The flow I would like is:

  1. update the secret dynamically within our database
  2. update the secret within the vault
  3. all running instances would get the updated secret (ie: using Hashicorp’s templates???)

Is this option available with fly’s standard server? I assume not, but wanted to check before I started figuring out how to do it myself.

Not sure fly.io has something built-in for that.

I would go meta here and just spawn a special app on the fly.io that would manage the secrets rotation for all other apps.

That management app would use flyctl and maybe cron to do that. The only possible pitfall is that you would need to authorize that flyctl somehow. I’m not sure how one can achieve that in a headless setting. Another potentially unpleasant part is that affected apps would be restarted on every change of secrets, but you probably expect that anyway.

1 Like

Adding to what Hypermind said above…

I am sure Fly’s thinking about supporting fast-rotating secrets, as it isn’t really that niche of a usecase.

That said, if I were you, I’d attempt my hand at envelope encryption. A pre-shared master secret key (which doesn’t change as often, if at all) is deployed to all app instances via fly secrets. And actual secrets themselves encrypted with the same master key are downloaded into the app instances every hour (or whatever the desired schedule is) from a preset location (S3 / DynamoDB).

Another option might be to squeeze out secrets on-the-fly, based on normalized time (say, once every hour) or some counter, from a preshared key and a key-derivation function [0], but this scheme isn’t suitable for all use cases, and easy to get wrong. Also, I am not a cryptographer, so you should probably not take me seriously on these matters :wink:

[0] For a HOTP service we run, we generate new keys from the same shared secret depending on user identity, random salt, and some session identifiers like time of request and source ip): https://github.com/celzero/otp/blob/f6bb5593c5173a284417863806ad6d5a57b19539/src/core/gen.js#L17-L19

1 Like

Good suggestions. Yeah, I would assume this is something on their roadmap. For now, I have enough experience with Vault that I’ll just go that route to start although the envelope encryption approach is interesting and something I’ll have to take a closer look at.