Curious how folks here have been storing customer secrets? It feels like Vault is still the safe bet, but their prod deployment guide looks rather daunting and I haven’t seen any mentions of anyone setting it up successfully here.
I also vaguely remember that Fly.io used Vault internally to build their own secret management features, presumably dogfooding their own platform to do so. Would love some pointers on how to get started from anyone from Fly reading this (a full guide would be even more amazing, but I realize this is might not be a common enough use case to get that kind of first class treatment).
Alternatively, if anyone’s had success with alternatives to Vault for secure customer secrets storage that might be easier to get started with, I’d love to learn more.
Store secret key with HSM / Software-SM like fly secrets.
Use HKDF to derive per-customer keys (this is super tricky to get right; https://archive.is/OORx9).
Encrypt / decrypt customer payloads (stored in a blob store / file system / database) with derived keys (DK).
Of course, key rotation becomes super cumbersome, so you may choose to employ “envelope encryption”, where instead of encrypting / decrypting with DK, you encrypt / decrypt “data keys” (key wrap), with which the actual payload can be encrypted / decrypted. Key rotation then only involves re-encrypting “data keys” instead of the entire payload.
Thanks for sharing! That sounds like a very reasonable approach, but the article you linked on HKDF is exactly why I’m wary about rolling my own solution for this haha…
Vault seems like a giant pain to deploy on Fly, especially if we’re looking for a region agnostic setup to actually take advantage of being on Fly, but it would allow me to sleep better at night knowing customer secrets are stored and accessed using state of the art best practices.
I did actually learn about Doppler from a customer a while back, but as far as I could see they seem to be a platform for handling internal secrets (API keys and such for services used by our own eng teams, for instance), not for handling secrets that customers store on our platform so we can access private resources to perform tasks for them (i.e. github access/refresh tokens to access repos, npm access tokens to access private packages/registries, arbitrary env vars for deployment, etc)? Infisical is another similar product I heard about that just came out of YC a while back.
Can either of these guys be used to store customer-provided secrets somehow? Maybe I’m missing something.
Hey everyone! Founder of Infisical here (https://infisical.com). We’re an open source secret management platform – helping with everything from securely storing secrets, to automatically distributing them across your infra, to preventing secret leaks, etc. If you have any questions, feel free to join our community Slack (https://infisical.com/slack).
Also, yes, Infisical can be used to store customer-provided secrets. In fact, within every project, you can create a folder per each of your customers using our API.
I’d love to do that using the API but I don’t see any means of doing that from your API docs. It just retrieves all the secret for an environment and not based on folder/path. Perhaps, please give pointers to how this can be done using the API to create and retrieve secrets to a folder.
I’m also considering Doppler and will explore it. I’m not sure if they have support for folder. otherwise, with their pricing, I think one can create projects for each customer. Since unlike Infiscal, they don’t charge per project
Hey all, founder of Doppler here! First I want to take a moment to thank everyone on the thread who has mentioned Doppler, it means a lot to our team (we are chatting about it in Slack)!
We have a fair amount of customers who store customer secrets in Doppler. Each Doppler project support configs (groups of secrets scoped to an environment: dev, staging, prod). Configs inherit their secrets from the environment. This is particularly valuable when most of the secrets in an environment are going to to be the same across customers with a couple differing. What we typically see is a config is created per customer and then the specific secrets that are different per customer are overridden. If there are no shared secrets, we can handle that case by leaving the environment secrets empty. You can do all of this easily through our API, CLI, and Terraform.
Thanks for the answer. In a case where by customer secrets are unique to each customer, is config still an optimal solution? From what I saw in the docs just now, it seems secrets in a config are shared, with the values being different.
Yup configs are still an optimal solution! It depends on how unique the secrets are. If the secret names are the same, you can add the names into the environment and keep the values blank. If the secret names are not shared, then you can keep the environment secrets empty and just add the customer secrets to the configs.
