What are some ways to prevent someone from accidentally destroying an app?

I’m working on transitioning my company’s hosting to Fly.

At one point I am going to add the other developers to the organization. Besides cautioning people not to mess around with fly apps destroy, what are some ways I can prevent someone from accidentally doing something like fly apps destroy my-production-database? And what happens if someone does that by accident?

This is possible with org/deploy tokens, but it’s a bit complicated. Quick overview:

  • Org tokens are limited to managing a single org and its resources
  • Deploy tokens are limited to managing a single app and its resources

From the sounds of it, an org token most suits your needs, but just letting you know it’s possible to do a similar thing with a deploy token if needs be.

First, create an org token. You have to do this interactively at the moment, there’s no way to pass the organization as a flag.

fly tokens create org

Now we’re going to attenuate the token using fly tokens attenuate. This command takes a token to attenuate and a list of caveats to append. If the terminology here is confusing, and you’re curious, have a read of some of the links I’ve left at the bottom.

Take a look at the output of fly tokens debug -t ORG_TOKEN using the org token you’ve just generated. In my case, I see:

[
  {
    "location": "https://api.fly.io/v1",
    "caveats": [
      {
        "type": "Organization",
        "body": {
          "id": 341243,
          "mask": "rwcdC"
        }
      },
      ...
    ]
  },
  ...
]

The relevant part here is the Organization caveat, which has the mask rwcdC corresponding to: read, write, create, delete, control. For a full description of what these mean see this stuff.

The important point here is that this org token has d in the mask, so the owner of this token can destroy resources associated with the org. So we want to add an Organization caveat with the mask rwcC, so that we can still do all the stuff other than deleting.

There was a bit of reading of source code in the macaroon repo to figure out how to create a valid payload for the fly tokens attenuate command, but the result is understandable:

[
  {
    "type": "Organization",
    "body": {
      "id": 341243,
      "mask": "rwcC"
    }
  }
]

The id I’ve just taken from the output of fly tokens debug above. The payload is a list, because the command expects a list of caveats.

So now we can attenuate. FYI: -f is the file to read caveats from. I’m just using bash redirection <() to directly pass a string.

fly tokens attenuate -t "$ORG_TOKEN" -f <(echo '[{"type": "Organization", "body": {"id": 341243, "mask": "rwcC"}}]')

Now if I try to destroy an app:

FLY_API_TOKEN="$ATTENUATED_TOKEN" fly apps destroy APP_WITHIN_THAT_ORG
Destroying an app is not reversible.
? Destroy app APP_WITHIN_THAT_ORG? Yes
Error: Not authorized to destroy this app.

So I guess the idea would be to mint tokens like this for each of your developers and then require that they use them in this way. They could export FLY_API_TOKEN in their .bashrc if they want to avoid always having to type it out as above.

Useful resources for understanding macaroons:

edit: I’ve just added a --org flag to fly tokens create org, so this can be done non-interactively now.

2 Likes

Thanks, I was able to write a script based on your instructions that automates the process somewhat.

Ideally IMO this should be a self-service part of the Team website interface. That way I could just send invites with pre-determined no-destruction roles to everyone, and they’d figure out fly auth login etc.

Yeh we definitely have much bigger ambitions for this stuff than is currently possible. Glad it worked for you, and thank you for the feedback!

@jfent You’re welcome! Two followup questions:

  1. The updated token has two caveats with the same ID (rwcdC and rwcC), is that expected? I can imagine some ambiguity would follow if the backend e.g. searches caveats by ID.
  2. Are org/deploy tokens literally limited to deploy-related commands? I read through the related links and it seems that way. When I create an org token and try to do something like fly postgres connect -a ..., I get a “Not authorized to manage this organization::feature.” error. If it wasn’t clear from my post, what I really wanted was a token equal in everything to a personal one, except for the ability to destroy stuff. I still need people to be able to ssh etc.
  1. Yes this is fine and down to how Macaroons work. If you’re curious about this, the “Macaroons” section of the blog post I linked above explains it further. Here’s an image from the post that illustrates basically the same idea that you’re referencing:

The macaroon originally allowed rwx on all files, has been attenuated to only allow r on cat.jpg, but still keeps the earlier “allow all” caveat. It’s built into the spec of macaroons that you can’t remove earlier caveats, you’re always just attenuating.

  1. No they’re not but unfortunately having done some investigation it looks like at the moment you need delete capability in order to do certain things, for example fly ssh console (fly pg connect is basically syntactic sugar for fly ssh console).

If you’re able to come up with an explicit list of commands (fly ssh console, fly pg connect, …) that you’d like to be invokable with a token like this, I can give you a better idea of how close to possible it is currently. If it’s close, we might be able to push the needle just enough to make this viable for you. I can’t promise anything though!

1 Like

Thanks for the detailed response!

The real answer is I’m not sure what commands I’m going to need; the only certainty I have is that it would be nice to have personal tokens without deletion capability.

fly ssh console, fly pg connect and fly proxy probably covers 90% of our usage, but I don’t want to scramble whenever we eventually need to restore a snapshot or something, so I’m going to go with the personal tokens for everyone after all.

No problem and yeh totally understandable. If after a while you discover that you do have a pretty consistent set of commands, drop a message in this thread or tag me in a new one and we can pick it back up!

1 Like

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.