How to create attenuated token with practical validity longer than 10½ min?

I’m running a Celery auto scaling system on Fly.io that uses the Machines API to start/stop worker pools based on queue depth. I created an attenuated token with just rC permissions (read + control) so that my service can read machine state (r permission) and start and stop machines (C Permission) based on load.

Problem seems to be, i have this scoped to my org and app and it works. but it seems that fly.io adds a 3rd party caveat to this token that limits its validity to 10.5 min. my understanding is that i cant hit the /aaa endpoint to get a new discharge macaroon without auth’‘ing with having OAuth session credentials for an account which i don’t want to expose within a running host.

is there a way around this?

Hm… I don’t recall ever seeing an unrequested time limit imposed by attenuation, actually. (I had one that was used for weeks and weeks, in the past.) I just tried anew, and it’s fully usable 90 minutes later:

$ FLY_API_TOKEN="$(fly tokens create deploy --name base-of-rC --expiry 2h)" \
  fly tokens attenuate <<'EOF'
[
  {"type": "Machines",
   "body": {"machines": {"": "rC"}}}
]
EOF

(Assumes Bash syntax.)

Perhaps yours was unknowingly based on a login-style token, instead? I know those have shorter expiry, although 10 minutes is a little surprising…

Spent a bit of time digging into this

  1. Org tokens - Same 3P caveat, 10.5min expiry ✗
  2. Deploy tokens - Same 3P caveat, 10.5min expiry ✗
  3. Readonly tokens - Only read access, can’t start/stop machines ✗
  4. Machine-exec tokens - For executing commands IN machines, not API control ✗

Macaroon authentication system Fly.io uses as i understand it:

Example simple caveat:
“This token can only be used for organization ID xxx App YYY”
“This token expires on 2027-01-20”
“This token can only read/control (rC), not create/delete”

  [                                                                                                                                                                               
    {                                                                                                                                                                             
      "type": "Organization",                                                                                                                                                     
      "body": {                                                                                                                                                                   
        "id": xxxx,                                                                                                                                                            
        "mask": "rC"                                                                                                                                                              
      }                                                                                                                                                                           
    },                                                                                                                                                                            
    {                                                                                                                                                                             
      "type": "Apps",                                                                                                                                                             
      "body": {                                                                                                                                                                   
        "apps": {                                                                                                                                                                 
          "yyyy": "rC"                                                                                                                                                        
        }                                                                                                                                                                         
      }                                                                                                                                                                           
    }                                                                                                                                                                             
  ]                                                                                                        

These are first-party caveats - restrictions checked by Fly.io themselves.

Fly.io Adds 3rd party caveats to this when you attenuate them.

Third-Party (3P) Caveats

3P caveats are restrictions that require another service to verify.

The flow:

  1. You present your token to Fly.io
    Token says: “I’m valid, BUT you must also prove user auth with api.fly.io/aaa

  2. Fly.io sees the 3P caveat and says: “Okay, prove you’re authenticated”

  3. You must get a “discharge macaroon” from https://api.fly.io/aaa/v1

    • This discharge proves you’re an authenticated user
    • The discharge has a ValidityWindow (10.5 minutes)
  4. You send BOTH tokens together:
    Authorization: Bearer fm2_,fm2_

  5. Fly.io checks:
    ✓ Original token is valid
    ✓ Discharge token is valid from the 3P service
    ✓ ValidityWindow hasn’t expired
    → Request allowed

How this process works.

When you run:
fly tokens create deploy -a appname -x 8760h

Fly.io creates a token that says:

  • “Valid for 1 year”
  • “BUT also requires 3P authentication from api.fly.io/aaa

The 3P service (api.fly.io/aaa) is Fly.io’s authentication/authorization service. It issues discharge tokens that:

  • Prove you’re an authenticated user (not stolen/leaked token)
  • Expire quickly (10.5 minutes) for security
  • Are meant to be refreshed by human users logging in

What’s in the Tokens

When i worked to debug the tokens, we get

Token Part 1 (from api.fly.io/v1):

                                                                                                                                       
  {                                                                                                                                                                               
    "type": "Organization",                                                                                                                                                       
    "body": {"id": x, "mask": "rC"}  // First-party: org scoped                                                                                                          
  },                                                                                                                                                                              
  {                                                                                                                                                                               
    "type": "3P",  // Third-party caveat                                                                                                                                          
    "body": {                                                                                                                                                                     
      "Location": "https://api.fly.io/aaa/v1",  // "Check with this service"                                                                                                      
      "Ticket": "OOC5H3gf..."  // Cryptographic ticket to exchange                                                                                                                
    }                                                                                                                                                                             
  }                                                                                                                                                                               

Token Part 2 (discharge from api.fly.io/aaa/v1):

  {                                                                                                                                                                               
    "type": "ValidityWindow",  // THIS IS THE PROBLEM                                                                                                                             
    "body": {                                                                                                                                                                     
      "not_before": 1768805891,                                                                                                                                                   
      "not_after": 1768806521   // Expires in 10.5 minutes                                                                                                                        
    }                                                                                                                                                                             
  }                                                                                                                                                                               

The deploy token that I created lasted for more than 90 minutes, as I said, so this really isn’t true in general.

How old is your account? Perhaps this is a limitation of the Free Trial…

not a free account running about 45 machines, its only this token, it happens consistently re-generated it about 5 or 6 times.

if you run a fly tokens debug token you get

fly tokens debug n10jxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
[
  {
    "location": "https://api.fly.io/v1",
    "caveats": [
      {
        "type": "Organization",
        "body": {
          "id": ########,
          "mask": "rwcdC"
        }
      },
      {
        "type": "3P",
        "body": {
          "Location": "https://api.fly.io/aaa/v1",
          "VerifierKey": "dMxV2xyLeJxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx",
          "Ticket": "OOC5xxxxxxxxxxxxxxxxxxxxxxxx"
        }
      }
    ]
  },
  {
    "location": "https://api.fly.io/aaa/v1",
    "caveats": [
      {
        "type": "ValidityWindow",
        "body": {
          "not_before": 1768879806,
          "not_after": 1768880436
        }
      },
      {
        "type": "FlyioUserID",
        "body": xxxxxxxxxxxx
      },
      {
        "type": "IsUser",
        "body": {
          "uint64": xxxxxxxxxx
        }
      }
    ]
  }
]

now with a one liner

fly tokens debug n10jxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx| jq -r '.[] | select(.location == "https://api.fly.io/aaa/v1") | .caveats[] | select(.type == "ValidityWindow") | .body | (((.not_after - .not_before) / 60) | tostring) + "minutes"'                                                                                                                                                                       

you get 10.5minutes

Ah… I’m pretty sure it’s ignoring the positional argument (the n10j... string).

When I did fly tokens debug tomato, I also got a 10½ minute validity window.

(And not the expected no such token error, :tomato:.)

Try this instead:

$ FLY_API_TOKEN="$(fly tokens create deploy --name throwaway --expiry 2h)" fly tokens debug

(Bash syntax again.)

The fly tokens machinery wants you to do this kind of manipulation with the -t knob or the FLY_API_TOKEN environment variable, otherwise you’re unknowingly operating on the short-lived login Macaroon, I’d guess. It’s pretty non-intuitive…

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.