I can't access `/.fly/oidc_token` from a container

containers
, ,
1

I’m using Fly containers via machine-config=, it’s working as expected (almost) but I can’t access the /.fly/oidc_token file that I need for a specific service.

[info]Error: operation error KMS: GetPublicKey, get identity: get credentials: failed to refresh cached credentials, failed to retrieve jwt from provide source, unable to read file at /.fly/oidc_token: open /.fly/oidc_token: no such file or directory

On fly.toml, I defined:

[env]
  AWS_ROLE_ARN = 'arn:aws:iam::__REDACTED__:role/my-role'

When connected to the main containers via fly ssh console, I can see other AWS_ variables that were defined by fly stack (I assume).

# env | grep AWS_
AWS_ROLE_ARN=arn:aws:iam::__REDACTED__:role/my-role
AWS_ROLE_SESSION_NAME=fly-containers-test@6e822357a75d08
AWS_WEB_IDENTITY_TOKEN_FILE=/.fly/oidc_token
# ls -lhga /.fly/oidc_token
ls: cannot access '/.fly/oidc_token': No such file or directory

Should I do anything special here ?

Reference: Using Containers with Flyctl - #17 by endersonmaia

1 Like
3

Timing is perfect as I just got answers.

The good thing about containers is that they provide isolation. The bad thing about containers is that they provide isolation. In this case, you don’t have access to /.fly/oidc_token.

Instead you need to use our API to obtain this information:

Here is an example of how you would use this with AWS:

curl -f --unix-socket /.fly/api http://localhost/v1/tokens/oidc \
  -d'{ "aud": "sts.amazonaws.com", "aws_principal_tags": true }' \
  -XPOST > $AWS_WEB_IDENTITY_TOKEN_FILE
4

It works, but seems that it’s also accessible from other containers.

For ex.:

I define 3 containers, but want only one of them to have AWS credentials, when I run the command that fills the /.fly/oidc_token this file will be acessible for other containers.

I understand that some part of the fly machine init process is already preparing the environment, since I get AWS_ROLE_SESSION_NAME, and AWS_WEB_IDENTITY_TOKEN_FILE environment variables created by only defining the AWS_ROLE_ARN, but the file is not filled with the content when I use containers.

5

Is it not possible to run this command in the container that you want to have access and put the result in some place other than the /.flydirectory?

6

Sure, its possible.

I was just thinking about having a consistent way of managing the containers like we manage machines, being transparent.

But your suggested solution does the job, and is already marked as solved. :slight_smile:

Thanks!