So you’ve got an app serving AI cat images generated from the weather forecast running as an ECS task in us-east-1 but customers in Australia and Europe have been complaining that the latency is too high. You quickly realise that this is going to require replicating your ECS tasks and ECR images into ap-southeast-2 and eu-central-1 while also setting up a load balancer to direct traffic. A daunting task.
You’ve been using Fly.io to host some of your other apps, however you need to be able to read the weather forecast and training data from an S3 bucket. Accessing this outside of AWS is going to involve creating an AWS user and setting the user’s key ID/secret as fly secrets within your app. Your security team has caught wind of your plan to create an AWS user and are asking questions about rotating credentials, sharing secrets and observability of Fly.io.
Suddenly using Fly.io is no longer as easy as running fly launch and it might actually be easier to setup your app in multiple regions in AWS. However you’ve just heard that Fly.io has launched an OIDC provider for machines with a slick AWS integration. You can now access AWS services from your Fly.io machines as easily as setting the AWS_ROLE_ARN and configuring Fly.io as an identity provider in your AWS account.
Note: You can also authenticate against other OIDC providers like Azure, Google Cloud Platform, and HashiCorp Vault we’re just not doing any magic for them yet.
How to Read from an S3 Bucket
Reading object from S3 in your Fly.io machines is as easy as:
- Creating a Fly.io app using
fly launchin the same place as your Dockerfile - Find the slug for your organisation with
fly orgs list - Create an OpenID Connect Identity provider in your AWS account with these settings (This only has to be done once per org):
Provider URL: https://oidc.fly.io/<org-slug>
Audience: sts.amazonaws.com
- Create an IAM Role when choosing the trusted entity select:
Web Identity -> Identity Provider -> oidc.fly.ioand select the AmazonS3ReadOnlyAccess policy. - Set the
AWS_ROLE_ARNas an environment variable in yourfly.tomland the AWS SDK will know how to do the rest.
[env]
AWS_ROLE_ARN = "arn:aws:iam::<ACCOUNT_ID>:role/<ROLE_NAME>"
- Deploy your app
fly deploy --region ams,iad,syd - Clone it into your other 2 regions
fly clone -r ams && fly clone -r syd
NOTE: If you’re setting this up for your personal org the slug is a little harder to find but you can find it in the url when you click on “Apps” on the dashboard.
How does this work?
How does this black magic work you ask? We’ve just rolled out a new endpoint in our machines api which mints OpenID Connect (OIDC) tokens for machines. You can grab one right now through the unix socket at /.fly/api by running
curl --unix-socket /.fly/api -X POST "http://localhost/v1/tokens/oidc" in any of your machines and get a JWT with the following claims:
{
"app_id": "3179105",
"app_name": "app-name",
"aud": "https://fly.io/<org-slug>",
"exp": 1712615659,
"iat": 1712615059,
"image": "image:latest",
"image_digest": "sha256:48933d82921c947df7858e84b841046bd7352b39b4705a00cf458704b73e46bf",
"iss": "https://oidc.fly.io/<org-slug>",
"jti": "f2cfdee1-becc-4d32-bcb9-af07a2b041a5",
"machine_id": "e286534eb706e8",
"machine_name": "machine-name",
"machine_version": "01HTZWZD0V5FYBCE1PGYKSD3YX",
"nbf": 1712615059,
"org_id": "288875",
"org_name": "org-slug",
"region": "sea",
"sub": "org-slug:app-name:machine-name"
}
This token can then be used to authenticate against any 3rd party which supports OIDC tokens. Third parties verify it’s contents against the OpenID configuration at https://oidc.fly.io/<org-slug>/.well-known/openid-configuration. The keen eye’d of your might have noticed our decision to include the org name in the issuer, this is a hardening feature offered at a premium on other platforms we’ve included by default.
We’ve also baked some extra magic into this if you’re using this to access AWS services. Nothing too fancy though, we write the token to a file at /.fly/oidc_token every 9 minutes to keep it fresh and set the AWS_WEB_IDENTITY_TOKEN_FILE and AWS_ROLE_SESSION_NAME environment variables. The AWS SDK handles the rest!
Why Use OIDC?
You might have also noticed that this is a feature also provided by GitHub actions. To summarise their documentation:
Using OIDC tokens allow for good security practices like:
- No Long-Lived Secrets: You don’t need to add your cloud credentials as long-lived secrets. Instead you can configure an OIDC trust to allow your Machines to request short-lived access tokens.
- Authentication(AuthN) and Authorization(AuthZ) management: You have more granular control over how machines can use credentials using your cloud providers authN and authZ tooling.
- Rotating Credentials: Our tokens are only valid for 10 minutes before they expire, ensuring these and your cloud credentials are rotated frequently.
Customizing token claims
Currently you can customize the audience (aud) claim of your tokens by providing a value for aud in the body of the POST request. This lets you specify the recipient of the token. This is what it looks like in a curl request.
curl --unix-socket /.fly/api -X POST "http://localhost/v1/tokens/oidc" --data '{"aud":"sts.amazonaws.com"}'
The sub claim follows the format org-slug:app-name:machine-name and is not customizable but if you’ve got a particular use case or there’s another claim you’d like to see added we’d love to hear about it!
Creating AWS Trust Policies
You can leverage the trust policy of an IAM role to restrict which machines in your organisation can assume a role in AWS. For example the following policy would only allow machines in the app foo-bar within your org to assume the role it’s attached to.
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"Federated": "arn:aws:iam::012345678910:oidc-provider/oidc.fly.io/<org-slug>"
},
"Action": "sts:AssumeRoleWithWebIdentity",
"Condition": {
"StringEquals": {
"oidc.fly.io/<org-slug>:aud": "sts.amazonaws.com",
},
"StringLike": {
"oidc.fly.io/<org-slug>:sub": "org-slug:foo-bar:*"
}
}
}
]
}
If anyone would like to see something similar for any other 3rd parties like GCP or Azure. I’d love to hear what your use cases look like!
but if you could confirm as soon as possible that would be great.