Here at Fly.io, we’re pretty big fans of Tailscale for a lot of reasons, and we especially like the simple, identity aware network access control. We’re also pretty big fans of Fly.io, and we have a whole stable of internal applications and services deployed on our platform that we need to use every day.
We found ourselves with a constant need to connect these deployments to our Tailnet, and each time we solved it a slightly different way - leading to duplicated effort and inconsistency.
So we came up with a solution that was stateless, easy to understand and maintain, as well as providing us the reliability and security that we needed to control access to our apps, so we wanted to share it with you.
Our approach
The pattern we decided on was to utilise Tailscale subnet routing to expose an organisation’s 6PN to the Tailnet, and then leverage Tailscale’s powerful ACL system to manage access to it.
This works well enough if you have a small deployment within one organisation. As you may know, all apps within an organisation are network-accessible via its 6PN, and it’s simple to share that connectivity on to the Tailnet.
But because we have a number of organisations, we took it one step further and utilised a low-key yet amazing feature of the Fly.io platform - Flycast! This allows a Fly app to securely provide internal-only, load-balanced, region-aware connectivity to a service via your Fly.io 6PN, even across organisations. As a bonus, thanks to the Tailnet and Fly.io’s 6PN, your traffic is encrypted in transit.
Here’s a fun little diagram to show how we’re putting this together:
First, the one-off setup for the infrastructure we need!
Tailscale: Define a new Tag
Define a tag in your Tailnet policy file:
{
// (other tailnet policy file entries here)
"tagOwners": {
"tag:fly-router": [],
}
}
Tailscale: Create an OAuth client
Follow the instructions for creating an OAuth client in Tailscale.
Allow the OAuth client Read and Write permissions for the Devices scope, and provide the tag:fly-router tag as created in the previous step.
This allows the OAuth client to manage devices with the fly-router tag, including creating auth keys to join devices to the Tailnet under that tag.
You can use an access key instead, but that will be tied to a user and expire after 90 days after which you will have to renew it and re-deploy.
Fly.io: Create a new organisation
$ fly orgs create tailscale-router
New organizations start on the $5/mo Hobby Plan.
Your organization tailscale-router (tailscale-router) was created successfully. Visit https://fly.io/dashboard/tailscale-router/billing to add a credit card and enable deployment.
Visit the URL and add your credit card if required.
Fly.io: Create a new application
$ fly apps create -o tailscale-router
? Choose an app name (leave blank to generate one):
New app created: twilight-moon-8960
Fly.io: Set the OAuth key as a secret
$ fly secrets set -a twilight-moon-8960 --stage TS_AUTHKEY="tskey-client-<id><key>"
Fly.io: Deploy the Tailscale image
Use this simple fly.toml to deploy the official Tailscale container image:
app = "twilight-moon-8960"
[build]
image = "tailscale/tailscale"
[processes]
app = "export PATH=$PATH:/usr/local/bin; containerboot\""
[env]
# TS_AUTHKEY set via secret
TS_HOSTNAME = "fly-router"
TS_EXTRA_ARGS = "--advertise-tags tag:fly-router"
# Leave this commented for now:
# TS_ROUTES = ""
Run fly deploy. You should see the host fly-router appear in your Tailscale device list. If you run in to trouble, start with fly logs.
Fly.io: Figure out your 6PN subnet
Run fly ips private and look at the IP that is returned, this address is in your 6PN which we want to route via Tailscale. For example:
ID REGION IP
9829fc10a47998 syd fdaa:3:d3ad:beef:192:2ef7:abcf:2
From here, you can decide to route your entire 6PN (e.g. if you’re deploying this in to an existing organisation with apps you want to provide access to), or the subnet within the 6PN where services are Flycast in to:
- Full 6PN:
fdaa:3:d3ad::/48 - Flycast only:
fdaa:3:d3ad:0:1::/80
Update your fly.toml to set the TS_ROUTES environment variable to tell tailscaled which subnet it is responsible for, and then re-deploy the application.
From here on, we’ll assume you chose to expose the Flycast subnet.
Optional: Tailscale: Automatically approve routing:
Normally, when a Tailscale device boots and wants to advertise a route, you have to approve it via the console. You can optionally configure Tailscale to allow this device to advertise its route without approval:
{
// (other tailnet policy file entries here)
"autoApprovers": {
"routes": {
"fdaa:3:d3ad:0:1::/80": ["tag:fly-router"],
},
}
}
That’s all the setup done! Now, for each app you want to access via Tailscale, you can perform the following steps from the app:
- For the app you wish to expose to your tailnet, use
fly ipsto release any public addresses your app may have assigned. - Make sure your app has a service block that defines its services.
- Flycast your app in to your
tailscale-routerorganisation and note the address:fly ips allocate-v6 --private --org tailscale-router- nb: You currently need to be a member of both organisations to accomplish this
You’ve now Flycast’d a Fly.io app in to a subnet that is routable from Tailscale. It won’t work yet, but that’s all the Fly config required. From there, the following Tailscale configuration will need to be made by a Tailscale network admin:
- Define an entry for the application in the
hostssection of the ACL file pointing to the Flycast’d 6PN address. - Define an entry in the
aclssection of the ACL file to govern who may access the application.
Something like this:
{
// (other tailnet policy file entries here)
"hosts": {
"fly-app-1": "fdaa:3:d3ad:0:1::2", // Flycast address from step 3
},
"acl": {
// (other ACL entries here)
{
"src": ["group:application-users"],
"dst": ["fly-app-1:443"],
"action": "accept",
},
}
You should now be able to access your application from a Tailscale-connected device by using the address you were given in step 3. Set yourself a nice DNS record and you are good to go!
Extra credit
Failover and Load balancing
We currently have one Tailscale subnet router providing access to all of our apps. Fly machines are transient, so it would be nice to have some redundancy here. Easy enough!
fly scale count 2
You will now see two devices in Tailscale, fly-router and fly-router-1. Depending on your plan, these instances may be Failover or may be Load balanced, the details are in Tailscale’s documentation.
Regional Routing
Wait, did that page say Regional routing? We’re freakin’ Fly.io, of course we want to do that! Simply scale out your Tailscale router app to the regions you want to cover:
fly scale count 4 -r syd,iad,lhr,sin
And before you know it, you’re covering the world. Tailscale will use its DERP subsystem to route a client’s traffic to the nearest subnet router, and Fly.io’s Flycast will route from that Tailscale router to the nearest application instance. Hot stuff.
Some fixes and features we want to allow for down the line:
- Currently uses userspace networking, find out why
TS_USERSPACE=0fails on Fly machines - An easy way to provision DNS and TLS certificates - can’t use MagicDNS or
tailscale certfor subnet-routed services. - Construct a Macaroon that would allow users of specific orgs (and only those users) to Flycast services in to the
tailscale-routerorganisation.
Hope this comes in handy for those that heavily rely on Fly.io and Tailscale! Happy to answer any questions about this approach, or where the other ones fell short for us. Let us know if there’s any other Tailscale tips or tricks that have worked for you!
