I’m still searching for the right way to implement multi tenancy using fly.
One org per tenant sounds good in terms of isolation. However, it’s currently not possible to setup billing for organisations programmatically.
Would it be possible to create multiple private networks without creating an organisation for each?
Otherwise, would it be possible to configure billing once for multiple orgs?
Any other ideas?
We do support multiple private networks, but haven’t exposed a way to create them yet. That’s probably the best way to do this, we’ll see if we can get network names exposed through the create app mutation.
It’s now later this week: when you create an app with GraphQL, you can now pass a
network field. This will attach it to a network namespace for the org. You can use any string value that can be a subdomain here (ie:
It’s not available in
flyctl yet, we have more work to do to make this a first class feature, but it will work for you what you need! The caveat is: Wireguard peering will only use the default network on your organization.
Hey, Jens, can I bug you to say a bit more about what the architecture you’re thinking about is? What needs to be shared between the (isolated, per-your-customer) networks you’d be creating? Do we need to be able to peer between networks? Is that all-or-nothing or does it need to be configurable?
We’re noodling about this on Slack right now, is why I’m asking.
I want to create a custom network per tenant. For each tenant, I’d like to deploy multiple apps across the globe, all should see each other. No tenant should see internal services of other tenants. Some apps will be publicly exposed. Additionally, I’d like to allow tenants to join external services into their wireguard network. The idea is that tenants can have non-exposed services in their clouds which they can use from within the wireguard network.
A bit more context if that helps. I’d like to create an API mesh. Use any service you want to form a mesh of federated APIs over a secure network. Then have gateways on the edge to protect the APIs and cache responses.
As we’re already making it complicated. I was thinking about an easy way for tenants to add services to their mesh. The obvious choice would be to deploy to fly, too. In that case, would it be possible to allow apps from other orgs to join another internal network? I guess I wouldn’t be the only one, interested in this feature. If you find a generic enough solution to connect anything with anything plus a way to define access policies, that would enable a lot of use cases.
So there’s, in this design, no explicit need for tenant-A (on network-A) to ever talk to any Fly service on tenant-B’s network-B, or for that matter any other Fly network? There’s no bridging, at all, between networks? You just need it to be straightforward to deploy things into those separate networks, and to pass a network to
flyctl wireguard (or its GQL mutations or whatever).
As I said above, it would be desirable to have a bridge between networks. I want to operate proprietary encapsulated software inside my own org. Each tenant should have their own network. However, if tenants want to run their own workloads on fly, they should be able to connect to “their” network if possible so that the edge gateways can securely talk to their apps while I can operate the gateway stack separately. I don’t want to wrap fly completely to allow them to schedule their own apps. My focus is on APIs alone. If that’s too complicated, it’s not a deal breaker. Connecting peers to custom networks is a good start.
Ok, let’s see how big a deal adding multi-network support to Fly WireGuard will be; I’m guessing it’s pretty easy. I’ll get back to you here.
Hey @thomas, what’s the state of this?
It’s still on the docket, but I don’t know where it is. The backend for it exists, but I don’t think it’s hooked up to the API yet. It’ll happen!
You can put an app on a specific network-id (the org your app is on has a default network, but you can create others) with
flyctl apps create --network.
Apps on their own network won’t be able to talk to apps on other networks. There’s no current bridging you can do from one network to another; our private network enforcement is very simple (which is why we trust it).
Wait…what? This is uber neat. When was this feature announced?
I regret that I may have revealed its existence here, because I hate it. But it’s not going away (it’s useful with Fly Machines for large-scale multitenant stuff).
How can this be done for when you’re creating an app via the machine Rest API?
You can pass a
network param, there’s an example here: Running user code on Fly Machines · Fly Docs
Great to learn about custom private networks! Is there a way to limit the data transfer amount of a certain app? I feel like this would be useful for FaaS.
How do you specify a specific network ID for a Postgres Cluster? It is not provided as a mutation param in the GraphQL service.
I replaced the createPostgresCluster mutation with the createApp mutation followed by the createVolume mutation. This solves my issue.