We do support multiple private networks, but haven’t exposed a way to create them yet. That’s probably the best way to do this, we’ll see if we can get network names exposed through the create app mutation.
It’s now later this week: when you create an app with GraphQL, you can now pass a network field. This will attach it to a network namespace for the org. You can use any string value that can be a subdomain here (ie: customer-network-1245).
It’s not available in flyctl yet, we have more work to do to make this a first class feature, but it will work for you what you need! The caveat is: Wireguard peering will only use the default network on your organization.
Hey, Jens, can I bug you to say a bit more about what the architecture you’re thinking about is? What needs to be shared between the (isolated, per-your-customer) networks you’d be creating? Do we need to be able to peer between networks? Is that all-or-nothing or does it need to be configurable?
We’re noodling about this on Slack right now, is why I’m asking.
I want to create a custom network per tenant. For each tenant, I’d like to deploy multiple apps across the globe, all should see each other. No tenant should see internal services of other tenants. Some apps will be publicly exposed. Additionally, I’d like to allow tenants to join external services into their wireguard network. The idea is that tenants can have non-exposed services in their clouds which they can use from within the wireguard network.
A bit more context if that helps. I’d like to create an API mesh. Use any service you want to form a mesh of federated APIs over a secure network. Then have gateways on the edge to protect the APIs and cache responses.
As we’re already making it complicated. I was thinking about an easy way for tenants to add services to their mesh. The obvious choice would be to deploy to fly, too. In that case, would it be possible to allow apps from other orgs to join another internal network? I guess I wouldn’t be the only one, interested in this feature. If you find a generic enough solution to connect anything with anything plus a way to define access policies, that would enable a lot of use cases.
So there’s, in this design, no explicit need for tenant-A (on network-A) to ever talk to any Fly service on tenant-B’s network-B, or for that matter any other Fly network? There’s no bridging, at all, between networks? You just need it to be straightforward to deploy things into those separate networks, and to pass a network to flyctl wireguard (or its GQL mutations or whatever).
As I said above, it would be desirable to have a bridge between networks. I want to operate proprietary encapsulated software inside my own org. Each tenant should have their own network. However, if tenants want to run their own workloads on fly, they should be able to connect to “their” network if possible so that the edge gateways can securely talk to their apps while I can operate the gateway stack separately. I don’t want to wrap fly completely to allow them to schedule their own apps. My focus is on APIs alone. If that’s too complicated, it’s not a deal breaker. Connecting peers to custom networks is a good start.
You can put an app on a specific network-id (the org your app is on has a default network, but you can create others) with flyctl apps create --network.
Apps on their own network won’t be able to talk to apps on other networks. There’s no current bridging you can do from one network to another; our private network enforcement is very simple (which is why we trust it).
I regret that I may have revealed its existence here, because I hate it. But it’s not going away (it’s useful with Fly Machines for large-scale multitenant stuff).
Great to learn about custom private networks! Is there a way to limit the data transfer amount of a certain app? I feel like this would be useful for FaaS.