So first of all I want to congratulate you on your design, because this is exactly the type of design that we are building our platform for, so this question has prompted some good internal discussion.
Maybe you know all this already, but this design exceeds the “Fly Launch” PaaS experience that we offer, and you’d be better suited to using the Fly Machines API directly, either through the CLI or through the HTTP API. If you haven’t read it yet, you should give a close read of fly machine run and, if you’re using the HTTP API, Create a Machine with services. Most of what you ask for is easy: varying args, automatic Machine destruction, isolation. It’s all there.
The one tricky part is DNS. You have your finger right on the difficulty: there are a couple different ways to handle this but all are awkward in different ways. I’ll give you three designs. Two are doable right now, but awkward. The third is probably ideal, but we’re… not quite there yet (but very much want to be).
Option 1: One Fly App, with additional custom Proxy
In this design, you’d create all your user’s ad-hoc Machines within the same App. You’d then also create a proxy App which would receive connections from the Fly Proxy and pass them along to the right Fly Machine. It would look like this:
User <==> Fly Proxy <==> Your custom proxy app <==> The Machine you started for that user
The advantages of this design are that it’s doable right now, it allows Fly Proxy to handle TLS and you can just set a wildcard DNS entry. The disadvantages are that the cost is somewhat higher because you need to pay for the proxy app machines, and the indirection caused by the custom proxy means that one of Fly.io’s main features, global availiblity, is compromised (unless you spin up a custom proxy in every region). If you have a user in Australia but your nearest custom proxy is in Europe, then packets have to go to Europe and back, even if you spin up the user-specific Fly Machine in Australia.
This is the design I would probably implement if I had to solve this problem right now.
Option 2: Multiple Apps with TLS passthrough
In this design, every time you’d spin up a new Machine for a user, you’d do that in its own Fly App. You’d pay for an exclusive IPv4 address (it doesn’t have to be IPv6) for that App, and you’d disable Fly Proxy TLS termination. Instead, you’d terminate TLS yourself in the Machine with a wildcard certificate you’d provide yourself.
The advantages of this design are that it’s doable right now and it takes greater advantage of the Fly Platform. The disadvantages are that it’s more expensive and that you’ll have to update your DNS with custom A entries for every subdomain.
Option 3: One App & Fly Proxy routes on Domain
As part of a Fly App, you define various Services, and not every Machine has to provide every Service. This is true for Fly Apps defined through Fly Launch, as well as through the Machines API. But right now, you can only differentiate services by their port. If it were possible to distinguish services based on Domain Name, then for each of your users, you could spin up a Machine which declared a service only for that particular user’s custom domain name.
This is ideal. It’s the lowest cost of these three designs, while taking advantage of all the platform’s features, and also allowing you to only set a wildcard DNS entry. The only difficulty is… we haven’t released this feature yet. But it’s in development!
My suggestion would be to begin building the first design. Then when we have the feature described in option three implemented, it will not be that difficult to transition out. But this is a lot to dump on you and I’m sure you’ll have followup questions. I’ll be following this and be happy to help you answer them.
thanks!