Organization-scoped Wildcard Certificates

Hi everyone,

Curious if anyone has any advice for my situation. I would best describe my need as a re-usable wildcard certificate scoped to organization-level rather than app-level. Specifically:

  • I will be creating and destroying apps frequently.
  • I need custom domains for each of them.
  • It is not plausible for me to issue individual certificates for each app (both cost-wise and scale-wise e.g. Let’s Encrypt ratelimits)
  • My needs mean I cannot use the fly-replay header, specifically due to the 1 MB limit not being flexible enough.
  • I looked at terminating TLS myself inside each app, but support for this would require I use IPv6 addresses which are not widely supported enough for my use-case.

My ideal solution here is being able to create the wildcard certificate ONCE and being able to allocate it to multiple apps. Is there an idiomatic fly-orientated way to solve this?

I am not on the Product team and so you should not take the fact I’m an employee as giving me all that much authority on this question, but I do have a thought:

Have you reviewed the design space enough to be sure that creating and destroying Fly Apps is what you want? If I hear “creating and destroying frequently”, that makes me think that you’re creating and destroying things in response to user input. I can tell you that we more often discuss creating and destroying Fly Machines within an app as the way to create compute resources in response to user input. What are you looking for that a Fly App gives you but a Machine does not?

Hey @john-fly, thanks for your response. You may be right that I am missing something - I’ll provide more information and perhaps you can clarify.

I want to be able to:

  • Create something (intentionally not using the word app or machine here!) that is uniquely addressable via custom domain in response to user input. E.g.
  • Have this thing be isolated from other things like it e.g. I don’t want load balancing across them.
  • Have that thing be created and running a docker image with some varying args (all things need to run the same docker image, but with different args)
  • Have the hostname for this thing be created on-the-fly. Only setup involved would ideally be pointing the DNS record to the shared IPv4.
  • Subsequently bring this thing down again when the user is finished with it. On average, this would be a couple of hours.

As far as I can tell, I need to create and destroy apps simply because of the isolation - if I create machines within an app, incoming traffic is load-balanced across them, no?

Let me know if I’m being silly :slight_smile: thanks!

So first of all I want to congratulate you on your design, because this is exactly the type of design that we are building our platform for, so this question has prompted some good internal discussion.

Maybe you know all this already, but this design exceeds the “Fly Launch” PaaS experience that we offer, and you’d be better suited to using the Fly Machines API directly, either through the CLI or through the HTTP API. If you haven’t read it yet, you should give a close read of fly machine run and, if you’re using the HTTP API, Create a Machine with services. Most of what you ask for is easy: varying args, automatic Machine destruction, isolation. It’s all there.

The one tricky part is DNS. You have your finger right on the difficulty: there are a couple different ways to handle this but all are awkward in different ways. I’ll give you three designs. Two are doable right now, but awkward. The third is probably ideal, but we’re… not quite there yet (but very much want to be).

Option 1: One Fly App, with additional custom Proxy

In this design, you’d create all your user’s ad-hoc Machines within the same App. You’d then also create a proxy App which would receive connections from the Fly Proxy and pass them along to the right Fly Machine. It would look like this:

User <==> Fly Proxy <==> Your custom proxy app <==> The Machine you started for that user

The advantages of this design are that it’s doable right now, it allows Fly Proxy to handle TLS and you can just set a wildcard DNS entry. The disadvantages are that the cost is somewhat higher because you need to pay for the proxy app machines, and the indirection caused by the custom proxy means that one of’s main features, global availiblity, is compromised (unless you spin up a custom proxy in every region). If you have a user in Australia but your nearest custom proxy is in Europe, then packets have to go to Europe and back, even if you spin up the user-specific Fly Machine in Australia.

This is the design I would probably implement if I had to solve this problem right now.

Option 2: Multiple Apps with TLS passthrough

In this design, every time you’d spin up a new Machine for a user, you’d do that in its own Fly App. You’d pay for an exclusive IPv4 address (it doesn’t have to be IPv6) for that App, and you’d disable Fly Proxy TLS termination. Instead, you’d terminate TLS yourself in the Machine with a wildcard certificate you’d provide yourself.

The advantages of this design are that it’s doable right now and it takes greater advantage of the Fly Platform. The disadvantages are that it’s more expensive and that you’ll have to update your DNS with custom A entries for every subdomain.

Option 3: One App & Fly Proxy routes on Domain

As part of a Fly App, you define various Services, and not every Machine has to provide every Service. This is true for Fly Apps defined through Fly Launch, as well as through the Machines API. But right now, you can only differentiate services by their port. If it were possible to distinguish services based on Domain Name, then for each of your users, you could spin up a Machine which declared a service only for that particular user’s custom domain name.

This is ideal. It’s the lowest cost of these three designs, while taking advantage of all the platform’s features, and also allowing you to only set a wildcard DNS entry. The only difficulty is… we haven’t released this feature yet. But it’s in development!

My suggestion would be to begin building the first design. Then when we have the feature described in option three implemented, it will not be that difficult to transition out. But this is a lot to dump on you and I’m sure you’ll have followup questions. I’ll be following this and be happy to help you answer them.

1 Like

Hi @john-fly - thank you very much for the detailed response and for taking the time to discuss with the team, I’m sure you’re all very busy so I do greatly appreciate it.

As you’ve presented them, Options 1 and 2 are pretty much exactly where I ended up in my original investigation, so I’m glad I was on the right track. Thanks for the summary on them.

Option 3 is, as you say, clearly the best. I’m not in a rush to build yet so can thankfully wait for the feature to become available!

Just a few quick questions if you have capacity to answer:

  • Is Option 3 implemented as SNI routing or will it require HTTP traffic for the Host header?
  • Any ideas on release timeframe? :grin:

Thanks again for the detailed write-up!

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.