Migrating custom TLS from Cloudflare/AWS to Fly

An app currently uses CloudFlare spectrum (TCP passthrough) for DDoS mitigation. This forwards over the internet to AWS (Toronto), where nginx handles TLS termination with a lua openresty extension (for some 60k custom domains). Next, on to a small set of non-clustered varnish instances handling cache, and finally, a Rails app.

I’d like to look into how Fly could improve this setup. A few questions:

  1. Should moving TLS termination to the edge should help speed up connection time?
  2. Would it theoretically be faster to pass the unencrypted traffic to the core app through your network instead of using the current path over the internet of Cloudflare->openresty?
  3. I read your article on the 5-hour CDN, which seems like it would apply here for also moving cache to the edge. I’m interested in this because of the exorbitant quotes from CDN providers for hosting 60k custom domains. What would be a good way to estimate Fly costs given N specific regions?
  4. Related to 3: This app could be simplified by removing the TLS handling, but as mentioned, that comes at a high cost compared to the zero cost of LE certs used today. Would a volume discount be applicable for an application with this number of custom certs?
  5. Do you offer the same or similar kinds of DDoS mitigation as companies like Cloudflare or Fastly?

Hey this sounds fun!

  1. yes, moving tls to the edge will help quite a bit. TLS handshakes are one of the leading causes of slow page loads.
  2. Maybe theoretically faster but it’s hard to tell! We should do a better job of routing to an app than the internet at large.
  3. Fly costs are pretty simple, it’s $0.10 per cert we manage plus VM/disk usage. Someone with a similar workload as yours pays about $200/mo for VMs + disks in all our regions.
  4. Yes, we can work out a volume discount. Would you mind emailing me (kurt fly.io)?
  5. We have similar network level ddos protections in place. CloudFlare has a bunch of HTTP level mitigations (which you may not be using) that we don’t provide, but OpenResty is a nice place to solve some of that.

Thanks for the info.

This app uses AWS RDS and it’s unlikely it could be migrated fully to Fly without a similar managed DB. Would there be a viable option for connecting an AWS VPC to a Fly application internal network? Would that be better than just connecting over the internet from the edge locations?

Another question: how easy would it be to automate certificate setup?


Automating certs is pretty easy, we have a GraphQL API for managing certificates: SSL for Custom Domains · Fly

For RDS, the best option is to run a Wireguard peer inside a VPC. We have a basic terraform setup for this: GitHub - fly-apps/rds-connector: Trivial Terraform example for a WireGuard peer to RDS

Wireguard to VPC peering is important, so we’ll help as much as we can to get that going.