Migrating custom TLS from Cloudflare/AWS to Fly

1

An app currently uses CloudFlare spectrum (TCP passthrough) for DDoS mitigation. This forwards over the internet to AWS (Toronto), where nginx handles TLS termination with a lua openresty extension (for some 60k custom domains). Next, on to a small set of non-clustered varnish instances handling cache, and finally, a Rails app.

I’d like to look into how Fly could improve this setup. A few questions:

  1. Should moving TLS termination to the edge should help speed up connection time?
  2. Would it theoretically be faster to pass the unencrypted traffic to the core app through your network instead of using the current path over the internet of Cloudflare->openresty?
  3. I read your article on the 5-hour CDN, which seems like it would apply here for also moving cache to the edge. I’m interested in this because of the exorbitant quotes from CDN providers for hosting 60k custom domains. What would be a good way to estimate Fly costs given N specific regions?
  4. Related to 3: This app could be simplified by removing the TLS handling, but as mentioned, that comes at a high cost compared to the zero cost of LE certs used today. Would a volume discount be applicable for an application with this number of custom certs?
  5. Do you offer the same or similar kinds of DDoS mitigation as companies like Cloudflare or Fastly?
2 Likes
2

Hey this sounds fun!

  1. yes, moving tls to the edge will help quite a bit. TLS handshakes are one of the leading causes of slow page loads.
  2. Maybe theoretically faster but it’s hard to tell! We should do a better job of routing to an app than the internet at large.
  3. Fly costs are pretty simple, it’s $0.10 per cert we manage plus VM/disk usage. Someone with a similar workload as yours pays about $200/mo for VMs + disks in all our regions.
  4. Yes, we can work out a volume discount. Would you mind emailing me (kurt fly.io)?
  5. We have similar network level ddos protections in place. CloudFlare has a bunch of HTTP level mitigations (which you may not be using) that we don’t provide, but OpenResty is a nice place to solve some of that.
2 Likes
3

Thanks for the info.

This app uses AWS RDS and it’s unlikely it could be migrated fully to Fly without a similar managed DB. Would there be a viable option for connecting an AWS VPC to a Fly application internal network? Would that be better than just connecting over the internet from the edge locations?

Another question: how easy would it be to automate certificate setup?

2 Likes
4

Automating certs is pretty easy, we have a GraphQL API for managing certificates: SSL for Custom Domains · Fly

For RDS, the best option is to run a Wireguard peer inside a VPC. We have a basic terraform setup for this: GitHub - fly-apps/rds-connector: Trivial Terraform example for a WireGuard peer to RDS

Wireguard to VPC peering is important, so we’ll help as much as we can to get that going.

2 Likes
5

I got this going pretty easily. I’ll only note that the wireguard-tools on Ubuntu 20 requires resolvconf but doesn’t list it as a dependency. So you need to install resolvconf as well.

After setting up a peer in YYZ with a VPC in the same region, I’m seeing a ping latency of 30ms. I’m guessing this means you are not hosted in the same DC? This is a bit too high for us to run applications against RDS from inside Fly, unfortunately. But it should be acceptable for the cached domains.

6

Wow that’s really high. I’m mildly surprised it’s >5ms. Can you do a traceroute between yyz on Fly and AWS Toronto?

7

Not much info from either side, even with rltraceroute.

# rltraceroute6 bz-fly.internal -p 8080
traceroute to bz-fly.internal (fdaa:0:2d04:a7b:aa2:d15e:182f:2) from fdaa:0:2d04:a7b:ce2:0:a:2, port 8080, from port 49865, 30 hops max, 60 bytes packets
 1  * * *
 2  * * *
 3  fdaa:0:2d04:a7b:aa2:d15e:182f:2 (fdaa:0:2d04:a7b:aa2:d15e:182f:2)  36.654 ms  36.473 ms  36.385 ms

/ # traceroute fdaa:0:2d04:a7b:ce2:0:a:2
traceroute to fdaa:0:2d04:a7b:ce2:0:a:2 (fdaa:0:2d04:a7b:ce2:0:a:2), 30 hops max, 72 byte packets
 1  *  *  *
 2  *  *  *
 3  fdaa:0:2d04:a7b:ce2:0:a:2 (fdaa:0:2d04:a7b:ce2:0:a:2)  40.336 ms  36.402 ms  36.432 ms
8

Oh, can you try a public IP? Wireguard will hide all the hops.

9 
# tcptraceroute a36ce8f11452a4d61b9d03e301409f65-6f12b5a6cc9eac30.elb.ca-central-1.amazonaws.com
Selected device eth0, address 172.31.6.38, port 42647 for outgoing packets
Tracing the path to a36ce8f11452a4d61b9d03e301409f65-6f12b5a6cc9eac30.elb.ca-central-1.amazonaws.com (15.223.137.205) on TCP port 80 (http), 30 hops max
 1  * * *
 2  ec2-15-223-137-205.ca-central-1.compute.amazonaws.com (15.223.137.205)  0.283 ms  0.245 ms  0.250 ms
 3  ec2-15-223-137-205.ca-central-1.compute.amazonaws.com (15.223.137.205)  0.622 ms  0.523 ms  0.345 ms
 4  ec2-15-223-137-205.ca-central-1.compute.amazonaws.com (15.223.137.205)  0.443 ms  0.400 ms  0.437 ms
 5  ec2-15-223-137-205.ca-central-1.compute.amazonaws.com (15.223.137.205) [open]  0.430 ms  0.618 ms  0.476 ms

This is a lot faster, interestingly.

10

Oh well the obvious problem here is that we don’t have a wireguard gateway in YYZ. You’re likely connecting through another region.

That’s fixable! We’ll bring one up today or tomorrow.

11

OK, interesting. It shows up in yyz so I assumed it was hosted there :smiley:

12

If you run fly platform regions you can see where gateways currently exist. You’re probably getting one in Chicago.

13

You can create a wireguard peer in yyz now. Give it a try and see how it works?

14

OK, I set it up but am not able to ping my app from the AWS instance. I did wg-quick down on the old one and wg-quick up on the new. DNS resolution fails and I can’t reach the DNS server either by IP.

15

@jsierles we gave you the wrong IP with your wireguard config. We cleared those out, will you make a new one and see if it works for you?

16

Works! Down to 8ms.