Clarification on static IP4 routing

I’m planning to make a blogging system and let customers use their own domains, so I’ll be managing SSL internally (probably with certmagic). I’d like this to be portable, so I don’t want to use the Fly managed custom domains.

If I set up static IPv4 routing, is that IP reserved exclusively for my app? I can point sites.myblogengine.com to the IP address, and ask all my users to just make CNAMEs to sites.myblogengine.com? When the request reaches Fly will the request be routed to my app purely based on the IP address?

That’s right! You “own” IPs as long as they’re attached to your app.

What certificate store are you going to use for certmagic?

DynamoDB, Global Tables. Will likely need to walk regions on keys not being found in order to handle the request arriving at a different region from the one it was sent and stored at. Certmagic also requires a lock, that might wind up being a dummy. Don’t want to implement a global lock.

@sudhir.j Why is that necessary? Certmagic has a built in cache for certs. When I was looking into it I didn’t see a reason to use a distributed store for it but am curious why you do.

I’m not @sudhir, but Fly apps don’t have persistent disks and you might be running multiple instances at a time. If Certmagic is configured with just local disk, it’ll vanish between deploys and only one instance will have a certificate at any given time.

Yeah, the default cert is file system, and let’s say there are 10 instances in New York, 5 in London and 3 in Singapore. If only the file system cache is used, there would need to be 18 certs issued, which is a problem. They would also need to be constantly re-issued every time the instance restarted, because the hard disk is ephemeral. LE would likely throttle you, especially if there are a lot of certs and instances.

But even that would work only in the DNS challenge, in the HTTP and TLS challenge, the file system cache is a non-starter. There’s no guarantee that the verification check will reach the same instance or the region that asked for it and currently has the token. To have all instances in a region share the cache, we’d want to put them in a DynamoDB table (or a database or S3) in that region. Since the same cert is applicable no matter what the region is, we’d want it replicated to all regional copies of the database, which Global Tables will do.

The walking of tables when a key not found is to handle the edge case where a cert is requested from London and the verification call goes to New York. The challenge token would have been stored in London, and will sync up to New York eventually, but that may happen after the verification call. So if New York can’t find a key it’s trying to verify it needs to check London and Singapore as well — walk all active database regions (can do this in parallel) — until if finds the key somewhere or all return empty.

@kurt also just wanted check — if I do a double CNAME redirect I can still maintain full portability while using the Fly certs, right? For instance, if I ask my customers to redirect their domains to sites.myplatform.com which I then CNAME to myapp.fly.dev, I can use the Fly certs manager without locking my customers in to Fly, right? And I don’t even need to buy a static IPv4 for this?

I’m okay with the cert manager fees, I’m more concerned with not exposing Fly to my customers and retaining the option to move somewhere else later if I have to. Don’t want to get fully into managing my own certs unless I have to.

Yes that’s how most people do it. For our own management, the important bit is having an AAAA record that points to your IPv6 address. A cname to <app>.fly.dev accomplishes that.

1 Like