Awaiting certificate for custom apex domain

Questions / Help
1

Hello Fly community :balloon: I’m trying to follow SSL for Custom Domains · Fly Docs to set up a Fly app with certificates on a custom apex domain, acmedns.dev (no wildcard cert required). I’ve tried requesting certificates a couple times over the past 18 hours but no joy yet, so I’m hoping someone can see where I’ve missed a step.

The registrar is Porkbun, the authoritative nameservers are self-hosted (e.g. not Cloudflare, so I don’t think the records are being proxied), and GLUE records are in place.

Here’s the current report from flyctl certs show acmedns.dev:

The certificate for acmedns.dev has not been issued yet.

Hostname                  = acmedns.dev

DNS Provider              = charlestonroadregistry

Certificate Authority     = Let's Encrypt

Issued                    = 

Added to App              = 11 hours ago

Source                    = fly

You are creating a certificate for acmedns.dev
We are using lets_encrypt for this certificate.

You can direct traffic to acmedns.dev by:

1: Adding an A record to your DNS service which reads

    A @ 168.220.89.5

You can validate your ownership of acmedns.dev by:

2: Adding an AAAA record to your DNS service which reads:

    AAAA @ 2a09:8280:1::622d

 OR 

2: Adding an CNAME record to your DNS service which reads:

    CNAME _acme-challenge.acmedns.dev => acmedns.dev.9r1wy.flydns.net.

I believe the records exist and resolve since I’m able to dig them from a couple different networks. Here’s what I’m seeing from one of them:

$ dig acmedns.dev A
...
acmedns.dev.            2710    IN      A       168.220.89.5
...
$ dig acmedns.dev AAAA
...
acmedns.dev.            2711    IN      AAAA    2a09:8280:1::622d
...

I initially had not added a CNAME record (seems like sometimes it can get in the way?), but after waiting several hours and not getting a certificate I deleted the custom domain, readded it, and included an _acme-challenge record just in case. It’s present now:

$ dig _acme-challenge.acmedns.dev CNAME
...
_acme-challenge.acmedns.dev. 1934 IN    CNAME   acmedns.dev.9r1wy.flydns.net.
...

The solution given in Awaiting Certificates - #4 by kurt is to make sure port 443 is being served which I believe is the case. I get an error when using acmedns.dev as the Host but the Fly app name works (the 404 is expected since the process is listening but there’s no content to serve):

$ curl https://acmedns.dev/
curl: (35) error:0A000126:SSL routines::unexpected eof while reading
$ curl https://acmedns-dev-1.fly.dev/
404 page not found

Any ideas will be appreciated!

2

Porkbun :smile: ! I’ve used them for a lot of personal/test stuff, so I can say off the bat that apex certs have worked for me in the past, if memory serves. Happy to look into this!

1 Like
3

Hi again- so I tried this out with Porkbun’s nameservers. The only RRs I added were A/AAAA with a TTL of 10m.

I was able to get a cert for an apex domain on a test app in minutes (treides.net with hellofly:latest). I used fly certs add treides.net app-name)

Checking on our end, I can’t find the CSR in our logs, but that could be an error on my part.That said, by way of ruling things out: would you be willing to try using the flyctl cli if you haven’t already?

I did let the records sit for a few hours, but it sounds like that’s what you did, too :sweat_smile:

Thank you, by the way, for your thorough initial report of the problem-- we’re happy to help get to the bottom of this!

4

Thank you, I’m definitely happy to try ideas! :smile:

Fly CLI

As suggested, a few hours ago I deleted the existing custom domain and then added it back using the Fly CLI. A quick summary of the commands I ran to make sure I’m not missing something obvious:

$ flyctl -a acmedns-dev-1 certs remove acmedns.dev                                  
? Remove certificate acmedns.dev from app acmedns-dev-1? Yes                                             
Certificate acmedns.dev deleted from app acmedns-dev-1
$ flyctl -a acmedns-dev-1 certs add acmedns.dev                                
You are creating a certificate for acmedns.dev
We are using lets_encrypt for this certificate.
...

Unfortunately, still no certificate yet:

$ flyctl -a acmedns-dev-1 certs show acmedns.dev
The certificate for acmedns.dev has not been issued yet.

Hostname                  = acmedns.dev

DNS Provider              = charlestonroadregistry

Certificate Authority     = Let's Encrypt

Issued                    = 

Added to App              = 6 hours ago

Source                    = fly
...

Nameservers

Thank you also for confirming that a domain with Porkbun nameservers is working as expected. Because the set of nameservers being used differs, perhaps there’s something incompatible with my setup. For treides.net as you said they’re the Porkbun nameservers:

$ dig @a.gtld-servers.net treides.net NS                                            
...
;; QUESTION SECTION:
;treides.net.                   IN      NS

;; AUTHORITY SECTION:
treides.net.            172800  IN      NS      curitiba.ns.porkbun.com.
treides.net.            172800  IN      NS      fortaleza.ns.porkbun.com.
treides.net.            172800  IN      NS      maceio.ns.porkbun.com.
treides.net.            172800  IN      NS      salvador.ns.porkbun.com.
...

On acmedns.dev I’ve set custom nameservers (technically it’s a single nameserver with two names to meet the requirements, sufficient for testing purposes) and since the nameservers are on the acmedns.dev domain I’ve also used the GLUE records feature to boostrap their IP addresses (they’re returned in the Additional Section):

$ dig @ns-tld1.charlestonroadregistry.com acmedns.dev NS
...
;; QUESTION SECTION:
;acmedns.dev.                   IN      NS

;; AUTHORITY SECTION:
acmedns.dev.            10800   IN      NS      ns1.acmedns.dev.
acmedns.dev.            10800   IN      NS      ns2.acmedns.dev.

;; ADDITIONAL SECTION:
ns1.acmedns.dev.        3600    IN      A       168.220.89.5
ns1.acmedns.dev.        3600    IN      AAAA    2a09:8280:1::622d
ns2.acmedns.dev.        3600    IN      A       168.220.89.5
ns2.acmedns.dev.        3600    IN      AAAA    2a09:8280:1::622d
...

I’m not sure how widespread support is for this setup. dig handles it, but maybe this is an esoteric corner of DNS? It seems possible that some sort of infinite lookup loop could happen in the certificate acquisition process if the Additional Section IPs aren’t being used, thus yielding a classic comedy sketch:

Q. Where is the nameserver for acmedns.dev?
A. ns1.acmedns.dev.
Q. Where is ns1.acmedns.dev?
A. Ask the nameserver for acmedns.dev.
Q. Where is the nameserver for acmedns.dev?
A. …

Along similar lines, does it matter that the nameserver is not only hosted on Fly but is actually the same app? (I’m reaching for anything I’ve done that seems out of the ordinary.)

Thank you again for the ideas. I’m hopeful that we’ll be able to figure out what’s up.

5

Hello again! I’ve had an opportunity to try some experiments with encouraging results. The solution I’ve reached isn’t ideal and I’m hopeful that there’s something Fly might be able to tweak to enable certificates for custom apex domains with self-hosted nameservers (though I readily admit that this is a niche use case :slight_smile: ). In the meantime I’ll share what I’ve tried, learned, and found to work in case it’s helpful to the community.

The quick summary:

  • :white_check_mark: Fly-provisioned certificates for an apex domain when using hosted nameservers (e.g. treides.net with Porkbun’s nameservers)
  • :x: Fly-provisioned certificates for an apex domain when using self-hosted nameservers with GLUE records (e.g. acmedns.dev through Porkbun, xcvb.dev through Google Domains)
  • :white_check_mark: Caddy-provisioned certificates for an apex domain when using self-hosted nameservers with GLUE records (e.g. xcvb.dev through Google Domains)

In more detail, I was concerned about how unusual GLUE records might be. Turns out they’re reasonably common though not all registrars support them. To try and rule out Porkbun-specific oddities I provisioned a new domain name (xcvb.dev) through Google Cloud Domains and configured GLUE records through Google Domains. The Fly app serving xcvb.dev is configured similarly to acmedns.dev (running a customized acme-dns instance) with self-hosted nameservers.

When I requested a certificate for the apex domain xcvb.dev through flyctl I had the same experience as on acmedns.dev: lots of timeouts, some errors, validation never succeeded, and the certificate was never issued.

So I modified my fly.toml to ask the Fly proxy to give me unmodified port 80 and port 443 TCP traffic (instead of terminating TLS and handling HTTP forwarding), installed caddy in front of acme-dns, and asked caddy to get a certificate for xcvb.dev. It worked! :tada:

There are of course some downsides to running like this long-term (caddy certificate storage is now a deal, the PROXY protocol needs to be used to get the actual remote client IP address, things like the fly-replay header won’t work, and so on), but because it shows that acquiring an ACME cert is possible I think we’re able to narrow down the troubles I was having to something about Fly’s certificate acquisition process.

This is all a long way of saying Fly is making difficult things possible :1st_place_medal: , and I appreciate being able to host an apex domain with self-hosted nameservers on Fly! If there’s any more troubleshooting I’m able to offer please don’t hesitate to ask.