Hi,
after fixing our _acme records to be able to generate fly.io certs on one of our apps/domains that are routed through Cloudflare, the certificate is not being issued (has been almost an hour since the cert was added to the app, but hasn’t been issued yet). I’ve tried deleting and recreating the certificate to no avail.
Thanks,
Jakob
At first glance, the Let’s Encrypt certificate authorization were invalid due to the wrong value in the TXT record.
Manual testing shows the correct value is there now. I’m investigating.
There was a code change that introduced a subtle bug. This should now be fixed.
@jakob.murko would you mind retrying?
@jerome retried by deleting and recreating the certificate but still no issuance yet after 10 mins…
Let’s Encrypt does a check from multiple vantage points, I just looked at your _acme-challenge. record and it seems to be cached to an old value in some locations?
Did you previously set the _acme-challenge. record to a TXT with a long-ish TTL? It might still be cached in some Let’s Encrypt check locations. At least I’m seeing it cached in a few locations such as Amsterdam.
@jerome The TTL was set as Auto in Cloudflare’s dashboard, which I think correlates to 5 mins for TXT records…so it should’ve flushed by now.
Btw, is the final dot in the TXT record necessary/correct?
DNS world propagation is pretty complex and sometimes fragile. In this case, I consistently see a TXT record value starting with YJH5V3A9PUQ1YB4jEuI_ in Amsterdam in various tests I’m doing. I believe Let’s Encrypt tests for a valid DNS record in Amsterdam. Unfortunately their error messages are sometimes unclear, all we know is that the TXT record is wrong in one of their testing location.
The only solution right now might be to wait a little bit.
You can keep checking https://dnschecker.org and https://www.whatsmydns.net/ for _acme-challenge.<your-hostname> as a TXT record to see if it changes.
We have yet to integrate a solution that can pre-verify from multiple vantage points to give more insightful error messages.
Yes, it means the hostname is fully qualified.
@jakob.murko Just checked again and now some locations show a different TXT record value than the one I saw earlier? This is weird. The same locations that had issues now show a TXT record value starting with: 2akz85RxQbVNo1w-
Yeah, that’s weird. Btw, I see the same problem on our other hostname/instance (weird TXT records on the Amsterdam and Yekaterinburg servers on the site you linked). I’m not sure why that could be happening…if it doesn’t resolve I might have to also contact Cloudflare? Btw, I think the issuance might succeed if I temporarily bypass Cloudflare’s proxy for the hostname (“grey cloud” instead of “orange cloud”)?
That’s very weird. Kind of sounds like compromised DNS servers?
I’m not sure about contacting Cloudflare. It can’t hurt, but I’m not sure they’ll have a good response.
You should definitely bypass Cloudflare’s proxy for that particular CNAME.
I didn’t mean the acme CNAME/TXT records (I added both in a effort to resolve the situation), but the actual A and AAAA records for the actual domain. Also, when running dig I get good results for the TXT and CNAME entries. Can you provide more insight into what error letsencrypt is returning when trying to generate the cert?
The AAAA is the only one used for generating certificates. It uses the Let’s Encrypt ALPN verification challenge.
So you did the DNS verification flow to try and resolve the issue initially?
All we get is:
Incorrect TXT record "x" found at _acme-challenge.your.hostname
Looking at it again and it seems like you have the wrong thing setup. It used to be setup right but now the TXT record is completely wrong. It seems like you’ve set the TXT record to the value we need as a CNAME, not as a TXT.
You need a CNAME _acme-challenge.your.hostname. pointing to your.hostname.xyz.flydns.net., but it looks like you have a TXT with the value of your.hostname.xyz.flydns.net.. If you have both, you need to remove the TXT. I remember this worked earlier Today where a TXT query returned the correct value, recursively fetched from the provided CNAME.
Ah, yes I had both records set, CNAME and TXT. I have now removed the TXT…let’s see what happens?
Now I get the problem, though - the CNAME is pointing to flydns which actually returns the proper challenge response. Meanwhile, since I was setting the TXT record directly, instead of setting the secret I was setting the flydns hostname that was supposed to return the secret as the actual secret itself. And that makes more sense maybe why the results were mixed with the DNS queries (some were being resolved via CNAME and some via TXT directly)?
Looks like only the AMS and Yekaterinburg DNS servers are actually RFC compliant, since RFC1034 states: If a CNAME RR is present at a node, no other data should be present; this ensures that the data for a canonical name and its aliases cannot be different.
The above has fixed the issue, cert was issued shortly after removing TXT entries