Deleted SSL cert still being served

[moz]

Hello,

I created a cert for notmyrealdomain.com on the 14th and all was working fine from the moment I set it up. A few days later I realized I needed a wildcard instead so today (19th), I deleted the cert for notmyrealdomain.com and created a wildcard cert "*.notmyrealdomain.com". I probably futzed around a few times creating / deleting certs trying to get things to work, but right now I only have one wildcard cert.

The new wildcard cert shows as verified with the ECDSA cert issued:

If I run curl -v -D - -o /dev/null -sS https://notmyrealdomain.com the output is:

It still shows as serving the apex cert created on the 14th. If it were the new wildcard one, I’d expect the start date to be the 19th and the common name (CN) to be *.notmyrealdomain.com. Same thing if I look up on: SSL Checker

Perhaps I’ve got this all wrong but this leads me to believe that my new cert is not being served somehow but the deleted old one is? I need a CNAME for auth.notmyrealdomain.com to point to cname.workosdns.com and I don’t think this will work until I get the wildcard cert in place.

If anyone has run into anything similar and has suggestions, I’d love some advice. Not sure what to do now. Perhaps just wait? Sorry, I’m probably out of my depth with all this.

[lubien]

Hi there!

Can you verify if your domain DNS had the old records removed? Also worth mentioning it sometimes takes a few days for the records to propagate so in this case waiting might be the way to go.

[moz]

Thank you Lubien.

Yep, old DNS records removed.

Any insight as to why it looks like an old cert that’s been deleted from the fly.io dashboard still seems like it’s being served? I read a few community posts where people reported something similar(ish): ERR_SSL_PROTOCOL_ERROR using wildcard certificate - #7 by jascha

Does the old cert need to be manually purged somehow?

[lubien]

Can you send me the name of the app so we can take a look?

[moz]

Sure thing, thank you. zigmo.fly.dev

[lubien]

I just verified here and appears to be your records still point to Fly.io at least to me

If the records are deleted this seems mostly like a DNS propagation issue so all we can do is wait

[moz]

Hi @lubien

Still not having any luck here. I deleted all certs from Fly and all DNS records. I then waited until no records were propagating on https://www.whatsmydns.net. (red x’s all the way down)

I then created the wildcard cert again:

Then I setup the required records again and waited until DNS propagation showed all green check marks https://www.whatsmydns.net

These are the DNS records:

No dice yet. New wildcard cert isn’t being issued and running curl -v -D - -o /dev/null -sS https://notmyrealdomain.com still shows an apex cert that was deleted two days ago as being served. Not sure what else to do here. I can’t help but think this old and revoked / deleted cert that is being served is the root of the problem. Can this be purged from the system somewhere?

Thanks again for any help

[lubien]

I’m interested in this bit

I assume the obfuscated part is *.yourdomain.com so this properly works but I can slightly see an “a” at the start of the AAAA record which would mean this is not the case.

If these two are set as “yourdomain.com” this explains why yourdomain.com still points to Fly and why the wildcard doesn’t yet:

You need to set an A record to *.yourdomain.com using the IP 188.93.151.46 and please note some DNS providers sometimes autocomplete “.yourdomain.com” so if you just put “*” they might already fill in the rest.

I can verify your CNAME is set properly:

➜ dig CNAME _acme-challenge.yourdomain.com +short
http://yourdomain.com.myejy.flydns.net.

Unlike your A/AAAA records:

➜ dig A foo.yourdomain.com

; <<>> DiG 9.10.6 <<>> A foo.yourdomain.com
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NXDOMAIN, id: 21262
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;foo.yourdomain.com.		IN	A

[moz]

Thank you @lubien

You were right about the records. I updated and still no luck.

DNS Records:

Certificate is still stuck waiting in fly:

AAAA Record

A Record

CNAME

[kurt]

This seems to have been rate limited by Lets Encrypt. The specific error is:

Error creating new order :: too many certificates (5) already issued for this exact set of domains in the last 168 hours: [domain]: see Duplicate Certificate Limit - Let's Encrypt

You can search here for the domain to se when exactly it was issued: https://crt.sh/

According to the Lets Encrypt forums, the only “fix” is to wait 7 days from the last issuance.

Were these all attempts to create wildcard certs on Fly.io or did you have this setup somewhere else as well?

[moz]

Thank you @kurt

Yes, these attempts were either through the Fly CLI or dashboard. I did change name servers from hover to dnsimple in between though. Sorry for futzing around so much with all the cert creation…although I am fairly certain I didn’t create as many as shown when you search https://crt.sh/. Not sure if that list double counts somehow. But yep, I did go over the limit of 5 I could just never get the first wildcard to work and tried out a bunch of stuff to no avail.

Happy to wait this out. Appreciate your reply