At some point semi-recently SSL using custom domains stopped working for one of my apps. It’s a low-traffic app which I only access every couple of weeks, so it took a while for me to notice it was down.
The *.fly.dev URL (https://prod-app-epp.fly.dev/) works as expected, and the app displays.
The custom domain (https://emailpingpong.com/) however, does NOT, because there’s some issue with SSL/TLS. All the correct DNS records are in place.
To clarify - this is something that used to work, and broke at some point without me doing any changes. Based on my understanding the whole SSL-termination is done by fly.io, so as long as my app is up and running I’m expecting SSL for custom domains to “just work” (as long as DNS records and certs are in order).
Here’s some diagnostics info:
# Firefox error message:
An error occurred during a connection to emailpingpong.com. PR_END_OF_FILE_ERROR
# Diagnosing it a bit more with openssl directly:
$ openssl s_client -connect emailpingpong.com:443
CONNECTED(00000003)
4047DD54CB7F0000:error:0A000126:SSL routines:ssl3_read_n:unexpected eof while reading:ssl/record/rec_layer_s3.c:321:
# Output of `fly certs check':
$ fly certs check '*.emailpingpong.com'
The certificate for *.emailpingpong.com has been issued.
Hostname = *.emailpingpong.com
DNS Provider = aws
Certificate Authority = Let's Encrypt
Issued = rsa,ecdsa
Added to App = 3 months ago
Source = fly
Please let me know if there’s any other piece of diagnostics info that might be useful. I’m afraid I’m a bit stuck not knowing what to check next.
You’ll need an extra certificate for the root hostname emailpingpong.com. This is a common issue we should fix by adding the root hostname as an additional servername on the certificate.
That’s spot on @jerome - thanks a bunch!
As soon as I did fly certs add emailpingpong.com things started working.
It would be great to do just like you suggested - to add the apex domain as an additional server name on the certificate. Or perhaps even modify the UI / CLI a bit to make it a bit more obvious that things might not be properly set up. I guess I got a bit tricked by all the green checkmarks, and didn’t think properly about what might be wrong
To have SSL properly set up for the apex domain (e.g. example.com) and any of it’s subdomains (e.g. app.example.com) the output of your fly certs list should look something like this:
$ fly certs list
Host Name Added Status
example.com 5 minutes ago Ready
*.example.com 3 months ago Ready
If you go to “Certificates” in the Fly control panel you can click “Check again”. Does that give any insights?
I believe you need to have all the DNS records in place for things to work properly. Do you still have the A record and the CNAME (ACME Challenge) record set up? I’m asking because previously I’ve done the mistake of “cleaning up” (deleting) the ACME Challenge record, because I thought it wasn’t needed after initial setup. I’m thinking it’s probably needed when renewing the certificate as well.
EDIT: I just saw that you removed and added the certificate again and that fixed things. Since it stopped working after about 3 months it very much seems like the auto-renew of the certificate didn’t work for whatever reason…just like you said.
Didn’t have time to look at the possible error output for that check.
It was added 3 months ago, and usually we renew every 2 months.
@mkozak it’s possible the hostname didn’t pass our own verification checks and we kept rescheduling the verification further away in time. I can’t tell anymore because we don’t store each verification attempt’s output (that would be a lot of data).
I’ve found the issue - I must have accidentally deleted the verification cname record in the past 3 months. That’s why it was working fine during this time, but when it tried to renew couldn’t verify my ownership of the domain (A and AAA records were intact)
