App is not accessible through the fly.dev domain: ERR_CERT_REVOKED

Hi all!

My app suddenly started to return SSL error when accessing through the fly.dev domain:

https://skaro.fly.dev/

Logs are fine, app looks healthy and it is also accessible via IP

http://213.188.219.207/

Are there currently any issues on fly.dev domain? Thank you!

fly.dev itself seems to be fine.
Example: Visit https://debug.fly.dev/

Must be something on Fly’s end…

This likely has to do with Let’s Encrypt’s certificate revoking a whole lot of certificates that used the ALPN proof. You’ll need to try and get a new certificate somehow :grimacing:

1 Like

Ah, that’s right.
I see your certs in the affected certs list (referred to in the above link).

Thanks! This explains the reason why this is happening.

Funny thing is that it is possible to manage certificates for custom domains from flyctl, but not for the app.fly.dev domain. Hope that it will be resolved next week on fly.io side.

We have reissued all certificates we had generated with Let’s Encrypt. All old certificates have been revoked.

Normally .fly.dev use our wildcard and do not require their own certificate.

Is this a fairly old app?

@jerome no, this is relatively new - only 2 months old. Created it normally with flyctl launch without any manual configs

@Andrey_Marchenko Sorry for the late response, I believe we should now be serving a valid certificate.

We’ll work on fixing the issue upstream so this doesn’t happen again.

1 Like

@jerome thanks a lot! All good now, thanks for help

What do we need to do to get new certificates for our custom domain? Still seeing the cert revoked error (app.getelements.com)

It looks like there might be an issue because that subdomain SOA doesn’t match the apex SOA. I think we have a fix going out, and if so this should resolve itself in the next 20 mins.

The way Lets Encrypt validates these has a bunch of edge cases. I’m actually not sure how you got a cert the first time!

I deleted and recreated the certs about an hour ago, still same thing. Revoked cert. This is kinda a critical-level thing for us.

@Ben_Kraus this should now be fixed. Can you confirm?

It is not. https://app.getelements.com

Sorry, I just noticed this myself. Apparently curl doesn’t complain. I’m looking at this now.

Ok it should now be good. We had a revoked ECDSA certificate and good RSA certificate. We always prioritize ECDSA if the client support it and so that got picked up first.

About 10 minutes ago I deleted my old certs and created new ones, but I’m still having this error. I assume I just need to wait for something to propagate?

https://nezteb.net/ gives me the same ERR_CERT_REVOKED error.

I’m not sure why the ECDSA certs are red.

Root domain cert:

Wildcard subdomain cert:

I’m also not sure why the root domain verification is green but the wildcard one is not.

EDIT: I just saw ECDSA Cert Not Assigning - #2 by kurt, so I’m hoping that despite having the RSA cert and still getting the revoked error, this will eventually resolve itself?

DOUBLE EDIT: Or maybe this? Site unresponsive, ERR_SSL_PROTOCOL_ERROR - #7 by kurt

Yesterday evening I completed a procedure to purge our cache of revoked certificates. The problem should now be solved for all certificates.

@Nezteb this should now be working

1 Like

hi @jerome can you take a look at our instance https://coscene-opencat-server.fly.dev/ we are seeing similar errors, tried a lot of things but no good