Awaiting Certificates

Questions / Help
1

Hi Fly folks. :wave:

I’m trying to setup MinIO with a custom domain name and I’m hitting a snag. I tried setting up using the speed run instructions from the blog post and added A and AAAA records but that seemed to not issue the certificate. I realized that I probably wanted CNAME instead so switched the DNS records per the docs page but I’m still not seeing it issued.

$ fly certs show minio.test.litestream.io
The certificate for minio.test.litestream.io has not been issued yet.

Hostname                  = minio.test.litestream.io

DNS Provider              = name

Certificate Authority     = Let's Encrypt

Issued                    = 

Added to App              = 22 hours ago

Source                    = fly

Your certificate for minio.test.litestream.io is being issued. Status is Awaiting certificates.

I’m sure I’m doing something dumb but I’m not sure what. Do I need to remove the cert and start over? Also, I haven’t seen any instructions regarding an acme challenge mentioned in other posts so maybe that’s what I’m missing?

2

I believe A/AAAA or CNAME is fine with certs.

Do you use Cloudflare for the DNS? Only its orange-cloud/proxy DNS records can cause issues, I’ve found. The DNS record may need to be grey-cloud/non-proxy.

Failing that, it’s worth trying adding the acme challenge DNS entry too. The CLI does not reveal that as far as I know, so you have to get it from the Fly.io dashboard. If you sign in to that, click on the app, and look at certificates in there it should show the DNS entry to add. Which again, will need to be non-proxied (if using something like Cloudflare).

Failing that … not sure!

1 Like
3

We’re checking on this. A/AAAA should be fine, but so should CNAME!

4

@benbjohnson1 ok we figured it out. Your app doesn’t have a service on port 443, so we can’t issue the certificate (Lets Encrypt only connects to 443). If you add port 443 to your app config, you should be good to go.

5

Yep, it was the port 443. Thanks!

6

I’m also having this issue. How do you add the port? If it’s in toml file then it’s already there.

app = my-app

kill_signal = "SIGINT"
kill_timeout = 5
processes = []

[env]
  PORT = "8080"

[experimental]
  allowed_public_ports = []
  auto_rollback = true

[[services]]
  http_checks = []
  internal_port = 8080
  processes = ["app"]
  protocol = "tcp"
  script_checks = []

  [services.concurrency]
    hard_limit = 25
    soft_limit = 20
    type = "connections"

  [[services.ports]]
    handlers = ["http"]
    port = 80

  [[services.ports]]
    handlers = ["tls", "http"]
    port = 443

  [[services.tcp_checks]]
    grace_period = "1s"
    interval = "15s"
    restart_limit = 0
    timeout = "2s"
7

I have solved the issue with CNAME record temporary, but I would like to do via A & AAAA records.
For that, what to do? I mean on fly side and my application. I have followed the whole documentation.

8

You can see your app IPs with fly ips list. Set an AAAA record pointing to the IPv6 address, and set an A record pointing to the IPv4 address and you’ll be good!

9

yeah but I have already done that, it was awaiting for certification (Your certificate for … is being issued. Status is Awaiting certificates.) for a few hours. Then I did research in here, it says

Awaiting Certificates - #4 by kurt

So, say my config is right, no problem? It’s just a matter of time on DNS?

10

Certificates should be generated within a few minutes. If they aren’t, something is probably misconfigured.

11

Okay, so, is it safe to remove CNAME record and create A & AAAA records? Will it effect the current certificate?

12

I’m having similar issues and can’t make sense of what’s going on.

I’ve followed fly’s SSL for Custom Domains guide. It seems like the system can’t decide if certs were applied correctly. Checking the dashboard (Dashboard > Certificates > View) and they are listed as “verified”.

Also via command line:

$ flyctl certs show mnyou.org
The certificate for mnyou.org has been issued.

Hostname                  = mnyou.org

DNS Provider              = iwantmyname

Certificate Authority     = Let's Encrypt

Issued                    = rsa,ecdsa

Added to App              = 2 days ago

Source                    = fly

Note:

The certificate for mnyou.org has been issued.

But then:

$ flyctl certs check hostname
Error Could not find AppCertificate

IMO the documentation needs updating because I’m not doing anything extreme or custom and it’s not working as described.

13

Hey there, taking a look at a few different queries for your hostname I can see that there is an additional A record which points to an IP address outside of our network:

 curl ipinfo.io/104.198.14.52
{
  "ip": "104.198.14.52",
  "hostname": "52.14.198.104.bc.googleusercontent.com",

There’s another cert that’s been in place since before this was added to the app, which I can see when curling https://mnyou.org and connecting on 104.198.14.52:

* start date: Apr 21 21:00:29 2022 GMT
*  expire date: Jul 20 21:00:28 2022 GMT
*  subjectAltName: host "mnyou.org" matched cert's "mnyou.org"

But there’s also cert configured for your Fly app from 2 days ago, which you can see when you force a connection over 66.51.120.170:

 curl -vvv https://mnyou.org --resolve 'mnyou.org:443:66.51.120.170'
* Added mnyou.org:443:66.51.120.170 to DNS cache
* Hostname mnyou.org was found in DNS cache
*   Trying 66.51.120.170:443...
* TCP_NODELAY set
* Connected to mnyou.org (66.51.120.170) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN, server accepted to use h2
* Server certificate:
*  subject: CN=mnyou.org
*  start date: May 24 16:28:26 2022 GMT
*  expire date: Aug 22 16:28:25 2022 GMT
*  subjectAltName: host "mnyou.org" matched cert's "mnyou.org"
*  issuer: C=US; O=Let's Encrypt; CN=R3
*  SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x55f4b0ee92f0)
> GET / HTTP/2
> Host: mnyou.org

When you run that last command, are you running flyctl certs check hostname or flyctl certs check mnyou.org ?

If you got that response while supplying your domain instead of hostname, then it’s possible that the extra A record is tripping fly certs check up. Thanks for the question!

14

Just bumping this because I have the same issue. Wildcard certificate is still being issued since yesterday whereas the other one got issued really quick.

╰─$ fly certs list --config  fly.production.toml
Update available 0.0.337 -> v0.0.355.
Run "fly version update" to upgrade.
Host Name                 Added                Status
dert.gg                   20 hours ago         Ready
*.dert.gg                 21 hours ago         Awaiting certificates

╰─$ fly certs check "*.dert.gg" --config fly.production.toml
Update available 0.0.337 -> v0.0.355.
Run "fly version update" to upgrade.
The certificate for *.dert.gg has not been issued yet.
You are creating a wildcard certificate for *.dert.gg
We are using lets_encrypt for this certificate.
16

Ah, I think I might see the issue. Since you’re moving from a SAN to a wildcard cert, you’d need to create a CNAME record for the DNS-01 challege so that we can validate it.

But It doesn’t look like that’s set up yet when I run that domain against dnschecker.

17

Sorry, I didn’t really get what you’re saying. How’s it that I’m moving from SAN to a wildcard?

you’d need to create a CNAME record for the DNS-01 challege so that we can validate it.

How exactly do I do that? I’ve already followed everything mentioned here and set A, AAAA and CNAME (_acme-challenge.dert.gg) records accordingly for both IPv4 and IPv6.

Screenshot 2022-07-21 at 19.33.16
Screenshot 2022-07-21 at 19.33.161532×370 21.3 KB

18

oh, my mistake with the link! but, I’m not seeing any answers for _acme-challenge.dert.gg either:

Is it possible you’re using a DNS provider which is doing something unusual with their CNAME queries? What happens when you dig that subdoman against a large public resolver?

19

I’m using Namecheap and using their private email, might that be causing the problem here? I can try turning that off and see again.

I’m wondering how I’ve got the other certificate though for dert.gg. Doesn’t it require the same thing? At least the same CNAME is mentioned when I view the certificate status on the UI.

Screenshot 2022-07-21 at 20.41.21
Screenshot 2022-07-21 at 20.41.211297×325 34.7 KB

Edit: Turned off the private email service just now.

20

We use a different ACME challenge (tls-alpn-01) for requesting a cert if the CN only needs to match the apex domain, as it’s way more convenient for this use-case.

But it’s unlikely that tls-alpn-01 will ever support wildcards—if you’d like to know why, this thread goes into a little more detail:

1 Like
21

I think I’ve figured the problem. In the host name of CNAME record I was typing in the whole thing (i.e. _acme-challenge.dert.gg) but I was told I am not supposed to do that.

Well, I don’t know much about it, and I believe it was the CLI tool that tricked me into doing that if I recall the output correctly, because if you go to the UI, that shows a different thing which is correct I think but I was following the CLI tool.

My certificate just got issued by the way, thank you @eli !

2 Likes