Best way to understand SSL errors on nginx proxy

Questions / Help
1

We’ve been tasked with updating our reverse proxy to accommodate a custom domain including SSL certs managed by a third party.

We’ve been happily using fly as our nginx reverse proxy but now I’ve encountered an issue and I can’t tell if it is nginx shaped, or fly shaped!

We’ve set up our Dockerfile to securely download the certificate and key files from an S3 bucket and place them into our server.

We’ve then created a server block in our nginx config that looks for that server and if so applies the SSL cert.

Commented out properties were originally in, but trying to have the most minimal setup for testing.

  listen 443 ssl;
  server_name test.domain.net;

  ssl_certificate /etc/nginx/ssl/domain.cert;
  ssl_certificate_key /etc/nginx/ssl/domain.key;

  # ssl_protocols TLSv1.3;
  # ssl_prefer_server_ciphers on;
  # ssl_ciphers EECDH+AESGCM:EDH+AESGCM;
  # ssl_ecdh_curve secp384r1;
  # ssl_session_timeout  10m;
  # ssl_session_cache shared:SSL:10m;
  # ssl_session_tickets off;
  # ssl_stapling on;
  # ssl_stapling_verify on;
  # resolver 8.8.8.8 8.8.4.4 valid=300s;
  # resolver_timeout 5s;

  # add_header X-Frame-Options DENY;
  # add_header X-Content-Type-Options nosniff;
  # add_header X-XSS-Protection "1; mode=block";

  error_log /var/log/nginx/domain.log debug;
  # ...

We get a generic SSL error back when visiting the site, and no specific errors in any logs, including the specific debug log we created.

Because we’re clearly getting an SSL error back (makes sense, we’re playing with SSL), and the fact that the domain.log file is created (just not populated) I’m fairly confident this server block has been correctly seen by nginx. However the lack of any output makes me wonder if fly could be getting in the way?

Any pointers would be gratefully received.

2

Do you have the tls handler set in your fly.toml? If you want your app to handle TLS you will need to do TCP passthrough (removing the handlers from the service in fly.toml).

3

Ah interesting, yes we do.

  [[services.ports]]
    handlers = ["tls", "http"]
    port = "443"

I presume that would mean we’d need to manually handle all domains coming in, not just this one case?

4

By removing the tls handler we’ve been able to see traffic actually hitting the newly created fly app. However there still seems to be issues that seem to jump between nginx and fly.

If we want to control our own SSL, do we even need the http handler for going into nginx?

This is my current setup, which fails the SSL handshake;

With the fly.toml I’m unsure if we need the http handlers, although when I try and remove them (and update the below nginx.conf to look for 443 and not 8080) I get healthcheck errors.

app = "my-app"

kill_signal = "SIGINT"
kill_timeout = 5
primary_region = "lhr"
processes = []

[env]

[experimental]
  auto_rollback = true

[[services]]
  internal_port = 8080
  protocol = "tcp"

  [services.concurrency]
    hard_limit = 200
    soft_limit = 180
    type = "connections"

  [[services.ports]]
    force_https = true
    handlers = ["http"]
    port = 80

  [[services.ports]]
    handlers = ["http"]
    port = 443

  [[services.tcp_checks]]
    grace_period = "1s"
    interval = "15s"
    restart_limit = 0
    timeout = "2s"

With this nginx.conf file I’ve tried to follow a few different guides on this, this seems to be the setup that should work for the basic use-case.

server {
  listen              8080 ssl;
  server_name test.domain.net;

  ssl_certificate          /etc/ssl/certs/my-cert.crt;
  ssl_certificate_key /etc/ssl/private/my-key.key;

  error_log /var/log/nginx/test-domain.log debug;

  client_max_body_size 40M;
  port_in_redirect     off;

  if ($http_x_forwarded_proto = "http") {
    return 301 https://$http_host$request_uri;
  }

  proxy_set_header X-Forwarded-Host $host;
  proxy_set_header X-Origin-Host $host;
  proxy_set_header X-Forwarded-For $remote_addr;
  proxy_set_header X-Real-IP $remote_addr;

  location /endpoint {
    proxy_pass https://proxied-app.app;
  }

Finally, the Dockerfile which really just copies the certs and the nginx.conf file

FROM nginx

RUN apt-get update \
  && apt-get install -y curl unzip

RUN curl https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip -o awscliv2.zip \
  && unzip awscliv2.zip \
  && ./aws/install \
  && rm -rf aws awscliv2.zip

ARG AWS_ACCESS_KEY_ID
ARG AWS_SECRET_ACCESS_KEY

RUN mkdir /etc/nginx/ssl

RUN aws s3 cp s3://domain-proxy-files/keys/my-cert.crt /etc/ssl/certs/my-cert.crt
RUN aws s3 cp s3://domain-proxy-files/keys/my-key.key /etc/ssl/private/my-key.key

RUN chmod 644 /etc/ssl/certs/my-cert.crt
RUN chmod 600 /etc/ssl/private/my-key.key

COPY nginx.conf /etc/nginx/conf.d/nginx.conf

Any help would be greatly appreciated, I’m getting to the stage where I may abandon Fly for this particular project and use something like CloudFlare.

5

I’ve done some more digging and it seems like fly is still inserting itself between things.

From the deployed server;

openssl s_client -connect localhost:443

Returns a much different set of errors than running it from a local machine;

openssl s_client -connect test.domain.net:443

On the server I see;

CONNECTED(00000003)
Can't use SSL_get_servername
depth=0 O = "CloudFlare, Inc.", OU = CloudFlare Origin CA, CN = CloudFlare Origin Certificate
verify error:num=20:unable to get local issuer certificate
verify return:1
depth=0 O = "CloudFlare, Inc.", OU = CloudFlare Origin CA, CN = CloudFlare Origin Certificate
verify error:num=21:unable to verify the first certificate
verify return:1
depth=0 O = "CloudFlare, Inc.", OU = CloudFlare Origin CA, CN = CloudFlare Origin Certificate
verify return:1
---
Certificate chain
 0 s:O = "CloudFlare, Inc.", OU = CloudFlare Origin CA, CN = CloudFlare Origin Certificate
   i:C = US, O = "CloudFlare, Inc.", OU = CloudFlare Origin SSL Certificate Authority, L = San Francisco, ST = California
---

Which I think makes sense, because I’m asking it to verify localhost when this is a cert for test.domain.net (I’m editing the actual domain being used of course)

When I run openssl locally and point it at fly, either pointing directly at the blah.fly.dev or pointing it at my correct subdomain (test.domain.net) I get;

CONNECTED(00000005)
8328020288:error:1404B42E:SSL routines:ST_CONNECT:tlsv1 alert protocol version:/AppleInternal/Library/BuildRoots/9e200cfa-7d96-11ed-886f-a23c4f261b56/Library/Caches/com.apple.xbs/Sources/libressl/libressl-3.3/ssl/tls13_lib.c:151:
---
no peer certificate available
---
No client certificate CA names sent
---
SSL handshake has read 5 bytes and written 294 bytes

I think this suggests that fly is doing something before the request even makes it to nginx.

6

You’ll actually need to remove all handlers to get raw tcp. Right now, we’re trying to handle connections as if they’re HTTP over plaintext, which isn’t true!

Setting:

handlers = []

Should get you what you need.

7

Thanks, that makes sense, and has got me a little further, but now (updated configs below) it is treating all requests made over https as being http.

curl https://test.domain.net/ 

 <html>
 <head><title>400 The plain HTTP request was sent to HTTPS port</title></head>
 <body>
 <center><h1>400 Bad Request</h1></center>
<center>The plain HTTP request was sent to HTTPS port</center>
<hr><center>nginx/1.23.4</center>
</body>
</html>

I don’t really understand why that would be the case, I tried removing the [[services.ports]] for 80 to see, but no network traffic would work, I also tried removing internal_port since that is maybe clashing with traffic coming in, but that also breaks things.

This is myfly.toml now;

app = "my-app"

kill_signal = "SIGINT"
kill_timeout = 5
primary_region = "lhr"
processes = []

[env]

[experimental]
  auto_rollback = true

[[services]]
  internal_port = 443
  protocol = "tcp"

  [services.concurrency]
    hard_limit = 200
    soft_limit = 180
    type = "connections"

  [[services.ports]]
    handlers = []
    port = 80

  [[services.ports]]
    handlers = []
    port = 443

  [[services.tcp_checks]]
    grace_period = "1s"
    interval = "5s"
    restart_limit = 0
    timeout = "2s"

And the relevant bit of my nginx.conf, of note, I don’t have anything to handle 80 traffic internally, I didn’t think I’d need to?

server {
  listen      443 ssl;
  server_name test.domain.net;

  ssl_certificate     /etc/ssl/certs/cert.crt;
  ssl_certificate_key /etc/ssl/private/key.key;
}
8

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.