# Best way to understand SSL errors on nginx proxy

**URL:** <https://community.fly.io/t/best-way-to-understand-ssl-errors-on-nginx-proxy/11817>\
**Category:** Questions / Help\
**Tags:** login-only\
**Created:** [March 29, 2023, 3:54pm UTC](https://community.fly.io/t/best-way-to-understand-ssl-errors-on-nginx-proxy/11817 "2023-03-29T15:54:34Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![tosbourn](https://sea1.discourse-cdn.com/fly1/user_avatar/community.fly.io/tosbourn/32/37505_2.png) [@tosbourn](https://community.fly.io/u/tosbourn)\
**Post date:** [March 29, 2023, 3:54pm UTC](https://community.fly.io/t/best-way-to-understand-ssl-errors-on-nginx-proxy/11817/1 "2023-03-29T15:54:34Z")

</div>

We’ve been tasked with updating our reverse proxy to accommodate a custom domain including SSL certs managed by a third party.

We’ve been happily using fly as our nginx reverse proxy but now I’ve encountered an issue and I can’t tell if it is nginx shaped, or fly shaped!

We’ve set up our `Dockerfile` to securely download the certificate and key files from an S3 bucket and place them into our server.

We’ve then created a server block in our nginx config that looks for that server and if so applies the SSL cert.

Commented out properties were originally in, but trying to have the most minimal setup for testing.

```nginx
  listen 443 ssl;
  server_name test.domain.net;

  ssl_certificate /etc/nginx/ssl/domain.cert;
  ssl_certificate_key /etc/nginx/ssl/domain.key;

  # ssl_protocols TLSv1.3;
  # ssl_prefer_server_ciphers on;
  # ssl_ciphers EECDH+AESGCM:EDH+AESGCM;
  # ssl_ecdh_curve secp384r1;
  # ssl_session_timeout 10m;
  # ssl_session_cache shared:SSL:10m;
  # ssl_session_tickets off;
  # ssl_stapling on;
  # ssl_stapling_verify on;
  # resolver 8.8.8.8 8.8.4.4 valid=300s;
  # resolver_timeout 5s;

  # add_header X-Frame-Options DENY;
  # add_header X-Content-Type-Options nosniff;
  # add_header X-XSS-Protection "1; mode=block";

  error_log /var/log/nginx/domain.log debug;
  # ...

```

We get a generic SSL error back when visiting the site, and no specific errors in any logs, including the specific debug log we created.

Because we’re clearly getting an SSL error back (makes sense, we’re playing with SSL), and the fact that the `domain.log` file is created (just not populated) I’m fairly confident this server block has been correctly seen by nginx. However the lack of any output makes me wonder if fly could be getting in the way?

Any pointers would be gratefully received.

---

<div class="post-metadata">

**Author:** ![lillian](https://sea1.discourse-cdn.com/fly1/user_avatar/community.fly.io/lillian/32/49616_2.png) [@lillian](https://community.fly.io/u/lillian)\
**Post date:** [March 29, 2023, 3:57pm UTC](https://community.fly.io/t/best-way-to-understand-ssl-errors-on-nginx-proxy/11817/2 "2023-03-29T15:57:31Z")

</div>

Do you have the `tls` handler set in your `fly.toml`? If you want your app to handle TLS you will need to do TCP passthrough (removing the handlers from the service in fly.toml).

---

<div class="post-metadata">

**Author:** ![tosbourn](https://sea1.discourse-cdn.com/fly1/user_avatar/community.fly.io/tosbourn/32/37505_2.png) [@tosbourn](https://community.fly.io/u/tosbourn)\
**Post date:** [March 29, 2023, 3:59pm UTC](https://community.fly.io/t/best-way-to-understand-ssl-errors-on-nginx-proxy/11817/3 "2023-03-29T15:59:49Z")

</div>

Ah interesting, yes we do.

```auto
  [[services.ports]]
    handlers = ["tls", "http"]
    port = "443"

```

I presume that would mean we’d need to manually handle all domains coming in, not just this one case?

---

<div class="post-metadata">

**Author:** ![tosbourn](https://sea1.discourse-cdn.com/fly1/user_avatar/community.fly.io/tosbourn/32/37505_2.png) [@tosbourn](https://community.fly.io/u/tosbourn)\
**Post date:** [April 5, 2023, 10:50am UTC](https://community.fly.io/t/best-way-to-understand-ssl-errors-on-nginx-proxy/11817/4 "2023-04-05T10:50:36Z")

</div>

By removing the `tls` handler we’ve been able to see traffic actually hitting the newly created fly app. However there still seems to be issues that seem to jump between nginx and fly.

If we want to control our own SSL, do we even need the http handler for going into nginx?

This is my current setup, which fails the SSL handshake;

With the `fly.toml` I’m unsure if we need the `http` handlers, although when I try and remove them (and update the below `nginx.conf` to look for 443 and not 8080) I get healthcheck errors.

```toml
app = "my-app"

kill_signal = "SIGINT"
kill_timeout = 5
primary_region = "lhr"
processes = []

[env]

[experimental]
  auto_rollback = true

[[services]]
  internal_port = 8080
  protocol = "tcp"

  [services.concurrency]
    hard_limit = 200
    soft_limit = 180
    type = "connections"

  [[services.ports]]
    force_https = true
    handlers = ["http"]
    port = 80

  [[services.ports]]
    handlers = ["http"]
    port = 443

  [[services.tcp_checks]]
    grace_period = "1s"
    interval = "15s"
    restart_limit = 0
    timeout = "2s"

```

With this `nginx.conf` file I’ve tried to follow a few different guides on this, this seems to be the setup that _should_ work for the basic use-case.

```plaintext
server {
  listen 8080 ssl;
  server_name test.domain.net;

  ssl_certificate /etc/ssl/certs/my-cert.crt;
  ssl_certificate_key /etc/ssl/private/my-key.key;

  error_log /var/log/nginx/test-domain.log debug;

  client_max_body_size 40M;
  port_in_redirect off;

  if ($http_x_forwarded_proto = "http") {
    return 301 https://$http_host$request_uri;
  }

  proxy_set_header X-Forwarded-Host $host;
  proxy_set_header X-Origin-Host $host;
  proxy_set_header X-Forwarded-For $remote_addr;
  proxy_set_header X-Real-IP $remote_addr;

  location /endpoint {
    proxy_pass https://proxied-app.app;
  }

```

Finally, the `Dockerfile` which really just copies the certs and the `nginx.conf` file

```plaintext
FROM nginx

RUN apt-get update \
  && apt-get install -y curl unzip

RUN curl https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip -o awscliv2.zip \
  && unzip awscliv2.zip \
  && ./aws/install \
  && rm -rf aws awscliv2.zip

ARG AWS_ACCESS_KEY_ID
ARG AWS_SECRET_ACCESS_KEY

RUN mkdir /etc/nginx/ssl

RUN aws s3 cp s3://domain-proxy-files/keys/my-cert.crt /etc/ssl/certs/my-cert.crt
RUN aws s3 cp s3://domain-proxy-files/keys/my-key.key /etc/ssl/private/my-key.key

RUN chmod 644 /etc/ssl/certs/my-cert.crt
RUN chmod 600 /etc/ssl/private/my-key.key

COPY nginx.conf /etc/nginx/conf.d/nginx.conf

```

Any help would be greatly appreciated, I’m getting to the stage where I may abandon Fly for this particular project and use something like CloudFlare.

---

<div class="post-metadata">

**Author:** ![tosbourn](https://sea1.discourse-cdn.com/fly1/user_avatar/community.fly.io/tosbourn/32/37505_2.png) [@tosbourn](https://community.fly.io/u/tosbourn)\
**Post date:** [April 5, 2023, 1:54pm UTC](https://community.fly.io/t/best-way-to-understand-ssl-errors-on-nginx-proxy/11817/5 "2023-04-05T13:54:20Z")

</div>

I’ve done some more digging and it seems like fly is still inserting itself between things.

From the deployed server;

```auto
openssl s_client -connect localhost:443

```

Returns a much different set of errors than running it from a local machine;

```auto
openssl s_client -connect test.domain.net:443

```

On the server I see;

```auto
CONNECTED(00000003)
Can't use SSL_get_servername
depth=0 O = "CloudFlare, Inc.", OU = CloudFlare Origin CA, CN = CloudFlare Origin Certificate
verify error:num=20:unable to get local issuer certificate
verify return:1
depth=0 O = "CloudFlare, Inc.", OU = CloudFlare Origin CA, CN = CloudFlare Origin Certificate
verify error:num=21:unable to verify the first certificate
verify return:1
depth=0 O = "CloudFlare, Inc.", OU = CloudFlare Origin CA, CN = CloudFlare Origin Certificate
verify return:1
---
Certificate chain
 0 s:O = "CloudFlare, Inc.", OU = CloudFlare Origin CA, CN = CloudFlare Origin Certificate
   i:C = US, O = "CloudFlare, Inc.", OU = CloudFlare Origin SSL Certificate Authority, L = San Francisco, ST = California
---

```

Which I _think_ makes sense, because I’m asking it to verify localhost when this is a cert for `test.domain.net` (I’m editing the actual domain being used of course)

When I run `openssl` locally and point it at fly, either pointing directly at the `blah.fly.dev` or pointing it at my correct subdomain (`test.domain.net`) I get;

```auto
CONNECTED(00000005)
8328020288:error:1404B42E:SSL routines:ST_CONNECT:tlsv1 alert protocol version:/AppleInternal/Library/BuildRoots/9e200cfa-7d96-11ed-886f-a23c4f261b56/Library/Caches/com.apple.xbs/Sources/libressl/libressl-3.3/ssl/tls13_lib.c:151:
---
no peer certificate available
---
No client certificate CA names sent
---
SSL handshake has read 5 bytes and written 294 bytes

```

I think this suggests that fly is doing something before the request even makes it to nginx.

---

<div class="post-metadata">

**Author:** ![kurt](https://sea1.discourse-cdn.com/fly1/user_avatar/community.fly.io/kurt/32/43757_2.png) [@kurt](https://community.fly.io/u/kurt)\
**Post date:** [April 5, 2023, 1:58pm UTC](https://community.fly.io/t/best-way-to-understand-ssl-errors-on-nginx-proxy/11817/6 "2023-04-05T13:58:08Z")

</div>

You’ll actually need to remove all handlers to get raw tcp. Right now, we’re trying to handle connections as if they’re HTTP over plaintext, which isn’t true!

Setting:

```auto
handlers = []

```

Should get you what you need.

---

<div class="post-metadata">

**Author:** ![tosbourn](https://sea1.discourse-cdn.com/fly1/user_avatar/community.fly.io/tosbourn/32/37505_2.png) [@tosbourn](https://community.fly.io/u/tosbourn)\
**Post date:** [April 5, 2023, 2:48pm UTC](https://community.fly.io/t/best-way-to-understand-ssl-errors-on-nginx-proxy/11817/7 "2023-04-05T14:48:41Z")

</div>

Thanks, that makes sense, and has got me a little further, but now (updated configs below) it is treating all requests made over https as being http.

`curl https://test.domain.net/ `

```auto
 <html>
 <head><title>400 The plain HTTP request was sent to HTTPS port</title></head>
 <body>
 <center><h1>400 Bad Request</h1></center>
<center>The plain HTTP request was sent to HTTPS port</center>
<hr><center>nginx/1.23.4</center>
</body>
</html>

```

I don’t really understand why that would be the case, I tried removing the `[[services.ports]]` for 80 to see, but no network traffic would work, I also tried removing `internal_port` since that is maybe clashing with traffic coming in, but that also breaks things.

This is my`fly.toml` now;

```auto
app = "my-app"

kill_signal = "SIGINT"
kill_timeout = 5
primary_region = "lhr"
processes = []

[env]

[experimental]
  auto_rollback = true

[[services]]
  internal_port = 443
  protocol = "tcp"

  [services.concurrency]
    hard_limit = 200
    soft_limit = 180
    type = "connections"

  [[services.ports]]
    handlers = []
    port = 80

  [[services.ports]]
    handlers = []
    port = 443

  [[services.tcp_checks]]
    grace_period = "1s"
    interval = "5s"
    restart_limit = 0
    timeout = "2s"

```

And the relevant bit of my `nginx.conf`, of note, I don’t have anything to handle 80 traffic internally, I didn’t think I’d need to?

```auto
server {
  listen 443 ssl;
  server_name test.domain.net;

  ssl_certificate /etc/ssl/certs/cert.crt;
  ssl_certificate_key /etc/ssl/private/key.key;
}

```

---

<div class="post-metadata">

**Author:** ![system](https://global.discourse-cdn.com/fly1/original/3X/6/e/6e8955d00cb218e186ebbe3a6ab06fefdce15290.svg) [@system](https://community.fly.io/u/system)\
**Post date:** [April 12, 2023, 2:49pm UTC](https://community.fly.io/t/best-way-to-understand-ssl-errors-on-nginx-proxy/11817/8 "2023-04-12T14:49:09Z")

</div>

Closed on April 12, 2023, 2:49pm UTC

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.