502 / bad gateway errors

1

hello Fly friends, it’s totally possible I messed something up here, but it appears my simple Fly Nginx proxies are giving me 502 errors with “tlsv1 alert access denied”. This started to happen as of Friday. See nikola-proxy for an example, though it happens with all my proxy service.

I assumed at first the 502s were coming via the service I was proxying to, but it looks like it might be happening in Fly. I’m working now on eliminating variables, etc, but when I sidestep the proxies, the 502 errors go away. This leads me to believe it might be a fly thing.

Any ideas?

Thank you!

2

Looking into this.

It might be due to the fact that the app is old-ish and it was before we issued a fix for “disappearing” certificates.

I’ll have you back in no time.

1 Like
3

okay thank you! I also re-deployed just now to see if that would do anything.

4

The 502 is not from us. Looks like a nginx 502.

I’ve “revived” your certificate. Handshakes are working on my end.

1 Like
5

Okay thank you, I will keep digging. Out of my depth re:nginx, but it’s such a simple config I’m surprised it broke like this!

6

Looks like you’re back? What was the issue?

7

I don’t think I’ve figured out the issue unfortunately. Still having 502s :frowning:

Not sure why Nginx is whining. And sadly don’t have the logging I need to debug this, so I’m working on that now.

8

Just a random guess here, but I’ve seen this error before with proxies when the origin and proxy have mismatched TLS settings. These were either due to unsupported TLS versions on either side, or an incorrect SNI hostname or Host header setting.

1 Like
9

I will look in to that, thank you! Much to learn on this front. FWIW, it looks like I do have some useful logging here actually. Nginx giving lots of “[error] 515#515: *77695 peer closed connection in SSL handshake (104: Connection reset by peer) while SSL handshaking to upstream, client:”

10

I’m getting a 404 visiting your app’s / path.

I do see a lot of 502s in your metrics though.

Can you try deploying the app? I wonder if nginx is “stuck” somehow.

11

not sure about the 404s. Probably because the / path on the server I’m proxying to I think 404s?

Re-deploying now.

12

Looks like the upstream you’re trying to reach has an invalid cert.

image
image1048×640 70.4 KB

Unless your nginx config is configured to trust that certificate or to ignore unverified peers, then it won’t be able to communicate with the upstream.

1 Like
13

Oh wow! I’ll investigate that. For my future learnings, how did you figure that out?

Thank you

14

Looking at your logs :slight_smile:. The logs from nginx shows which upstream they’ve encountered the error from:

e.g.

2021/05/10 18:14:07 [error] 515#515: *155052 peer closed connection in SSL handshake (104: Connection reset by peer) while SSL handshaking to upstream, client: <redacted>, server: proxy.nikolaapp.com, request: "GET /api/1/vehicles/<redacted>/ HTTP/1.1", upstream: "https://<useful ip>:443/api/1/vehicles/<redacted>/", host: "proxy.nikolaapp.com"

so I checked <useful ip>:433 and the certificate seemed invalid.

I’m not entirely sure if you put an IP in your config or if nginx resolved it to that IP and shows the latter in the logs though :thinking:

You might want to use a hostname there so it sends a SNI, perhaps you’ll get a proper cert? Maybe you’re already doing that.

If you don’t mind sharing your nginx config, I might be able to help a bit more!

1 Like
15

emailed you my config, thank you. I believe I am using a hostname. I wonder if perhaps the upstream server is sometimes using a valid cert and sometimes using an invalid one.

also, thank you so so much for your help here.

16

looks like there are a bunch came records linked to each other. I’m going to try to deploy with the last one in the chain.

1 Like
17

Jerome and I figured it out. Unsurprisingly, it was my fault! Thanks for your help everyone!