502 on nginx reverse proxy

[CitizenAbacus]

Hi, everyone, thanks in advance for any help.

I am testing out occluding my apps behind an internal IP and exposing only specific APIs through a reverse proxy. As a test of this in Fly I have a simple Micronaut app with a “Hello” endpoint. This would then be accessible through an NGINX reverse proxy. When I navigate to http://backend-one.fly.dev I get the “Hello” response as expected. Hooray. I then set up the reverse proxy with this nginx.conf.

events {}

http {
	server {
		listen 5005;
		listen [::]:5005;
		server_name _;
		
		location /public {
			proxy_pass 		http://backend-one.fly.dev/;
			#proxy_pass		http://host.docker.internal:8081/;
			proxy_set_header 	X-Forwarded-Host $http_host;
			proxy_ssl_name 		$host;
			proxy_ssl_server_name 	on;
		}
		
		location /internal {
			proxy_pass 		http://backend-one.internal/;
			#proxy_pass		http://backend-one:8080/;
			proxy_set_header 	X-Forwarded-Host $http_host;
			proxy_ssl_name 		$host;
			proxy_ssl_server_name 	on;
		}
	}
}

The commented out lines are how it runs locally in Docker, where it behaves as expected. For both of these apps I am using the same Dockerfile for local deployed to Fly.

The nginx container starts up, so I know that backend-one.internal is a valid host.

The /public location is able to access my “Hello” endpoint, although it redirects the browser rather than proxying as I would expect.

The /internal location, however, gives a 502. The error in the nginx logs read

 2025-08-16T01:24:12.707 app[6e827140b70d98] ord [info] 172.16.1.106 - - [16/Aug/2025:01:24:12 +0000] "GET /internal HTTP/1.1" 502 157 "-" "curl/8.5.0"

2025-08-16T01:24:12.707 app[6e827140b70d98] ord [info] 2025/08/16 01:24:12 [error] 658#658: *1 connect() failed (111: Connection refused) while connecting to upstream, client: 172.16.1.106, server: _, request: "GET /internal HTTP/1.1", upstream: "http://[fdaa:9:151a:a7b:563:4053:1022:2]:80/", host: "reverse-proxy-bold-wave-8884.fly.dev" 

These are similar to the log messages posted locally, but in that case the proxy works as expected. It’s worth noting that the Micronaut container has three IP addresses, two public and one private, and neither is the one given in the second message. My suspicion is that the reverse proxy is trying to redirect rather than proxy, the same way it was doing for the public host, which we would expect to fail.

Anyway, I’m stumped on this one. Does anyone have any insight?

[mayailurus]

@CitizenAbacus

proxy_pass http://backend-one.internal/;

Hi… I’m definitely not one of the local Nginx experts, but it’s the weekend and August, so…

With the .internal addresses, you need to specify the port that the app itself (Micronaut, in this case) is directly listening on. There isn’t a mechanism in the middle that translates port 80 → port 8080.

To see this in action, try SSHing into the Nginx Machine, install curl there (if it isn’t already), and then curl -i http://backend-one.internal:8080/ (or whatever the real port is).

Another thing that seems to trip people up with Nginx is the fact that the .internal addresses don’t auto-start, so be sure that your backend-one app always has at least one Machine awake.

The official doc for all this can be found here:

https://fly.io/docs/networking/app-services/#private-services-on-6pn-direct-wireguard-connections

(That same overall page also mentions a third kind of address, .flycast, which has aspects of both.)

@CitizenAbacus

The /public location is able to access my “Hello” endpoint, although it redirects the browser rather than proxying as I would expect.

My guess is that this was just repeating the 301 redirect that it got from Fly Proxy—on the backend’s .fly.dev address in that case—although, like I said, I don’t know Nginx that well.

$ curl -i 'http://backend-one.fly.dev/'
HTTP/1.1 301 Moved Permanently
location: https://backend-one.fly.dev/

(Probably you want http://backend-one.internal:8081/ as your proxy_pass for this part (/public), but I could be guessing your intent incorrectly.)

I’d suggest testing with curl -i instead of the browser, so you can see such things in detail…

Hope this helps!

[CitizenAbacus]

Including the port for the internal address absolutely fixed it, thanks.

I didn’t bother to post my cURL records before because all they just gave the response as expected for the public location. I realize now that it was because I was doing curl -L instead of curl -i. Oops. It’s still not clear to me why Nginx is doing a 301, but I’m sure I can figure that out.