Hello I have an app that runs nginx. This nginx service does its own TLS termination and acts as a reverse proxy routing requests to another service that is also hosted on fly.io. The app has the default shared ipv4 address attached. I created an A record in my DNS service (google domains) that points at this ip address. Using dig it appears the DNS changes have propagated but i am unable to access my app using my domain name. I am guessing something is misconfigured with my set up or maybe this is not supported at all.
Any insights would be appreciated.
fly.toml snipppet
note: i am attempting to let fly-proxy redirect http->https on port 80 but then forward all https traffic on port 443 to my ngnix server.
[[services]]
http_checks = []
internal_port = 443
processes = ["app"]
protocol = "tcp"
script_checks = []
[services.concurrency]
hard_limit = 25
soft_limit = 20
type = "connections"
[[services.ports]]
force_https = true
handlers = ["http"]
port = 80
[[services.ports]]
port = 443
I can post my nginx config as well but from the logs it doesn’t look traffic is making that far.
I don’t believe you can use a shared IPs with TLS termination on your app - AFAIK Fly will not do SNI-based matching (i.e. for use with shared IPs) on TLS.
I believe it’s either:
- Use a dedicated IP
or
- Let Fly proxy handle both HTTP & TLS (so your Nginx would receive the decrypted HTTPS as HTTP, with Fly handling TLS/certs/etc).
1 Like
Right on that does make sense re SNI. I will allocate a dedicated IP and see what happens. In regards to number 2 i actually tried this and still had problems:
[[services.ports]]
force_https = true
handlers = ["http"]
port = 80
[[services.ports]]
handlers = ["tls", "http"]
port = 443
Even with it still looks like nothing is making it past the fly proxy. For example chrome gives me ERR_CONNECTION_RESET and curl returns
* TCP_NODELAY set
* Connected to <domain> (66.241.124.229) port 80 (#0)
> GET / HTTP/1.1
> Host: <domain>
> User-Agent: curl/7.64.1
> Accept: */*
>
* Recv failure: Connection reset by peer
* Closing connection 0
curl: (56) Recv failure: Connection reset by peer
@joelsbrown1 is the connection reset immediate or does it take a few seconds? When I try accessing an app with no instances (included stopped instances), I see similar behavior, with the connection being reset a few seconds after it’s started. If you haven’t already, maybe try double-checking that an instance of your app is running?
1 Like
@MatthewIngwersen the connection reset is immediate. The app is up i can hit it via .fly.dev.
@Whistler ok yes that is great info. For 1 yes internal_port = 443 is still configured and you are right this doesn’t “work” but even with this setup i can see if requests make it all the way to my nginx instance. For example, when making a request to <app>.fly.dev the request is ultimately handled by my nginx instance and a 4xx with nginx error page is returned because like you said Nginx is listening for TLS on 443 and it is being sent HTTP.
To your second point I did set up TLS termination but i did not generate certs so yeah this is probably why this still didn’t work. I created a dedicated ipv4 IP and i am repointing DNS now. it seems like this is likely the only way to get this to work. just waiting for DNS propagation now. will report back. thanks again.
I looked at your app. Port 443 is still configured as using TLS terminated by fly-proxy - you need to remove the handlers = ["tls", "http"] line from your fly.toml, otherwise your nginx will never receive TLS data.
1 Like
@MatthewIngwersen @Whistler @lillian thank you all for responding so quickly to this thread. The quality of free support in these forums is unreal and honestly was not expecting it to be so good.
Ok so i think public networking is now working as expected:
- A record DNS entry is pointing at my apps public dedicated ipv4 IP.
- Fly proxy is redirecting http->https and proxying raw HTTPS request to my app with this config:
[[services]]
http_checks = []
internal_port = 443
processes = ["app"]
protocol = "tcp"
script_checks = []
[services.concurrency]
hard_limit = 25
soft_limit = 20
type = "connections"
[[services.ports]]
force_https = true
handlers = ["http"]
port = 80
[[services.ports]]
port = 443
I can see requests in the nginx logs now. Unfortunately something funky is going on with my private networking now as my nginx instance is returning 502 and i am seeing this in the logs:
2023/03/22 22:42:24 [error] 547#547: *13 recv() failed (104: Connection reset by peer) while reading response header from upstream, client: 172.16.131.106, server: <domain>, request: "GET / HTTP/1.1", upstream: "http://[fdaa:1:2944:0:1::4]:8080/", host: "<domain>"
ok yeah this looks like a problem with my upstream.
I added a private flycast ip to my app server and was trying to proxy using proxy_pass http://app-name.flycast:8080
I am realizing that is not going to work as my upstream app is not listening on 8080 externally. At this point i think i have enough to sort this out so i am going to mark this is solved and open up a new thread if i run into new issues.
Thank you again everyone!
You got it! Flycast addresses go through the proxy and don’t directly hit your app’s VM, so you need to set up services in your fly.toml for that, as you would for a public service.
1 Like