Proxy TCP through nginx to a FTP server on a fly machine

Questions / Help
1

I am passing FTP connections from Filezilla on my dev PC through my nginx gateway on fly to a FTP server on another fly machine.

My nginx gateway on fly:

# fly.toml

# FTP active mode port
[[services]]
  internal_port = 2121
  [[services.ports]]
    port = 2121
    handlers = ["proxy_proto"]
    proxy_proto_options = { version = "v2" }

# FTPS to FTP active mode port
[[services]]
  internal_port = 2121
  protocol = "tcp"
  [[services.ports]]
    port = 2222
    handlers = ["tls"]

# nginx.conf
stream {
  server {
    listen 2121 proxy_protocol;
    listen [::]:2121 proxy_protocol;

    proxy_protocol on;
    proxy_pass [fdaa:2:xxxx:a7b:cb:xxxx:1034:2]:2121;
  }
}

This works just fine for unencrypted traffic on port 2121, but the connection fails with port 2222. Filezilla says:

Connection established, waiting for welcome message...
Error:	Connection closed by server
Error:	Could not connect to server

When this happens, I see absolutely nothing in the logs. Neither in the logs of my FTP server nor in the nginx logs, where access logging is active and where it does log connections using port 2121.

From my understanding of the fly documentation, I’d expect my configuration in the second [[services]] block above to a) accept connections on port 2222, b) terminate TLS there and c) forward the decrypted TCP traffic to 2121 in nginx.

I also tried without proxy_proto_options = { version = "v2" } and with handlers = ["proxy_proto", "tls"], but I can’t make it work.

I’d appreciate any help with this.

2

Hi… Order matters in handler lists, so that may have contributed to the confusion:

https://community.fly.io/t/rabbitmq-on-fly-not-working/9565/2

Hope this helps a little!

3

I already suspected that and have tried the other way around, but it does not work either. Just retried, but no luck.

4

Sorry that didn’t pan out… If you SSH into the server and try to access localhost:2121 directly—via curl or socat, for example—do you see errors in the logs?

(The idea is to establish the kinds of failures that really are being recorded, by trying something that you know in advance should fail…)

5

Thank you for your thoughts on this.

Connecting on port 2121 works just fine from end to end.
What machine do you think I should ssh into? The nginx gateway one or the one with the ftp server? And how would I use curl or socat to test a FTP/S connection?

Since nothing shows up in the logs, I was guessing that whatever goes wrong happens between the fly proxy and my nginx gateway. But I don’t know if I misconfigured anything.

6

I was thinking the Nginx gateway, since that’s the one that we most think should be logging at least a minimal error…

Basically,

$ fly ssh console -a nginx-app-name
# apt-get update
# apt-get install --no-install-recommends curl
# curl http://localhost:2121/  # will fail, but does the failure get logged?

That’s not a bad guess, of course, and there have been regressions with handlers: ["tls"] mentioned in the forums in the past.

It often helps to narrow it down from both directions, though…

7

Working at it a little from the other direction…

I was able to verify that this interpretation is in fact correct, by testing with a toy app of my own construction:

app = "sewn-cat-****"
primary_region = "***"

[[services]]
  internal_port = 2121
  [[services.ports]]
    port = 2222
    handlers = ["tls"]

And with miniature Dockerfile…

ARG DEBIAN_VERSION=bullseye-20210902-slim

ARG RUNNER_IMAGE="debian:${DEBIAN_VERSION}"

FROM ${RUNNER_IMAGE}

RUN apt-get update -y && apt-get install -y socat \
    && apt-get clean && rm -f /var/lib/apt/lists/*_*

# simply echo the incoming tcp stream's lines to `stdout`...
#
# each `-d` adds extra logging...
CMD ["socat", "-d", "-d", "-d", "TCP6-LISTEN:2121,fork", "STDOUT"]

Then, from a client machine…

$ socat OPENSSL:sewn-cat-****.fly.dev:2222,pf=IP6 STDIO
foo
bar
^D

Those lines do appear in fly logs.

[info]2023/12/29 05:53:07 socat[317] N starting data transfer loop with FDs [6,6] and [1,1]
[info]foo
[info]2023/12/29 05:53:10 socat[317] I transferred 4 bytes from 6 to 1
[info]bar

(Along with a large amount of other debugging, as requested on the CMD line.)

8

Thank you for your help with this!

Ok, so this gives me:

curl http://localhost:2121/
curl: (56) Recv failure: Connection reset by peer

And in the nginx logs (with both access_log and error_log set to /dev/stdout) :

ams [info]2023/12/29 08:49:37 [error] 1024#1024: *20 broken header: "GET / HTTP/1.1
ams [info]Host: localhost:2121
ams [info]User-Agent: curl/7.74.0
ams [info]Accept: */*
ams [info]" while reading PROXY protocol, client: 127.0.0.1, server: 0.0.0.0:2121
ams [info]STREAM -- - - 29/Dec/2023:08:49:37 +0000 TCP 400 0 0 0.000

… while the “STREAM …” comes from my access_log format definition:

log_format basic 'STREAM -- $proxy_protocol_addr - $time_local '
                   '$protocol $status $bytes_sent $bytes_received '
                   '$session_time';

As for the socat-test, I changed the listen directive in nginx from 2121 to 2120 so that I can listen on 2121 with socat from inside the nginx machine. It looks to me like it works just fine like in your example.
On my dev PC, I do socat OPENSSL:test6.xxxxxxxxxxx.com:2222,pf=IP6 STDIO and with socat -d -d -d TCP6-LISTEN:2121,fork STDOUT on the nginx machine, I can receive “foo” and “bar”.

(I have put the handlers = ["tls", “proxy_proto”] back in the correct order.)

Not quite sure how to interpret these results… What’s your take on this?

9

This is indeed pretty strange… I’m beginning to suspect a misconfiguration on the client side, at this point…

When you were trying FileZilla, were you using its FTPES:// (“explicit” FTPS) syntax or its FTPS:// (“legacy implicit” FTPS) syntax?

https://wiki.filezilla-project.org/FTP_over_TLS

:dragon_face:

10

I am doing FTP and not SFTP:

filezilla

11

There are actually three different kinds of encrypted FTP: explicit FTPS, implicit FTPS, and SFTP.

(The link above explains the differences.)

The one shown in your screenshot (“explicit FTP over TLS”) looks like explicit FTPS. This won’t work with handlers = ["tls"], since it starts out as plaintext.

Is there an “implicit TLS” or “legacy TLS” option?

12

I see, thank you a lot for your help with this.

I think I understand the difference between explicit and implicit FTP over TLS. The above errors that the filezilla client reports are indeed failing with explicit. But unfortunately, if I try implicit, I see the following errors:

Status:	Connection established, initializing TLS...
Error:	GnuTLS error -110: The TLS connection was non-properly terminated.
Status:	Server did not properly shut down TLS connection
Status:	Connection attempt failed with "ECONNABORTED - Connection aborted".
Error:	Could not connect to server

Anything I can do to fix that?

If implicit FTP over TLS is said to be “legacy”, I probably would prefer to stick with explicit mode. But I guess that won’t be possible because that would mean that I have to provide the FTP server with the wildcard SSL certificate that Fly got for my application. But I don’t get access to that certificate…

13

Is your app using a shared IPv4? If that’s the case, only HTTP over ports 80 and 443 are supported.

If you need to expose port 2121, then you’ll need to use a dedicated IPv4 or IPv6.

14

Thank you for your insights.
I do have a dedicated IPv4 for the app in question. I already needed it because I also need a wildcard certificate which I think requires a dedicated IP.

15

I’ve just come across the following statement from the flycast documentation:

Use this feature under the following circumstances: You private service needs advanced proxy features like TLS termination or proxy protocol support.

Since I do use proxy_protocol, should I set up a flycast IP and forward the traffic from the gateway to the FTP server using the flycast address instead of the internal (machine’s) address?

16

Pushing before it auto-closes…
Any ideas, @jerome ?

17

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.