Exposing port 22 from an app to the internal network

I’ve got a webapp (Gitea) running, fronted by nginx, and everything is working fine. Gitea is a GitHub-like thing, and so it listens for SSH connections on port 22, which I map to another port like this:

[[services]]
  internal_port = 22
  protocol = "tcp"

  [[services.ports]]
    handlers = []
    port = 3022

and nginx upstream’s SSH traffic to that port. However, this also exposes port 3022 to the internet, which I would like to avoid doing.

Gitea runs an HTTP server on port 3000, and nginx can proxy traffic to that via the internal network, (i.e. without having to define a [[services.ports]] section for it), but I can’t do the same thing for port 22, I suspect because you’re using it for normal SSH traffic.

Is it possible to remap a port from inside a running app to the outside (i.e. like docker run -p 22:3022), but only for the internal network?

Port 22 is indeed used internally to make fly ssh console work. SSH is listening for connections on the private network specifically.

To get SSH working myself, I’ve had SSH listen on a port that was not 22 (and then you can map whatever port you want to it, or no ports at all).

For Gitea, it looks like setting env var SSH_PORT=8022 can help you there. (reference)

I think you’ll need/want to do that so Gitea doesn’t use port 22 at all (and hopefully the configuration is smart enough to configure Nginx to send to the correct upstream/port based on SSH_PORT).

Since Nginx is presumably running in the same VM, it should be able to reach SSH without you have to setup any services (You can think of services as using the -p 1234:1234 flag in Docker - it exposes ports to the world, but things listening in the container (VM on Fly) will still be listening for connections even if you don’t define a service).

What I don’t see in your Services definition is setting up Nginx’s port 3000 so that can be reached from the outside world. It sounds like you need that?

Since Nginx is presumably running in the same VM

I’ve got Gitea and nginx running in separate VM’s.

Gitea declares its HTTP port, but without a public service:

app = "my-gitea" ;
[[services]]
  http_checks = []
  internal_port = 3000
  protocol = "tcp"
  script_checks = []

nginx exposes a public 80/443:

app = "my-nginx"
[[services]]
  internal_port = 8080
  protocol = "tcp"
  script_checks = []

  [[services.ports]]
    force_https = true
    handlers = [ "http" ]
    port = 80

  [[services.ports]]
    handlers = [ "tls", "http" ]
    port = 443

and proxies HTTP traffic to Gitea over the internal network:

server {
    listen 8080 ;
    server_name my-nginx.fly.dev ;
    location / {
        proxy_pass http://my-gitea.internal:3000 ;
    }
}

But the same technique doesn’t work for port 22 e.g. for Gitea:

[[services]]
  internal_port = 22
  protocol = "tcp"

and for nginx:

[[services]]
  internal_port = 8022
  protocol = "tcp"

  [[services.ports]]
    handlers = []
    port = 8022

and:

upstream ssh_target {
    server my-gitea.internal:22 ;
}

server {
    listen 8022 ;
    proxy_pass ssh_target ;
}

I can connect to Gitea over SSH on port 22 from inside the Gitea VM, but not from outside. It has to be because fly.io is using port 22 for itself. The only way I could get things to work was to create a service mapping Gitea’s port 22 to 3022, and have nginx proxy traffic to my-gitea.fly.dev:3022, but this also makes port 3022 accessible from the internet.

For Gitea, it looks like setting env var SSH_PORT=8022 can help you there.

That setting is used to control what gets shown in UI. Gitea can actually be configured to run its own SSH server, but I would need to build the Docker image from scratch (so that I can expose the custom port), instead of using the stock image from Docker Hub. That’s why I was hoping there was a simple way of remapping a port, like [[services.ports]], but for the internal network only.

Exposing the port on the open internet isn’t a security risk per se - anything a hacker could do on that port, they can do on the public port that nginx is proxying - but it’s kinda icky :expressionless:, and if I ever want to e.g. put access controls on the port (e.g. to block abusive IP’s), they could be bypassed by going to the Gitea port directly.

Thanks!

  1. It sounds like using SSH_PORT=8022 in the Gitea VM should have it listen on a port that’s not 22, which might help?
  2. Then your Nginx upstream can use port 8022 (or whatever you decide) instead of port 22.

Have you tested that outside of Fly? I ask because Nginx needs to be proxying a tcp connections for SSH. I’m assuming you’re just not showing that config’s stream {} block for brevity, but I wanted to mention it just in case.

(The SSH_PORT thing was based on the info here: Configuring an SSH port other than 22 disables START_SSH_SERVER · Issue #7361 · go-gitea/gitea · GitHub, it looks like there may be several env vars that can accomplish that )

It sounds like using SSH_PORT=8022 in the Gitea VM should have it listen on a port that’s not 22

The Gitea docs are pretty clear that this setting is just for what’s shown in the UI:
SSH_PORT: 22: SSH port displayed in clone URL.

IOW, this thing:

ssh-clone

netstat shows that inside the VM, only port 22 is listening, not 8022.

Further down the Gitea doco page, there are other SSH settings, but these are for their built-in SSH server. By default, it just uses the system SSH server, which will be listening on port 22.

Have you tested that outside of Fly?

Yes, and it also works inside Fly, but only with a service mapping port 22 to something else. But this makes that port available on the internet, which is what I’m trying to avoid.

I’m assuming you’re just not showing that config’s stream {} block for brevity

I posted it earlier, but here it is again:

upstream ssh_target {
    server my-gitea.internal:22 ;
}

server {
    listen 8022 ;
    proxy_pass ssh_target ;
}

This is what I want to do, but SSH keys aren’t being accepted, presumably because it’s connecting to Fly’s SSH server outside the VM, instead of the one running inside the Gitea VM. The only way I could get it to work was to send SSH traffic to my-gitea.fly.dev:3022 i.e. explicitly to the SSH server running inside the Gitea VM, not Fly’s outside the VM, but this also exposes port 3022 to the internet.

It should be noted that, unless Fly do some funkiness to avoid the issue, this behaviour is exactly what you would expect to see.

Things are working, I just want to be able to map port 22 from inside the VM to another port outside the VM, similar to [[services.ports]], but on the internal network only. It should also be possible to do it by fiddling with Gitea and rebuilding the Docker image from scratch, but it would be nice to not have to do that.

I found that the gitea/gitea docker image runs openssh and the listening port can be configured with the SSH_PORT env var. SSH_LISTEN_PORT also works.

How about adding this to your fly.toml?

[env]
  SSH_PORT = "3022"

Once deployed, look for these logs from your gitea app to see what port the openssh server is listening on:

Server listening on :: port 3022.
Server listening on 0.0.0.0 port 3022.

How about adding this to your fly.toml?

[env]
  SSH_PORT = "3022"

After a lot of screwing around :expressionless: , I finally got things working.

I had tried setting SSH_LISTEN_PORT in Gitea’s app.ini, but it turns out that this is known (and undocumented) to not work in a Docker container :expressionless:

At startup, Gitea generates /etc/ssh/sshd_config from a template, and the port number is taken from this environment variable (my nginx is listening on a different port to Gitea, so I set SSH_LISTEN_PORT, not SSH_PORT).

Turns out, this is all you need to do (although it would probably be prudent to still configure it in app.ini), and you then no longer need to create a service for the port, and nginx can proxy SSH traffic to it over the private network.

Interestingly, you don’t need to create a new Dockerfile to EXPOSE the new port. From my understanding of how Fly works, this makes sense - it unpacks the Docker image into a VM and runs things there - but seems to be a bit of security hole. You control access in and out of a container by mapping ports and volumes, and so, strictly speaking, this port shouldn’t be available outside the VM (since it hasn’t been EXPOSE’d). Since apps run on a network that is private to the organization, this is probably not a big issue, but it’s something to be mindful of.

Thanks to everyone for their help - this was a little thing, but it was bugging me… :slight_smile:

1 Like

Please split this into a different topic if needed.

How did you get SSH working yourself? I am attempting to host a git server as well (not running anything like gitea, just a git server so i can do git push origin main and push it to a volume on my fly app)

Thanks!

How did you get SSH working yourself? I am attempting to host a git server as well

One thing I tried was to change the port number in sshd_config by doing an in-place sed. In my case, it didn’t work because when Gitea starts up, it generates the entire file from a template and overwrites the change (which is why setting the env.var. works), but if you’re just using a bog-standard base image, it should work. It sucks that you have to use a non-standard port, but I don’t think there’s anything you can do about that.

Figured it out, answer is here: Create new user with home dir in a volume - #2 by mhanberg