How can I use a Fly.io to reverse proxy SSH from a private network?

[halfer]

Introduction

I have recently been on the hunt for a reliable reverse proxy to tunnel into my private home network. It turns out not to be the trivial problem I envisaged! I thought Fly.io might be a great component in this system, and wondered if anyone in the community had tried it (or sees a networking flaw in my cunning plan).

The problem for most domestic server owners is that creating an SSH connection from the outside world into the house needs a very specific set-up. Firstly, one needs a public IP address, ideally a static one. Most domestic connections don’t have a static address, and folks on 4G/5G don’t even have a public one. Folks who do have a public one have to set up port-forwarding rules in their router, so port 22 reaches the right device on the internal LAN. In short, this is a bit of a hassle.

General solution

So the solution is for the domestic server to run some software that reverse-proxies a service (e.g. SSH or web etc.) by making an outbound connection to a bastion server. The user would just make a connection to the bastion, and software on the bastion accepts an outside world connection, and then creates a two-way path between the two.

In my case the internal device is a Raspberry Pi, but it can be any box capable of running a server. I mostly need SSH, but web/80 with Let’s Encrypt in front would be a great bonus.

Specific solutions

I looked at Self-Hosted Gateway from Fractal Networks, but didn’t like that the private LAN component uses Docker. I love Docker, but it looked like it is trying to share other containers, rather than sharing servers that are part of the default host build (SSH in particular).

I then looked at CloudFlare Tunnel. Although this promises to be a similar solution, it was a totally rotten dev experience. I got the daemon working, and this was reported in the web console, but it wasn’t clear what CF public endpoint I needed to use to carry the SSH traffic. I suspect that it was trying to force me to put a domain under their DNS management, and since that’s not a pain I was willing to go through, I was effectively blocked.

By way of illustrating the complexity of this problem, here is a list of attempts. I assume that each software author hasn’t been satisfied with existing art, and is having another go!

Finally I settled on PiTunnel, which took five minutes to set up. The user experience is excellent, but since there is no information on the website about the author or the company, I am not sure I want it in my network in the long term. But the free solution is OK for now, even if the SSH port changes every so often - presumably to “encourage” users to upgrade to a paid solution.

Using Fly.io

So I am seeking some pointers in the right direction. I have been told that I might look into SOCKS (not the woolly kind, apparently). Or maybe I need to run sshd in a Fly.io container, even though I don’t know if this uses TCP or UDP, and I’m pretty sure the networking minutiae is beyond my ken.

There’s some hints here that SSHing via the Fly command is already possible, and that one might look into Wireguard for this. But it is all still a bit hand-wavy for me presently - what repos/technologies could I look into next? I spent several hours on Cloud Flare to no good result, so could do with some initial focus.

(I wonder if I might look again at the one from Fractal Networks - maybe the private network daemon in Docker could connect to the host via host.docker.internal).

[lillian]

This is a fairly opinionated topic (as you saw with the awesome-tunneling repo)… here’s some thoughts that might help:

• SSH runs over TCP.

lillian@server:~$ sudo netstat -tulpn
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address    Foreign Address    State    PID/Program name    
tcp        0      0 0.0.0.0:22       0.0.0.0:*          LISTEN   899/sshd: /usr/sbin 

• The flyctl ssh command is used to connect to a VM running on Fly.io. I think you might be able to set it up so it is able to connect to your home computer, but it’s not intended to be used like that.

• most “tunneling” providers expect you to run a website, or something publicly accessible. (Fly.io, while not being explicitly a tunneling provider, does so as well, so there are some “gotchas” to be aware of.)
  This is why Cloudflare Tunnel expects you to register and add a domain.

• you mention that setting up port forwarding rules is a hassle, but if you do have a dedicated public IP (even if it changes once in a while), tunneling through a cloud server is likely a lot more effort than just doing that!

• if your public IP changes often, you can use a DDNS service to get a domain name that is automatically updated to your current public IP. It’s possible to do this with Cloudflare, but you will need to own a domain; there are other services that give you a DDNS enabled subdomain (sometimes for free).

All of that aside, here is how I would set up a Fly.io app that proxies SSH from a static public IP:

(I am assuming you use Ubuntu or another Debian-based Linux distribution, and have flyctl installed and logged in.)

1. Install wireguard-tools package

$ sudo apt install -y wireguard-tools
[...]

2. Create a Wireguard configuration to connect to the Fly.io private networking from your server

$ flyctl wireguard create
? Select Organization: lillian@fly.io (personal)
Creating WireGuard peer "interactive-server-lillian-fly-io-01GZVPVV4D88Y8DA55PPG2TKFK" in region "yyz" for organization personal

!!!! WARNING: Output includes private key. Private keys cannot be recovered !!!!
!!!! after creating the peer; if you lose the key, you'll need to remove    !!!!
!!!! and re-add the peering connection.                                     !!!!
? Filename to store WireGuard configuration in, or 'stdout':  fly-wireguard.conf
Wrote WireGuard configuration to fly-wireguard.conf; load in your WireGuard client

3. Connect to Wireguard

$ sudo wg-quick up ./fly-wireguard.conf
[#] ip link add fly-wireguard type wireguard
[#] wg setconf fly-wireguard /dev/fd/63
[#] ip -6 address add fdaa:0:fb94:a7b:9285:0:a:102/120 dev fly-wireguard
[#] ip link set mtu 1420 up dev fly-wireguard
[#] ip -6 route add fdaa:0:fb94::/48 dev fly-wireguard

4. Create a Fly.io app, and assign a dedicated IP

note that by default, apps get a shared IP, which is free but only supports HTTP services

$ fly apps create --name my-ssh-proxy
New app created: my-ssh-proxy
$ fly ips allocate-v4 -a my-ssh-proxy
? Looks like you're accessing a paid feature. Dedicated IPv4 addresses now costs $2/mo. Are you ok with this? Yes
VERSION	IP           	TYPE  	REGION	CREATED AT 
v4     	169.155.58.67	public	global	10s ago   	

5. Write the following fly.toml configuration file:

app = "my-ssh-proxy"
primary_region = "yyz"

[build]
image = "alpine"

[processes]
app = "sh -c 'apk add socat && socat tcp-l:22,fork,reuseaddr tcp:[$HOME_SERVER_WG_IP]:22'"

[env]
HOME_SERVER_WG_IP = "fdaa:0:fb94:a7b:9285:0:a:102" # put your own ip here

[[services]]
  internal_port = 22
  protocol = "tcp"
  processes = ["app"]
  auto_stop_machines = true
  auto_start_machines = true

  [[services.ports]]
    port = 22

6. Run fly deploy

$ fly deploy
==> Verifying app config
Validating /home/lillian/fly.toml
Platform: machines
✓ Configuration is valid
--> Verified app config
==> Building image
Searching for image 'alpine' remotely...
image found: img_n37qpo2nnxj4mz69

Watch your app at https://fly.io/apps/my-ssh-proxy/monitoring

Process groups have changed. This will:
 * create 1 "app" machine

No machines in group app, launching one new machine
  Machine 178116ddc13d89 [app] update finished: success
Creating a second machine to increase service availability
  Machine 9080562f650587 [app] update finished: success
Finished launching new machines

NOTE: The machines for [app] have services with 'auto_stop_machines = true' that will be stopped when idling

Updating existing machines in 'my-ssh-proxy' with rolling strategy
  Finished deploying

7. SSH to your server using the dedicated iP you created earlier:

Do note that since your server is publicly accessible, you should use key-based authentication and disable password authentication, but that is beyond the scope of this comment.

[bermudi]

I would like to suggest taking a look at https://shadowsocks.org/ if all you need to do is access your raspberry pi from anywhere