I followed the Elixir setup tutorial and I can’t get the ssh commands to work. It just hangs on Connecting to tunnel when I run fly ssh console. I tried the solutions mentioned in this thread to no effect. My situation may be slightly different, since I’ve never been able to make it work, as opposed to that poster having had it work initially and then stop working.
I’m not well-versed in using ssh thanks to mostly using tools like Heroku in the past, and I’m not sure if I understand the relationship between the ssh commands and Wireguard (or if there is a relationship), so I might be missing something simple, but I’m not sure how to investigate any further.
Are you on a corporate network or a VPN? Your corporate network provider may be blocking WireGuard connectivity; your VPN almost definitely is. Have you tried connecting with VPNs disabled?
Do you have a peer set up to connect to? You can check with with fly wg list to see your list of peers.
Are your instances currently accessible to you? If you follow the steps outlined in Private Networking are you able to get a successful connection into the VPN using your favourite WireGuard client?
Do you see a valid instance address when you do dig aaaa <app>.internal @fdaa:<xxx> (where @fdaa:xxx is your DNS server address from your wireguard.conf )
Okay, I’m not sure, but a VPN may have been the problem; I usually keep ProtonVPN on as much as possible. With that off, I followed the WireGuard steps, and I am able to ssh in when the WireGuard tunnel is running. I haven’t tried it via the non-WireGuard steps yet.
One issue after connecting: the symlink mentioned here doesn’t seem to exist in my app. When I run app/entry remote inside the shell, I just get /bin/sh: app/entry: not found. Has the symlink for running iex for Elixir apps changed since that guide was written?
Does a WireGuard tunnel need to be running for the fly ssh establish and fly ssh issue commands to work correctly, or are they an entirely unrelated way of doing it?
ssh establish and ssh issue actually don’t interact with WireGuard at all — they’re API-only commands. establish is a one-time command that sets up an SSH certificate for your organization — we should get rid of it and just manage that for you, but when SSH was introduced we thought it might be opt-in. issue just creates time-limited SSH user certificates to log in with.
Using establish and issue, you can use standard OpenSSH to log in directly to instances when you have a WireGuard tunnel established with your OS.
ssh console does something much more ambitious: it sets up a WireGuard tunnel on its own, using an agent process in flyctl. That tunnel is entirely in userland — your OS doesn’t know anything about it. Under the hood, it’s relying on the same certificates that establish and issue set up.
Oh interesting. So I don’t actually need to run the WireGuard app at all for that, then. Cool.
Having done some more testing on my end, it does look like the VPN was likely at least part of the problem. fly ssh console works fine with or without the WireGuard app running, but spits out this error
Error ssh: can't build tunnel for org_name: error fetching dialer: establish failed: err err handling establish: can't load organizations from config: Post "https://api.fly.io/graphql": unexpected EOF
if the VPN is on. Haven’t been able to get app/entry remote to work to launch iex once the connection is established, but things are otherwise working.
I also just updated to fly-ctl version 260 this morning from version 255 or 253 or something around there, in case that matters for any of the above.
Oh! I think the app/entry remote issue may be because I was using an older version of the Dockerfile for elixir. Updating that now and I’ll check to see if it works once I’ve got a successful deploy completed with the updated version.
Can confirm: the Dockerfile was the issue. After updating it from the setup for the old alpine build to the setup for the newer debian build, I am now able to get a running iex console via fly ssh console and then app/entry remote.
I’m not sure what the best solution to this is, but if there were a way to get notified when changes are made to the core deployment files (mainly the Dockerfile, fly.toml, and the GitHub deployment workflow stuff) so that I can have a sense in the future of when issues are happening because my deployment process is outdated, that would be a helpful feature.