fly ssh with a custom network ID

Questions / Help
1

Hello, I’m trying to ssh on to my fly instance (that is running HAProxy) in order to debug why it’s not able to send traffic to a WireGuard peer.

The caveat here is that I’m setting a custom network ID when creating the fly app and adding a WireGuard peer to that custom network.

As a sanity check, I am able to ssh on to my fly instance (and successfully proxy traffic) when I create a fly app using the “default” network ID (i.e. not specifying one).

These are the steps to reproduce the issue:

FLY_REGION="lhr"
WG_PEER_NAME="jason-tp"
WG_PEER_NETWORK="jason-tp-network"
FLY_API_TOKEN="YOUR-FLY-API-TOKEN"
FLY_ORG_ID="YOUR-FLY-ORG-ID"
APP_NAME="tpproxy-0s48qw"

The fly.toml file:

app = "tpproxy-0s48qw"
kill_signal = "SIGINT"
kill_timeout = 5

# Remove this section to build from the local Dockerfile
[build]
  image = "us.gcr.io/tinypilot-public-images/fly-haproxy:latest"

[experimental]
  private_network=true

[[services]]
  internal_port = 8085
  protocol = "tcp"

  [services.concurrency]
    hard_limit = 25
    soft_limit = 20

  [[services.ports]]
    handlers = ["tls", "http"]
    port = "443"

  [[services.tcp_checks]]
    grace_period = "1s"
    interval = "15s"
    port = "8085"
    restart_limit = 6
    timeout = "2s"

Create a new app using a custom network ID:

curl 'https://api.fly.io/graphql' \
  -H 'Content-Type: application/json' \
  -H "Authorization: Bearer ${FLY_API_TOKEN}" \
  --data '{
  "query": "mutation($input: CreateAppInput!){ createApp(input: $input) { app { id name organization { id slug } network runtime regions { name code } } } }",
  "variables": {
    "input": {
      "name": "'"${APP_NAME}"'",
      "runtime": "FIRECRACKER",
      "organizationId": "'"${FLY_ORG_ID}"'",
      "preferredRegion": "'"${FLY_REGION}"'",
      "network": "'"${WG_PEER_NETWORK}"'"
    }
  }
}'; echo

Note: I’ve purposely omitted the step where I add a WireGuard peer to the custom network, as it doesn’t affect my ssh issue

Deploy the instance

fly deploy \
  --env "EXTERNAL_PORT=8085" \
  --env "TINYPILOT_PROXY_TARGET=127.0.0.1:443"

The output of a successful deploy:

Deploying tpproxy-0s48qw
==> Validating app configuration
--> Validating app configuration done
Services
TCP 443 ⇢ 8085
Searching for image 'us.gcr.io/tinypilot-public-images/fly-haproxy:latest' remotely...
image found: img_ylj9x4dl1evwo1kq
Image: us.gcr.io/tinypilot-public-images/fly-haproxy:latest
Image size: 37 MB
==> Creating release
Release v1 created

You can detach the terminal anytime without stopping the deployment
Monitoring Deployment

1 desired, 1 placed, 1 healthy, 0 unhealthy [health checks: 1 total, 1 passing]
--> v1 deployed successfully

Now try to ssh onto the instance:

fly ssh console --app "${APP_NAME}" --command  bash

SSH fails with:

Connecting to tpproxy-0s48qw.internal...⣟ Error connect to SSH server: dial: lookup tpproxy-0s48qw.internal. on fdaa:0:28a6::3: no such host

Any advice to why I am not able to ssh onto the fly instance, would be greatly appreciated :slight_smile:

1 Like
2

Oh this is definitely a gap in flyctl. The SSH command doesn’t understand the network IDs yet, and transparently creates a wireguard peer for you. So it’s just landing you in the wrong network namespace.

We really need to teach flyctl about networks, I’ve created an issue here: Teach flyctl about app networks · Issue #417 · superfly/flyctl · GitHub

It’s possible that you can use your own wireguard peer + connection, then fly ssh establish and fly ssh issue to get hooked up with SSH, but I need to double check.

2 Likes
4

Thanks @kurt, I ended up deploying a shellinabox as an easy way to debug my networking issue. Turns out the HAProxy and WireGuard peer are not in the same network.

I posted another question here to get some advice to why that might be happening.