Cannot Connect to Machines on internal Service Port

I’m having trouble connecting to a launched machine on exposed service ports. I created a machine using the REST API as described in the docs. This machine can be returned via the API:

[
  {
    "id": "3287194b055785",
    "name": "default",
    "state": "started",
    "region": "sea",
    "instance_id": "01GFYZEY7ZD3X7GE5TFSBX2MSA",
    "private_ip": "fdaa:0:b9ba:a7b:2dbb:a7e3:c3ce:2",
    "config": {
      "env": {
        "ADMIN_SERVER_BASE": "0.0.0.0",
        "ADMIN_SERVER_HTTP_PORT": "8080"
      },
      "init": {
        "exec": null,
        "entrypoint": null,
        "cmd": null,
        "tty": false
      },
      "image": "registry.fly.io/capemay-test:abdb81c",
      "metadata": null,
      "mounts": [
        {
          "encrypted": false,
          "path": "/db",
          "size_gb": 0,
          "volume": "vol_ke628r69ww1rwmnp"
        }
      ],
      "restart": {
        "policy": ""
      },
      "services": [
        {
          "protocol": "tcp",
          "internal_port": 8080,
          "ports": [
            {
              "port": 80,
              "handlers": [
                "http"
              ],
              "force_https": true
            },
            {
              "port": 443,
              "handlers": [
                "http",
                "tls"
              ]
            }
          ]
        }
      ],
      "guest": {
        "cpu_kind": "shared",
        "cpus": 1,
        "memory_mb": 256
      },
      "metrics": null
    },
    "image_ref": {
      "registry": "registry.fly.io",
      "repository": "capemay-test",
      "tag": "abdb81c",
      "digest": "sha256:b64586ca091b0d0605dc146bf9af293ff89ed2955099675ba7915d2cedf0507e",
      "labels": {}
    },
    "created_at": "2022-10-22T04:23:22Z",
    "updated_at": "2022-10-22T04:33:26Z",
    "events": [
      {
        "type": "start",
        "status": "started",
        "source": "flyd",
        "timestamp": 1666413206920
      },
      {
        "type": "launch",
        "status": "created",
        "source": "user",
        "timestamp": 1666413197580
      }
    ]
  }
]

It is also returned via flyctl:

❯ fly machines list -a capemay-test                                                                                                  
1 machines have been retrieved.
View them in the UI here (​https://fly.io/apps/capemay-test/machines/)

capemay-test
ID              NAME    STATE   REGION  IMAGE                   IP ADDRESS                              VOLUME                  CREATED                 LAST UPDATED         
3287194b055785  default started sea     capemay-test:abdb81c    fdaa:0:b9ba:a7b:2dbb:a7e3:c3ce:2        vol_ke628r69ww1rwmnp    2022-10-22T04:23:22Z    2022-10-22T04:33:26Z

When connected to the WireGuard network I can ping the machine at the IPv6 IP:

❯ ping6 fdaa:0:b9ba:a7b:2dbb:a7e3:c3ce:2
PING6(56=40+8+8 bytes) fdaa:0:b9ba:a7b:177d:0:a:2 --> fdaa:0:b9ba:a7b:2dbb:a7e3:c3ce:2
16 bytes from fdaa:0:b9ba:a7b:2dbb:a7e3:c3ce:2, icmp_seq=0 hlim=62 time=10.089 ms
16 bytes from fdaa:0:b9ba:a7b:2dbb:a7e3:c3ce:2, icmp_seq=1 hlim=62 time=8.510 ms
16 bytes from fdaa:0:b9ba:a7b:2dbb:a7e3:c3ce:2, icmp_seq=2 hlim=62 time=8.816 ms

However, I cannot connect via the exposed port:

❯ curl -g -6 -X GET 'http://[fdaa:0:b9ba:a7b:2dbb:a7e3:c3ce:2]:80'                                                                                             
curl: (7) Failed to connect to fdaa:0:b9ba:a7b:2dbb:a7e3:c3ce:2 port 80 after 8 ms: Connection refused

capemay/core on  main   via  impure (nix-shell) 
❯ curl -g -6 -X GET 'http://[fdaa:0:b9ba:a7b:2dbb:a7e3:c3ce:2]:8080'
curl: (7) Failed to connect to fdaa:0:b9ba:a7b:2dbb:a7e3:c3ce:2 port 8080 after 7 ms: Connection refused

capemay/core on  main   via  impure (nix-shell) 
❯ curl -g -6 -X GET 'https://[fdaa:0:b9ba:a7b:2dbb:a7e3:c3ce:2]:443'
curl: (7) Failed to connect to fdaa:0:b9ba:a7b:2dbb:a7e3:c3ce:2 port 443 after 11 ms: Connection refused

Also, the output of the app doesn’t show the service ports or region (despite the API responding with it):

❯ fly info -a capemay-test                                                                                                           
App
  Name     = capemay-test          
  Owner    = personal              
  Version  = 0                     
  Status   = deployed              
  Hostname = capemay-test.fly.dev  

Services
PROTOCOL PORTS 

IP Addresses
TYPE ADDRESS            REGION CREATED AT           
v6   fdaa:0:b9ba:0:1::2        2022-10-22T04:48:32Z 

Anyone have advice?

Over 6pn, one must connect to internal_port (8080?), not the (public) exposed one. But I see you did so, and it didn’t work.

If you do want to connect to public ports over 6pn, I think (not sure) you can do so over a Flycast address, which you must explicitly assign to an app.

For a Machine app, instead exec fly m status <vm-id> -d -a <machine-app-name> (docs).

If you haven’t already, then fly ssh console "[6pn-addr]" (ref) and exec netstat -alpn to see if things are listening on ports you expect them to.

Oh, and if the app isn’t, then it must listen on :: or fly-local-6pn (ref, more).

This was the issue. I was only listening on an v4 address. Thanks for your help!

I wonder if there’s a spot in the docs I should PR to add this.

1 Like

Just ran into this today, too. Was using https://www.meilisearch.com/ and couldn’t get it work as expected. Would be great to add a note somewhere about this if possible. Maybe it exists somewhere already and I missed it :slight_smile:

1 Like

@markthethomas we just added it here. Private Networking · Fly Docs

Great!