Cannot connect to websocket server

Questions / Help
,
1

I’ve been using fly.io for a long time and it’s a great service. I recently tried to host an application that has a websocket server (port 3500) and a web server (port 4000). So I added this to fly.toml:

[http_service]
internal_port = 4000
force_https = true
auto_stop_machines = false
auto_start_machines = false
min_machines_running = 0
processes = ["app"]

[services]
internal_port = 3500
protocol = "tcp"
  [[services.ports]]
  handlers = ["http"]
  port = 5600

But when I try to connect to port 5600 (on my app’s IP address), it fails. When trying to test the connection with curl, it gives me “Connection reset by peer” error.
I ssh’ed into the app using fly ssh console command and tried to test my app using curl and it works there locally.

Is there anything I’m doing in the wrong way?

1 Like
2

I have the exact same problem.

3

I’m used to seeing websockets on the same port but different path. Rails does this by default, and we provide a demo that does this with Node: Vanilla with Candy Sprinkles · The Fly Blog, and other frameworks like Elixir LiveView and Laravel LiveWire run just fine here using websockets.

First thing to check: is the client trying to connect with wss: (as opposed to ws:) as the protocol? If so, you are going to want to add “tls” to the handlers: Public Network Services · Fly Docs

Next – as silly as it sounds – verify that you are actually listening on that port.

4

@rubys - thanks for your response. This is my setup:

The fly.toml seems to be correct, as the flyctl services list reports:

PROTOCOL        PORTS           HANDLERS        FORCE HTTPS     PROCESS GROUP   REGIONS MACHINES 
TCP             80 => 6902      [HTTP]          False           app             waw     2       
TCP             443 => 6902     [TLS,HTTP]      False           app             waw     2       
TCP             88 => 6901      [HTTP]          False           app             waw     2

Now, there are two php processes running two services:

php ratchet/server.php > ws.log &
php -S 0.0.0.0:6902 index.php

The ratchet server is initialized as below:

$server = IoServer::factory(
    new HttpServer(
        new WsServer(
            new Chat()
        )
    ),
    6901,
    '0.0.0.0'
);

$server->run();

Now, when I ssh into the container’s console and run:

curl -i -N -H "Connection: Upgrade" -H "Upgrade: websocket" -H "Host: 0.0.0.0" -H "Origin: http://0.0.0.0" -H "Sec-We
bSocket-Key: x3JJHMbDL1EzLkh9GBhXDw==" http://0.0.0.0:6901

I’m getting correct response:

HTTP/1.1 426 Upgrade Required
Connection: Upgrade
Upgrade: websocket
Sec-WebSocket-Version: 13
X-Powered-By: Ratchet/0.4.4

…but when I do the same from my computer’s local terminal (of course, providing the correct application hostname and the external port 88, I’m getting either this:

curl: (56) Recv failure: Connection reset by peer

or this:

curl: (52) Empty reply from server

So - addressing your advices:

  1. ws/wss - I’m not event that deep into testing - just a simple cURL test to see if the connection gets upgraded at least,
  2. there apparently is the websocket on local port 6901 as I can connect from containers console.

EDIT#1: Interesting - I have changed the ports in my fly.toml and set the port of main http service to something different than 80 (81 to be exact) and it is not working. Unfortunately, using this port for the webservice does not work either. I have also used different region - didn’t help. Any idea why changing the default port from 80 to 81 makes the service unavailable? Changing back to 80 fixes the issue (but websocket is not reachable in any case). Btw, I’m on the Hobby Plan with shared IP - does this make any difference?

EDIT#2: I start to suspect the problem with SSL. I have found the information, that for the .dev domain the HTTPS is enforced. Now, as an experiment, I modified the part of my fly.toml concerning only the main www service:

[[services]]
  protocol = "tcp"
  internal_port = 8080
  auto_stop_machines = true
  auto_start_machines = true
  min_machines_running = 0

  [[services.ports]]
    port = 443
    handlers = ["tls", "http"]

  [[services.ports]]
    handlers = ["tls", "http"]
    port = 8080

This should expose the service on internal 8080 to two external ports: 443 and 8080. This works only for the default 443 but for the latter returns the SSL_ERROR_SYSCALL when testing with cURL. Why is that?

EDIT#3 just to mention that my app is dockerized and work fine in a local environment.

EDIT#4 OK, after many hours of fighting with this, I managed to run the websocket service and bind it to the default HTTPS 443 port. This seems to indicate, that this is the only external port, that’s available and there is no way to use any other port, unless I’m missing something in my configuration or this is the limitation of the Hobby plan…

Any further clues what to check or look for?

5

I’ll look into this more Sunday/Monday, but two quick comments:

  • I don’t believe you can associate one internal port with two external ports. The original question related to mapping two internal ports to two external ports, which should be fine.
  • I don’t believe that port 443 is a limitation of the Hobby plan
6

OK, please do. I played a bit more with it, but whatever I do no other ports than 443 are available from the outside. All services running on internal ports are accessible with curl from the inside of the container.

I’m leaving the container running if you would like to take a pick if you can.
Below is my final fly.toml:

app = "myfoobar"
primary_region = "waw"

[build]

[[services]]
  protocol = "tcp"
  internal_port = 8080
  auto_stop_machines = true
  auto_start_machines = true
  min_machines_running = 0

  [[services.ports]]
    port = 8080
    handlers = ["tls", "http"]

[[services]]
  protocol = "tcp"
  internal_port = 6901

  [[services.ports]]
    port = 443
    handlers = ["tls", "http"]

[[services]]
  protocol = "tcp"
  internal_port = 6902

  [[services.ports]]
    port = 89
    handlers = ["tls", "http"]
7

I was able to reproduce this overall port-limitation phenomenon—with ordinary HTTPS over shared IPv4:

curl -6 https://my-app-name.fly.dev:8080/

Works as expected. But…

curl -4 https://my-app-name.fly.dev:8080/

…fails with SSL_connect: SSL_ERROR_SYSCALL.

If I understand correctly, this is a longstanding limitation of Fly’s implementation of shared IPv4—rather than a restriction in the Hobby plan per se:

https://community.fly.io/t/outage-connection-reset-by-peer-when-using-non-default-ports/9521/2

shared IP address […], which only supports HTTP/S on 80/443.

https://community.fly.io/t/fly-app-with-multiple-external-ports-only-working-for-port-443/9417/2

This is probably due to our recent release of shared IPs. These only support tls + http over port 443.

8

Thank you @mayailurus - that would explain everything.

I have reviewed the posts you cited and indeed it would be good if the docs expressed that exception concerning that shared IP limitation. Most of the time, when looking for the reason and solution, I was suspecting the issue related to the websockets not the more generic single port limitation. Especially that there were other posts signaling similar problems but without the real solution.

Anyway - thank you for your help, analysis and confirming the real source of the problem. I will either choose the dedicated IP or use two containers for two services I need.

Cheers!
Tom

1 Like
9

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.