TCP timeout between Flycast-connected NestJS microservices (Nestjs monorepo architecture)

[shopazhub]

Hey everyone,

I’ve deployed a NestJS monorepo microservice architecture on Fly.io — the setup includes an API Gateway (public app, HTTPS) and several private services (users, products, etc.) communicating over TCP.

Each private service uses Flycast for internal networking, and everything appears reachable:

• I can ping the private apps from the API Gateway container.
• DNS resolution works (<service>.internal resolves correctly).
• Logs confirm the gateway connects to the right internal addresses.

However, when I trigger routes (like /products/all) that should call the products service via TCP, I get:

2025-10-22T13:37:08.051 app[78432e7b401e48] jnb [info] [Nest] 642 - 10/22/2025, 1:37:08 PM LOG [API] App listening on port: 4000

2025-10-22T13:37:08.535 proxy[78432e7b401e48] jnb [info] machine became reachable in 4.184513031s

2025-10-22T13:37:11.761 app[78432e7b401e48] jnb [info] [Nest] 642 - 10/22/2025, 1:37:11 PM ERROR [ClientTCP] Error: read ECONNRESET

2025-10-22T13:37:38.572 app[78432e7b401e48] jnb [info] [Nest] 642 - 10/22/2025, 1:37:38 PM LOG [PRODUCT's controller] TimeoutError: Timeout has occurred

2025-10-22T13:37:38.578 app[78432e7b401e48] jnb [info] [Nest] 642 - 10/22/2025, 1:37:38 PM ERROR [ExceptionsHandler] Timeout has occurred

2025-10-22T13:37:38.579 app[78432e7b401e48] jnb [info] Error

2025-10-22T13:37:38.579 app[78432e7b401e48] jnb [info] at _super (/usr/src/app/node_modules/rxjs/dist/cjs/internal/util/createErrorClass.js:7:26)

2025-10-22T13:37:38.579 app[78432e7b401e48] jnb [info] at new TimeoutErrorImpl (/usr/src/app/node_modules/rxjs/dist/cjs/internal/operators/timeout.js:14:9)

2025-10-22T13:37:38.579 app[78432e7b401e48] jnb [info] at timeoutErrorFactory (/usr/src/app/node_modules/rxjs/dist/cjs/internal/operators/timeout.js:61:11)

2025-10-22T13:37:38.579 app[78432e7b401e48] jnb [info] at /usr/src/app/node_modules/rxjs/dist/cjs/internal/operators/timeout.js:34:43

It’s strange because the services seem reachable but not responding over TCP at runtime.

Here’s my setup summary:

• Monorepo using @nestjs/microservices with Transport.TCP
• Each service runs in its own Fly.io app
• host for TCP clients is set to the Flycast address (users.internal)
• Ports are open in fly.toml
• Works perfectly in local Docker Compose

Has anyone run into this behavior on Fly.io?
Is there something specific about Flycast or Fly’s internal networking that prevents stable TCP connections between apps? Any config tweaks, port mapping tricks, or deployment strategies that worked for you?

Would really appreciate some guidance or shared configs if you’ve solved something similar!

[mayailurus]

@shopazhub

Flycast address (users.internal)

That isn’t a Flycast address, though… (Those always end in .flycast.)

To use Flycast, you would need to explicitly allocate addresses. (And you also need [[services]] or [http_service] blocks, but it sounds like you do have those already.)

Hope this helps!

[shopazhub]

thank you, it was a mistake, i actually use the .flycast address, i removed all other public facing ip addresses and assigned a flycast address, when I ssh into the machine on the services and ping the flycast addresses i get successful responses even for TCP connections which the Nestjs uses for inter app communication

[lillian]

are your microservices listening on ipv6?

[mayailurus]

@shopazhub

Would really appreciate some guidance or shared configs if you’ve solved something similar!

By way of illustration, here’s a full example of a Flycast service that speaks plain TCP (i.e., not HTTP)…

app = "qua-melon"
primary_region = "ewr"
kill_signal = "SIGTERM"
swap_size_mb = 512

[[services]]
  internal_port = 8080
  protocol = "tcp"
  [[services.ports]]
    handlers = []  # i.e., bare TCP, no HTTP or SSL.
    port = 4321

Not being one of the forum’s Javascript experts, , I used Racket instead. But the following is just a listener on TCP port 8080, always responding “hello from $FLY_REGION”:

#lang racket/base

(require racket/tcp)

; the server's region, not the client's...
(define machine-region (getenv "FLY_REGION"))

(define (serve port msg)
  (define l (tcp-listen port))
  (eprintf "listening on ~v...\n" port)
  (let again ()
    (define-values (in out) (tcp-accept l))
    (write-string msg  out)
    (close-output-port out)
    (close-input-port   in)
    (again)))

(serve 8080 (format "hello from ~a.\n" machine-region))

And the Dockerfile corresponding to the above b.rkt…

FROM debian:bullseye-slim

RUN apt-get update  -y && \
    apt-get install -y --no-install-recommends racket procps iproute2

WORKDIR /app

COPY b.rkt b.rkt

CMD ["racket", "/app/b.rkt"]

Deploying, to both NYC and Sweden…

$ fly app create qua-melon
$ fly ips allocate-v6 --private
$ fly  deploy --ha=false --no-public-ips
$ fly m clone --region=arn

Verifying binding addresses and ports…

$ fly ssh console
# ss -tnlp 'sport = 8080'
State   Recv-Q  Send-Q  Local Address:Port  Peer Address:Port  Process                           
LISTEN  0       4       0.0.0.0:8080        0.0.0.0:*          users:(("b.rkt",pid=648,fd=7))    
LISTEN  0       4       [::]:8080           [::]:*             users:(("b.rkt",pid=648,fd=5))    

$ fly services list
Services
PROTOCOL  PORTS         HANDLERS  FORCE HTTPS  PROCESS GROUP  REGIONS  MACHINES 
TCP       4321 => 8080  []        False        app            ewr,arn  2       

(Note that it is listening on 0.0.0.0 and not just on IPv6, as @lillian was alluding to.)

To try it out, here’s a small helper config packet, Dockerfile.client, to save some repetitious typing:

FROM debian:bullseye-slim

RUN apt-get update  -y && \
    apt-get install -y --no-install-recommends socat procps iproute2

CMD ["sleep", "inf"]

And now…

$ fly console --region ewr --dockerfile Dockerfile.client
# socat STDIO TCP6:qua-melon.flycast:4321
hello from ewr.

But then, …

$ fly console --region cdg --dockerfile Dockerfile.client
# socat STDIO TCP6:qua-melon.flycast:4321
hello from arn.

Used the closest one!

(Indeed, that plus auto-start are the two main reasons for having Flycast in the first place, …)

[shopazhub]

they are sending using HOST:PORT, where HOST is the flycast address, so for instance my-app.flycast:4001but if you mean like binding, the services are binded to 0.0.0.0

[shopazhub]

thank you very much, I was able to use this approach in my Nestjs app to solve the problem