gRPC error "connection reset by peer" on example app and my own app

I have my own gRPC service running on Fly.io and I’ve also deployed Fly.io’s example gRPC service.

Whenever they’re called, both fail with:

Failed to dial target host "HOST:PORT": read tcp [2600:4041:5991:c100:2801:4c61:a1fd:d913]:50961->[2a09:8280:1::6:bbe7]:443: read: connection reset by peer

I’ve ssh’d into my service and confirmed that I can successfully call my service locally on the expected port. The call also shows in the logs on my Fly.io dashboard.

Here’s my .toml

app = "MY SERVICE"
kill_signal = "SIGINT"
kill_timeout = 5
processes = []

[[services]]
  internal_port = 3000
  protocol = "tcp"
  processes = ["app"]
  http_checks = []
  script_checks = []

  [services.concurrency]
    hard_limit = 250
    soft_limit = 200

  [[services.ports]]
    handlers = ["tls"]
    port = "443"

  [[services.tcp_checks]]
    interval = 10000
    timeout = 2000

  [services.ports.tls_options]
    alpn = ["h2"]

Here’s my Dockerfile:

FROM node:18

WORKDIR /usr/src/
COPY . .

RUN npm install
RUN npx prisma generate

EXPOSE 3000

CMD [ "npm", "start" ]

And my service starts with the following

  const server = new Server({
    "grpc.max_receive_message_length": -1,
    "grpc.max_send_message_length": -1,
  });

  server.addService(MyTfcService, new MyTfcServiceImpl());
  server.bindAsync(
    `0.0.0.0:3000`,
    ServerCredentials.createInsecure(),
    (err: Error | null) => {
      if (err) {
        throw err;
      }
      console.log(`🛰️  Listening on port ${port}`);
      server.start();
    }
  );

I’ve trawled a bunch of previous posts and still can’t see what’s going wrong… One guess I had is that my container is not being booted with port 3000 being mapped to some port that Fly.io’s edge proxy accesses, but I suppose that’s what the internal_port in the .toml is for.

Here are the other posts I looked at:

Server isn’t listening on all interfaces (both IPv4 + IPv6). May be this is how to do it?

server.bindAsync(
-   `0.0.0.0:3000`,
+   "3000",

Or:

server.bindAsync(
-   `0.0.0.0:3000`,
+   ":3000",

Or:

server.bindAsync(
-   `0.0.0.0:3000`,
+   "[::]:3000",

If you’re meaning to do gRPC over IPv4, you’d need to allocate a dedicated IPv4 address to your Fly app which costs $2/mo: Announcement: Shared Anycast IPv4 - #3 by kurt


Btw, is h2 TLS ALPN needed for gRPC?

Try over IPv4 and it will work, there is an issue where IPv6 TLS connections fail.

I don’t know if it’s related, but today I’ve started getting Errno::CONNRESET on my Rails app. It feels related.

My Rails app makes HTTP calls back to itself to download images from a 3rd party server (there’s a complex reason for this, but it’s not relevant). I think there’s something more core broken in Fly today (I have previous days where it worked fine)

9:02 ~ $ fly console ssh
/ # curl -k -D - https://myapp.fly.dev/rails/active_storage/representations/redirect/...
curl: (35) OpenSSL SSL_connect: Connection reset by peer in connection to myapp.fly.dev:443

If I do this from my local machine it works fine:

9:02 ~ $ curl -k https://myapp.fly.dev/rails/active_storage/representations/redirect/...
<!DOCTYPE html>
...

I tried directly using the IPv4 address (with -k to ignore the SSL certificate name error) and had exactly the same error.

Actually, I’ve got around this with DNS changes (as others reported it was only IPv6 affecting too). I had www at my domain pointing as a CNAME to Fly’s app subdomain, I changed that to be just an A record pointing to my app’s IP (therefore not involving fly’s IPv6 DNS entries) and all is working fine for my app now.

1 Like

Omg yes!!!

Adding an IPv4 IP worked.


@ignoramous’s suggestion didn’t work sadly, but that would have been great! When switching to:

server.bindAsync(
-   `0.0.0.0:3000`,
+   "[::]:3000",

I got error:

Failed to dial target host "my-host:3000": EOF

Would be interested to know more about why this is happening. @mike-ravkine is that an issue with Fly.io?


I can’t believe this is working now :face_holding_back_tears:. I’ve been trying for a couple of days to host a gRPC service somewhere and this was my last port of call before giving up and re-writting in GraphQL. It seems that Fly.io is the only place you can without a lot of the edge setup being left to you. Thank you so much!

2 Likes

Nice! (:


Consider reviving the gRPC app-guide (replacing the existing one rendered from a github readme): https://github.com/superfly/docs/blob/2bd275fe9f15d04655a9cd4b15465c4c1facd85f/app-guides/grpc-and-grpc-web-services.html.erb#L4 :wink: