Error: connect EHOSTUNREACH when sending requests to my fly.io app

1

Hi. I have a NodeJS app hosted on fly.io. The app consists of two sub-apps. One is a websocket server listening on port 34197 and the other is an expressjs app listening on port 8080. Whenever I send a request to my express app it comes back with Internal Server Error (500) with the following error:

Error: connect EHOSTUNREACH 2606:4700::6810:605e:443
    at TCPConnectWrap.afterConnect [as oncomplete] (node:net:1229:16)

Am I doing something wrong here? I have the same setup on a VPS and it’s working flawlessly. Here’s my fly.toml:

# fly.toml file generated for orca-voip on 2022-06-18T08:45:53+02:00

app = "orca-voip"
kill_signal = "SIGINT"
kill_timeout = 5
processes = []

[build]
  builder = "heroku/buildpacks:20"

[env]
  PORT = "34197"
  MODE = "PRODUCTION"
  API_PORT = "8080"

[experimental]
  allowed_public_ports = []
  auto_rollback = true

[[services]]
  http_checks = []
  internal_port = 8080
  processes = ["app"]
  protocol = "tcp"
  script_checks = []
  [services.concurrency]
    hard_limit = 25
    soft_limit = 20
    type = "connections"
  [[services.ports]]
    force_https = true
    handlers = ["http"]
    port = 80
  [[services.ports]]
    handlers = ["tls", "http"]
    port = 443

  [[services.ports]]
    handlers = ["tls", "http"]
    port = 8080

  [[services.tcp_checks]]
    grace_period = "1s"
    interval = "15s"
    restart_limit = 0
    timeout = "2s"

[[services]]
  http_checks = []
  internal_port = 34197
  processes = ["app"]
  protocol = "tcp"
  script_checks = []

  [services.concurrency]
    hard_limit = 25
    soft_limit = 20
    type = "connections"

  [[services.ports]]
    handlers = ["tls", "http"]
    port = 34197

The logs aren’t much help either:

2022-06-18T08:15:11Z app[374b0eb8] cdg [info]2022/06/18 08:15:11 listening on [fdaa:0:7002:a7b:ae02:374b:eb8:2]:22 (DNS: [fdaa::3]:53)
2022-06-18T08:15:12Z app[374b0eb8] cdg [info]> orca-backend@0.0.1 start
2022-06-18T08:15:12Z app[374b0eb8] cdg [info]> node ./dist/index.js
2022-06-18T08:15:15Z app[374b0eb8] cdg [info]info 2022-06-18T08:15:15.848Z Mongo connected
2022-06-18T08:15:15Z app[374b0eb8] cdg [info]info 2022-06-18T08:15:15.851Z Initializing services
2022-06-18T08:15:15Z app[374b0eb8] cdg [info]info 2022-06-18T08:15:15.852Z All services initialized
2022-06-18T08:15:15Z app[374b0eb8] cdg [info]info 2022-06-18T08:15:15.856Z API listening on port 8080
2022-06-18T08:15:15Z app[374b0eb8] cdg [info]info 2022-06-18T08:15:15.856Z Socket server listening on port 34197
2022-06-18T08:15:29Z app[374b0eb8] cdg [info]Error: connect EHOSTUNREACH 2606:4700::6810:605e:443
2022-06-18T08:15:29Z app[374b0eb8] cdg [info]    at TCPConnectWrap.afterConnect [as oncomplete] (node:net:1229:16)
2022-06-18T08:16:14Z app[374b0eb8] cdg [info]Error: connect EHOSTUNREACH 2606:4700::6810:605e:443
2022-06-18T08:16:14Z app[374b0eb8] cdg [info]    at TCPConnectWrap.afterConnect [as oncomplete] (node:net:1229:16)
2022-06-18T08:16:46Z app[374b0eb8] cdg [info]Error: connect EHOSTUNREACH 2606:4700::6810:605e:443
2022-06-18T08:16:46Z app[374b0eb8] cdg [info]    at TCPConnectWrap.afterConnect [as oncomplete] (node:net:1229:16)

Note that the same express server is serving static assets like a charm. Any help would be appreciated.

2

FWIW, I’m using a remote MongoDB Cloud (Atlas) instance to host my data.

3

I guess the question is what host is it trying to reach? What that “2606:4700::6810:605e” is. Clearly it can’t reach [something] and hence that error. That will help debug why. Whether it is a Fly-hosted domain, or example, or your MongoDB Cloud, which you mention. To find out, I’d recommend adding more logging in your Express app (rather than just using Fly’s output). That will show which hostname it wants to connect to. Most apps will have some kind of logging, so it may be as simple as increasing your log level to output more (info, debug, verbose etc). Else by using e.g Debugging Express

For example if it’s calling your app, maybe it’s using a relative path or localhost or something, and so that is not resolving. Or if it’s your mongodb (you may not want to reveal the hostname, that’s fine) it would point to an issue resolving external dns, its ssl, or something along those lines.

1 Like
4

I’m surprised debugging through express didn’t come to mind. Thanks for the suggestion, I’ll try it and post my findings.

1 Like
5

So I did a bit more digging and added some more logging it seems there’s a problem with my JWT verification middleware. Specifically, expressjwt with jwks.expressJwtSecret. It gives me this error right now:

2022-06-20T21:01:02.035 app[fe367fd5] cdg [info] Error: socket hang up
2022-06-20T21:01:02.035 app[fe367fd5] cdg [info] at connResetException (node:internal/errors:693:14)
2022-06-20T21:01:02.035 app[fe367fd5] cdg [info] at TLSSocket.socketCloseListener (node:_http_client:435:25)
2022-06-20T21:01:02.035 app[fe367fd5] cdg [info] at TLSSocket.emit (node:events:539:35)
2022-06-20T21:01:02.035 app[fe367fd5] cdg [info] at node:net:740:14
2022-06-20T21:01:02.035 app[fe367fd5] cdg [info] at TCP.done (node:_tls_wrap:581:7)

I verified that all environment variables are set as expected and used as expected in the code. My Auth0 middleware is the one causing or throwing that error and I have absolutely no idea why.

1 Like
6

This is a cloudflare ip: 2606:4700::6810:605e:443.

I can’t seem to connect to that IP on any port, which makes me think this is outside our infrastructure. Do you know what hostname it’s trying to connect to?

7

It’s my Auth0 Issuer URL which is https://orca-voip.eu.auth0.com.

8

Oh so this is interesting.

Error: connect EHOSTUNREACH 2606:4700::6810:605e:443

That looks like the library isn’t handling IPv6 properly. The actual host is 2606:4700::6810:605e, the port is 443 and it seems to have combined them into a corrupt IPv6 address.

9

The way it works is that the middleware sends tokens arriving with requests to the issuer to verify the token and decode its payload and then based on that decide whether said request should be allowed to continue. Problem is, it’s not completing that process. The request to the issuer is the one that fails.

10

Okay… that’s weird. Is there anything I can do on fly’s side to fix that?

11

Turns out this was on us. This was on a new server with a misconfigured IPv6 stack.

Should now be good.

We’re adding extra checks to make sure this doesn’t happen again.

2 Likes
12

It’s still happening on my end. Is there any steps I need to take?

13

My bad, it does look like it’s not working within your instance. It now works on the hosts which is what we were testing with.

Investigating more now.

14

Is there an estimate for when its ready? I’d appreciate it if you reply to this thread once it’s ready.

15

No estimate, it shouldn’t be too long. We’ll let you know once it’s fixed and we’ve tested it more thoroughly.

16

Sounds good. Thanks for the heads up and the quick responses!

17

It’s now fixed. A different misconfiguration on the specific host your app landed on. 2 different issues, unlucky!

2 Likes
18

How unlucky indeed :joy:. I can confirm it’s working on my end like a charm now. Thanks!

1 Like
19

@jerome Running into the same issue again. I can open a separate topic, but the issue seems to match mostly with the one described here.

Some differences include that the node application is making a request to another fly application that is running and is not experiencing any issues. The same request works in local machines and instances other than the fly node application.

request to https://src-dev-cms.fly.dev/api/... failed, reason: connect ENETUNREACH 2a09:8280:1::6:816c:443
1 Like
20

Having the same issue, started ~an hour ago. Tried allocating a new IPv6 to no avail. Running fetch from a node REPL to another Fly application (https://{appname}.fly.dev) results in:

Screenshot 2022-11-02 at 11.55.35
Screenshot 2022-11-02 at 11.55.351186×424 111 KB

1 Like