Private networking internal domain not working for one service

Questions / Help
1

Yo! Had this issue for a long while and we’re wanting to make use of the Fly.io internal networking more now so I want to solve this once and for all.

Basically, we have two services: Jaeger and Swagger running in Fly. Neither have any exposed ports so they’re fully internal. Jaeger can be accessed via our .internal domain but the Swagger/OpenAPI service cannot.

I’ve set up my Wireguard according to the Fly docs and I can happily access any of our services using the <service-name>.internal:<port> URL pattern. Except this one Swagger-UI service.

The only relevant difference I can see between these services is the port they run on. The internal ports are all that matter with internal Wireguard access like this (as is my understanding) and the Swagger container image exports 8080 as its default and Jaeger uses 16686.

What I’ve tried so far

I’ve tried re-deploying the service, Jaeger always works and OpenAPI always fails.

I’ve exposed Swagger-UI to the public via the following [[services]] config and it works fine, so I know that 8080 is the correct internal port on the container.

[[services]]
  http_checks   = []
  internal_port = 8080
  processes     = ["app"]
  protocol      = "tcp"
  script_checks = []

  [services.concurrency]
    hard_limit = 75
    soft_limit = 50
    type       = "requests"

  [[services.ports]]
    force_https = true
    handlers    = ["http"]
    port        = 80

  [[services.ports]]
    handlers = ["tls", "http"]
    port     = 443

  [[services.tcp_checks]]
    grace_period  = "1s"
    interval      = "15s"
    restart_limit = 0
    timeout       = "2s"

I’ve tried nslookup’ing both .internal domains, they both resolve to an address but only one address works:

nslookup + HTTP GET Jaeger :white_check_mark: 

❯ nslookup myteam-jaeger.internal
Server:  UnKnown
Address:  fdaa:0:b4b8::3

Name:    myteam-jaeger.internal
Address:  fdaa:0:b4b8:a7b:28df:5:5542:2

❯ http get http://[fdaa:0:b4b8:a7b:28df:5:5542:2]:16686
HTTP/1.1 200 OK

nslookup + HTTP GET OpenAPI :no_entry: 

❯ nslookup myteam-openapi.internal
Server:  UnKnown
Address:  fdaa:0:b4b8::3

Name:    myteam-openapi.internal
Address:  fdaa:0:b4b8:a7b:8e:309a:1e67:2

❯ http get http://[fdaa:0:b4b8:a7b:8e:309a:1e67:2]
No connection could be made because the target machine actively refused it

(Same result with fly dig)

Config files

Broken Swagger-UI service:

app          = "myteam-openapi"
kill_signal  = "SIGINT"
kill_timeout = 5
processes    = []

[build]
  image = "swaggerapi/swagger-ui"

[env]
  SWAGGER_JSON_URL = "https://myteam-service.fly.dev/openapi.json"
  WITH_CREDENTIALS = "true"

[experimental]
  allowed_public_ports = []
  auto_rollback        = true

Working Jaeger service:

app          = "myteam-jaeger"
kill_signal  = "SIGINT"
kill_timeout = 5
processes    = []

[build]
  image = "jaegertracing/all-in-one:1.42"

[env]
  BADGER_DIRECTORY_KEY   = "/badger/key"
  BADGER_DIRECTORY_VALUE = "/badger/data"
  BADGER_EPHEMERAL       = "false"
  SPAN_STORAGE_TYPE      = "badger"

[experimental]
  allowed_public_ports = []
  auto_rollback        = true

[mounts]
  destination = "/badger"
  source      = "jaeger_data"

So far all I can conclude is that our service name is cursed and I must choose another one (which I have not tried yet… maybe a last resort)

1 Like
2

I think i’ve might have run into a similar issuer earlier this month.

  • Can you curl the service running locally? You can fly ssh into the VM and hit the service via its local port. You could try curl http://0.0.0.0:<port> and curl -6 [::]:<port> to make sure the service is available on both ipv4 and ipv6 interfaces.
  • Make sure to bind your service(s) to both ipv4 an ipv6, generally that’s [::] but could highly depend on the underlying implementation.

Hope that helps tracking down your issue.

3

Not sure if it’s a typo:

  • http get http://[fdaa:0:b4b8:a7b:28df:5:5542:2]:16686
  • http get http://[fdaa:0:b4b8:a7b:8e:309a:1e67:2]

Why is the port omitted from the myteam-openapi.internal request? Does http://[fdaa:0:b4b8:a7b:8e:309a:1e67:2]:8080 work?

4

Yeah typo on copying to the post, the 8080 port doesn’t work.

5

ipv4 appears to be working fine:

❯ fly ssh console -a myteam-openapi
Connecting to fdaa:0:b4b8:a7b:8e:309a:1e67:2... complete
/ # curl 0.0.0.0:8080
<!-- HTML for static distribution bundle build -->
<!DOCTYPE html>
<html lang="en">
  <head>
    <meta charset="UTF-8">
    <title>Swagger UI</title>

but ipv6 doesn’t:

/ # curl -6 [::]:8080
curl: (7) Failed to connect to :: port 8080 after 0 ms: Couldn't connect to server

Not sure why that would be, is there something missing in my config? do I need ipv6?

6

Yes you’ll need ipv6 because the connection is being established over ipv6.

Your open api app needs to bind listen on [::]:8080, it’s likely listening on 0.0.0.0 only.

1 Like
7

Ugh, so I have to fork it and mess around with the nginx config…

Thanks for the help! I completely forgot about that little detail with the binding interface in containers!

8

We’re actively working on a workaround for that and making things “magically” work. This might come at the cost of more configuration in fly.toml for private networking (because Fly is oblivious to what you’re doing on your private network, it passes through packets and connections).

9

That’s great to hear! Even if this was some kind of tooling to help diagnose issues like this, it really was shots in the dark to figure it out.

I think a little more config would be fine. I do like the fact that the majority of things do just work and the internal network with wireguard is magical!

And with regards to the underlying service, I’ve opened a PR on the project: fix(docker): add missing ipv6 all-interfaces binding by Southclaws · Pull Request #8438 · swagger-api/swagger-ui · GitHub