Yo! Had this issue for a long while and we’re wanting to make use of the Fly.io internal networking more now so I want to solve this once and for all.
Basically, we have two services: Jaeger and Swagger running in Fly. Neither have any exposed ports so they’re fully internal. Jaeger can be accessed via our
.internal domain but the Swagger/OpenAPI service cannot.
I’ve set up my Wireguard according to the Fly docs and I can happily access any of our services using the
<service-name>.internal:<port> URL pattern. Except this one Swagger-UI service.
The only relevant difference I can see between these services is the port they run on. The internal ports are all that matter with internal Wireguard access like this (as is my understanding) and the Swagger container image exports 8080 as its default and Jaeger uses 16686.
What I’ve tried so far
I’ve tried re-deploying the service, Jaeger always works and OpenAPI always fails.
I’ve exposed Swagger-UI to the public via the following
[[services]] config and it works fine, so I know that
8080 is the correct internal port on the container.
[[services]] http_checks =  internal_port = 8080 processes = ["app"] protocol = "tcp" script_checks =  [services.concurrency] hard_limit = 75 soft_limit = 50 type = "requests" [[services.ports]] force_https = true handlers = ["http"] port = 80 [[services.ports]] handlers = ["tls", "http"] port = 443 [[services.tcp_checks]] grace_period = "1s" interval = "15s" restart_limit = 0 timeout = "2s"
I’ve tried nslookup’ing both
.internal domains, they both resolve to an address but only one address works:
nslookup + HTTP GET Jaeger
❯ nslookup myteam-jaeger.internal Server: UnKnown Address: fdaa:0:b4b8::3 Name: myteam-jaeger.internal Address: fdaa:0:b4b8:a7b:28df:5:5542:2
❯ http get http://[fdaa:0:b4b8:a7b:28df:5:5542:2]:16686 HTTP/1.1 200 OK
nslookup + HTTP GET OpenAPI
❯ nslookup myteam-openapi.internal Server: UnKnown Address: fdaa:0:b4b8::3 Name: myteam-openapi.internal Address: fdaa:0:b4b8:a7b:8e:309a:1e67:2
❯ http get http://[fdaa:0:b4b8:a7b:8e:309a:1e67:2] No connection could be made because the target machine actively refused it
(Same result with
Broken Swagger-UI service:
app = "myteam-openapi" kill_signal = "SIGINT" kill_timeout = 5 processes =  [build] image = "swaggerapi/swagger-ui" [env] SWAGGER_JSON_URL = "https://myteam-service.fly.dev/openapi.json" WITH_CREDENTIALS = "true" [experimental] allowed_public_ports =  auto_rollback = true
Working Jaeger service:
app = "myteam-jaeger" kill_signal = "SIGINT" kill_timeout = 5 processes =  [build] image = "jaegertracing/all-in-one:1.42" [env] BADGER_DIRECTORY_KEY = "/badger/key" BADGER_DIRECTORY_VALUE = "/badger/data" BADGER_EPHEMERAL = "false" SPAN_STORAGE_TYPE = "badger" [experimental] allowed_public_ports =  auto_rollback = true [mounts] destination = "/badger" source = "jaeger_data"
So far all I can conclude is that our service name is cursed and I must choose another one (which I have not tried yet… maybe a last resort)