Trouble connecting to private app

1

I deployed a private app that is only accessible internally within the organization. When I try to connect to http://private-app.internal:8080, I receive the following error:

# This request is made from another Fly app within the organization
> Req.post!("http://private-app.internal:8080/messages/new")
** (Mint.TransportError) non-existing domain
    (req 0.3.0) lib/req.ex:641: Req.request!/1

The private-app is listening to IPv4 and IPv6 as per suggested in [3] & [4].

Can someone please shed some light on this problem?

References:

  1. Having issues connecting 2 golang servers over .internal network - Questions / Help - Fly.io
  2. Private application only accessible internally - Questions / Help - Fly.io
  3. Fly io Newbie: Making internal requests between apps - #5 by ignoramous - Build debugging - Fly.io
  4. Redis app refuses private connections - #5 by rugwiro - Fly.io
2

(not an elixir person)

Does Mint use IPv6 where available (changelog)? If not, you may have to enable it explicitly to use private network on Fly (6pn) which is IPv6-only (ref).

iirc, an empty [[services]] block in fly.toml for private-only apps is a must. Add that if it is missing.

3

Thank you for your suggestion.

I tried to enable IPv6 explicitly but it didn’t work. It returned the same error. Other HTTP clients (hackney, HTTPoison) also returned the same error.

> req = Req.new(url: "https://private-app.internal:8080/messages/new", connect_options: [transport_opts: [inet6: true]])
> Req.post!(req)
** (Mint.TransportError) non-existing domain
    (req 0.3.5) lib/req.ex:329: Req.post!/2

And I also made sure the services block fly.toml is empty and redeployed the app.

app = "private-app"
kill_signal = "SIGINT"
kill_timeout = 5
processes = []

[env]

[experimental]
  auto_rollback = true

[[services]]

References

  1. IPv6 Documentation · Issue #163 · sneako/finch · GitHub
  2. Connect to IPv6 addresses · Issue #554 · benoitc/hackney · GitHub
1 Like
4

Unrelated to the error you’re seeing but, make sure to connect over plaintext http (not https, unless you have self signed certs setup for the .internal domain) and that the server on the other side is also capable of serving plaintext http.

Just to be sure, you’re attempting to connect to private-app from another app within the same Fly org, correct?

I believe you’ve already checked,

  • flyctl dig _apps.internal TXT -a <app-name> (ex)
  • flyctl ips private -a <app-name> (ex)

…to ascertain if both the client app and the server (private-app) in your org have indeed been assigned 6pn IPs (if not, exec flyctl ips allocate-v6 --private -a <app-name>; docs)?

If so, can you try

  • flyctl ssh console -a private-app -C nslookup -type AAAA private-app.internal
  • flyctl console -a private-app -C nslookup -type AAAA global.private-app.internal

…and see what it prints (ex, docs)?

5

I finally found the culprit! Thanks to this thread and @ignoramous!

In my Dockerfile, I need to bind to :: instead of 0.0.0.0, like so:

FROM python:3.10-slim-bullseye

ENV PYTHONUNBUFFERED True
ENV APP_HOME /app
WORKDIR $APP_HOME
COPY . ./

RUN pip install --no-cache-dir -r requirements.txt

CMD ["uvicorn", "main:app", "--host", "::", "--port", "8080"]
6

For those who come across the same problem, here’s a short summary:

  1. Make sure your app is bind to :: instead of 0.0.0.0. In my case, I had to specify :: in Dockerfile for my python app like so:
FROM python:3.10-slim-bullseye

ENV PYTHONUNBUFFERED True
ENV APP_HOME /app
WORKDIR $APP_HOME
COPY . ./

RUN pip install --no-cache-dir -r requirements.txt

CMD ["uvicorn", "main:app", "--host", "::", "--port", "8080"]
  1. If you’re using http clients like Req or Finch, make sure you explicitly enable inet6 because Fly uses IPv6 for private apps [1].
> req = Req.new(url: "http://private-app.internal:8080", connect_options: [transport_opts: [inet6: true]])

[2][3]

  1. Make sure to connect over plaintext http [4].

And boom! Problem solved!

References

  1. 6PN addressing clarification - Fly.io
  2. Connect to IPv6 addresses · Issue #554 · benoitc/hackney · GitHub
  3. IPv6 Documentation · Issue #163 · sneako/finch · GitHub
  4. Trouble connecting to private app - #4 by ignoramous - elixir - Fly.io?
1 Like