Controlling access to a Fly app

nikolaswise · January 18, 2023, 11:10pm

I have a fly app running a server image. This is great and fantastic, but this server image implements no auth at all – everything is public all the time. Not ideal!

I need to restrict access to this server. My initial plan was to set an NGINX proxy up on Fly that proxies through the private network .internal address, and put basic auth on that.

However, no matter what I try (all the examples from fly, all the examples here) the proxy returns a 502.

What are my best approaches here? Does anyone have nginx running a proxy like this? How does yours work?

ignoramous · January 19, 2023, 12:02am

What do you intend to run behind Ngnix? Node? If so, you can choose to handle http basic auth in Node itself.

As for 502s, can you check if your reverse proxy / server is handling incoming connections as expected (example: over plaintext http, if you’re using Fly’s http-handler)? Can you share your app’s fly.toml?

I’d say basic auth is the simplest way to accomplish what you’re trying to do.

If you’re adventurous, you can bind your Fly VMs in a tailnet, or expose them over tailscale funnels…? I haven’t done so myself but it doesn’t sound too complicated. Another alternative would be to use tunnels like ngrok, or front your Fly VMs with Cloudflare Access serving from behind the Cloudflare proxy.

And, since you ask, what we do is, have the user send the secret (as part of the URL: https://domain.tld/serve/this/path/heres-my-secret) and match it against a pre-generated, hard-coded hmac valid against a preset msg for that secret (ref). This is an overkill, so you’d not want to do this.

nikolaswise · January 19, 2023, 1:16am

I’m running an off-the-shelf Rust server called Oxigraph - I don’t to mess around in the internals at all since its not my area.

The config on that app looks like this:

app = "oxigraph"
kill_signal = "SIGINT"
kill_timeout = 6
processes = []

[build]
  image = "oxigraph/oxigraph"

[env]

[mounts]
  source="oxigraph_data"
  destination="/data"

[experimental]
  allowed_public_ports = []
  auto_rollback = true

[[services]]
  http_checks = []
  internal_port = 7878
  processes = ["app"]
  protocol = "tcp"
  script_checks = []
  [services.concurrency]
    hard_limit = 25
    soft_limit = 20
    type = "connections"

  [[services.ports]]
    force_https = true
    handlers = ["http"]
    port = 80

  [[services.ports]]
    handlers = ["tls", "http"]
    port = 443

  [[services.tcp_checks]]
    grace_period = "1s"
    interval = "15s"
    restart_limit = 0
    timeout = "2s"

Right now my nginx.conf looks like:

upstream oxigraph {
  server oxigraph.internal:7878;
}

server {
  listen 8080;
  listen [::]:8080;

  server_name *.fly.dev;

  location / {
    proxy_ssl_name $host;
    proxy_ssl_server_name on;
    proxy_pass https://oxigraph;
  }
}

I dont really understand why this would work or not work.

pan · January 19, 2023, 3:35am

I think nginx is the correct approach here. You could also look into an API gateway like Kong.

What address is oxigraph bound to? I have had to use the IPv6 loopback address for Rust services in the past: [::]:7878

If you want to restrict access with nginx you should remove the external port mapping ([[services.ports]]) in your fly.toml since your service is currently exposed.to the internet on ports 80 and 443. I have also had issues in the past using .internal addresses while having external ports mapped.

nikolaswise · January 19, 2023, 3:41am

I’m not totally sure what you mean by address and using the [::]::7878 service; do you mean from my Oxigraph apps perspective or NGINX’s perspective?

Im removing the public ports from the oxigraph app now and trying the internal address again.

Whats weird is I can get nginx to proxy other sites of mine over regular http, but not this oxigraph at all, even over it’s http ports.

brian · January 19, 2023, 3:49am

Not using Nginx, but I’ve had success with combining Cloudflare Access and Caddy server. Full stack looks like this.

Cloudflare → Fly Proxy → Caddy app on Fly → Python app on Fly

More details:

Cloudflare is configured for authenticated origin pulls. I also use Cloudflare Access to manage authentication. It’s a pretty neat one-click solution.
The Caddy app is configured as a reverse proxy (listen to incoming connections on 8080, verify the connection is coming from CF, forward the request to 8080 on Python app via the app-name.internal hostname.
The Python app is not exposed to the public Internet.

With this setup, I get the following:

Email-based auth via CF access.
Caddy app can’t be accessed via IP or fly.dev domain thanks to the authenticated origin pull setup.
Python app only talks to Caddy via internal network.

brian · January 19, 2023, 3:50am

In case you’re interested, my Caddyfile looks something like this:

:8080 {
  log {
    output stdout
    format console
  }
  tls /ssl/certs/cert.pem /ssl/certs/key.pem {
    client_auth {
      mode require_and_verify
      trusted_ca_cert_file /ssl/certs/cf_origin_pull_ca.pem
    }
  }
  reverse_proxy fly-app.internal:8080
}

nikolaswise · January 19, 2023, 3:51am

Nooope – took my Oxigraph app off the public internet by removing the port services and deallocating the IP addresses. Now my nginx config looks like:

server {
  listen 8080;
  listen [::]:8080;

  server_name oxigraph;

  location /health {
    add_header Content-Type text/plain;
    return 200 'healthy';
  }

  location /sparql {
    proxy_ssl_name $host;
    proxy_ssl_server_name on;
    proxy_pass https://oxigraph.internal[::]:7878;
  }
}

Health check is okay, but /sparql is still 502.

nikolaswise · January 19, 2023, 5:23am

Update: I’ve tried every variation I can think of and still no dice.

nikolaswise · January 19, 2023, 4:44pm

This morning I simplified things; I cant get a reverse proxy to a node app on the the internal network, or over public http.

This works:

location / {
    proxy_pass https://nikolas.ws;
  }

This doesnt

location / {
    proxy_pass https://sparql-server.fly.dev;
  }

Why would that be the case? https://sparql-server.fly.dev resolves fine, but it 502’s on the proxy pass.

charsleysa · January 19, 2023, 8:48pm

Hi @nikolaswise

Can you please post your fly.toml for both your nginx app and your node app, and the nginx config?

nikolaswise · January 20, 2023, 4:47am

Sure – I’ve verified that my Rust server is listening on IPv6, but still: 502. Heres my nginx app:

fly.toml

# fly.toml file generated for stucco-proxy on 2023-01-19T15:10:54-08:00

app = "stucco-proxy"
kill_signal = "SIGINT"
kill_timeout = 5
processes = []

[env]

[experimental]
  auto_rollback = true

[[services]]
  http_checks = []
  internal_port = 8080
  processes = ["app"]
  protocol = "tcp"
  script_checks = []
  [services.concurrency]
    hard_limit = 25
    soft_limit = 20
    type = "connections"

  [[services.ports]]
    force_https = true
    handlers = ["http"]
    port = 80

  [[services.ports]]
    handlers = ["tls", "http"]
    port = 443

  [[services.tcp_checks]]
    grace_period = "1s"
    interval = "15s"
    restart_limit = 0
    timeout = "2s"

nginx.conf

server {
  listen 8080;
  listen [::]:8080;

  server_name _;
  rewrite ^/(.*) /$1 break;

  proxy_ignore_client_abort on;
  proxy_set_header  X-Real-IP  $remote_addr;
  proxy_set_header  X-Forwarded-For $proxy_add_x_forwarded_for;
  proxy_set_header  Host $http_host;
  proxy_set_header Access-Control-Allow-Origin "*";

  location / {
    proxy_pass https://oxigraph.internal:7878;
    proxy_pass_request_headers on;
  }

  location ~ ^(/|/query)$ {
      proxy_pass http://oxigraph.internal:7878;
      proxy_pass_request_headers on;
  }

  location ~ ^(/update|/store)$ {
      proxy_pass http://oxigraph.internal:7878;
      proxy_pass_request_headers on;
  }
}

And my Rust server:
fly,toml

# fly.toml file generated for oxigraph on 2022-11-30T20:58:47-08:00

app = "oxigraph"
kill_signal = "SIGINT"
kill_timeout = 6
processes = []

[env]

[mounts]
  source="oxigraph_data"
  destination="/data"

[experimental]
  allowed_public_ports = []
  auto_rollback = true

[[services]]
  http_checks = []
  internal_port = 7878
  processes = ["app"]
  protocol = "tcp"
  script_checks = []
  [services.concurrency]
    hard_limit = 25
    soft_limit = 20
    type = "connections"

  [[services.tcp_checks]]
    grace_period = "1s"
    interval = "15s"
    restart_limit = 0
    timeout = "2s"

Rust app logs:

 2023-01-20T04:45:55.631 app[cd5c53fd] sea [info] Preparing to run: `/usr/local/bin/oxigraph_server --location /data serve --bind [::1]:7878` as root

2023-01-20T04:45:55.657 app[cd5c53fd] sea [info] 2023/01/20 04:45:55 listening on [fdaa:0:f0fa:a7b:2d30:3:35bf:2]:22 (DNS: [fdaa::3]:53)

2023-01-20T04:45:55.859 app[cd5c53fd] sea [info] Listening for requests at http://[::1]:7878

Nginx Logs:

 2023-01-20T04:46:15.574 app[c8b7e969] sea [info] 2023/01/20 04:46:15 [error] 548#548: *23 connect() failed (111: Connection refused) while connecting to upstream, client: 172.16.7.58, server: _, request: "GET / HTTP/1.1", upstream: "http://[fdaa:0:f0fa:a7b:2d30:3:35bf:2]:7878/", host: "stucco-proxy.fly.dev"

nikolaswise · January 20, 2023, 4:58am

Pointed by proxy to ipv6 port 22 instead of 7878 since thats what apparently is being listened too, and now my nginx 502 is

 2023-01-20T04:57:22.118 app[f55a4758] sea [info] 2023/01/20 04:57:22 [error] 549#549: *6 SSL_do_handshake() failed (SSL: error:1408F10B:SSL routines:ssl3_get_record:wrong version number) while SSL handshaking to upstream, client: 172.16.7.58, server: _, request: "GET /update HTTP/1.1", upstream: "https://[fdaa:0:f0fa:a7b:2d30:3:35bf:2]:22/update", host: "stucco-proxy.fly.dev"

nikolaswise · January 20, 2023, 5:18am

What … is happening
oxigraph

 2023-01-20T05:17:16.950 app[77e18d62] sea [info] 2023/01/20 05:17:16 unexpected error: ssh: overflow reading version string

nginx

 2023-01-20T05:17:16.946 app[7af4de80] sea [info] 2023/01/20 05:17:16 [error] 548#548: *7 upstream sent no valid HTTP/1.0 header while reading response header from upstream, client: 172.16.7.10, server: _, request: "GET / HTTP/1.1", upstream: "http://[fdaa:0:f0fa:a7b:2d30:3:35bf:2]:22/", host: "stucco-proxy.fly.dev"

This feels like progress but I just cannot understand

catflydotio · January 20, 2023, 11:48pm

Hi @nikolaswise Sorry I haven’t got the key to getting the reverse proxy working, but I believe port 22 is a wrong turn. A program called Hallpass listens on port 22 of each Fly.io app VM, to allow you to use fly ssh.

charsleysa · January 21, 2023, 12:28am

Hi @nikolaswise

If your oxigraph app is internal only then you can remove the entire services section from the fly.toml for the oxigraph app.

For your proxy, you need to make sure that you always use http for the oxigraph urls as internally it will never use https and https is not supported (unless oxigraph itself handles https).

Make sure to keep using the 7878 port because port 22 is for ssh and is the wrong port to use for your purpose.

One thing I noticed is that you’re potentially binding to the wrong address. I see you have used [::1]:7878, try using [::]:7878 instead.

nikolaswise · January 21, 2023, 3:01am

Oh my god that worked. A hero.

FWIW:

I changed the port from 7878 app default on oxigraph to 8080
Removed the port handlers in the oxigraph fly.toml

Topic		Replies	Views
Documentation on limiting API traffic to internal-only? Questions / Help docs	10	1190	April 13, 2023
Secure .internal private networking endpoints Questions / Help	6	941	June 23, 2023
Connecting your Fly Apps to your Tailscale tailnet Show & Tell	8	2741	February 25, 2024
Route certain traffic via wireguard back into the office to use office IP to originate to backend services	2	222	November 5, 2023
Nextjs app, should I use nginx in front of it JavaScript nodejs	3	43	October 1, 2024

Controlling access to a Fly app

Related topics