Outbound bandwidth outbreak with 2 apps

[chientrm]

I have 2 apps:

• Main app is a content serving
• An nginx proxy app acts as jwt verifier and assign fly-instance-id

All outbound traffic is go from the main app to the proxy app and finally reach browser.

Somehow the main app traffic is at least 10 times more than the organic traffic to the user.

Here is my nginx config:

load_module /usr/lib/nginx/modules/ngx_http_auth_jwt_module.so;

events {}

http {
  map $jwt_claim_instance_id $instance_id {
    ~^"(.+)"$ $1;
    default $jwt_claim_instance_id;
  }
  server {
    listen 8080;
    listen [::]:8080;

    server_name myproxy.fly.dev;

    auth_jwt_key '/etc/nginx/rsa-public.pem' file;
    auth_jwt      $cookie_user;
    auth_jwt_alg  RS512;

    location /proxy {
      proxy_set_header fly-force-instance-id $instance_id;
      proxy_pass http://mymainapp.fly.dev;
      proxy_buffering off;
    }
  }
}

At this point, I have tried many tweaks but doesn’t work. Please help me!

[greg]

Hi,

It’s a total guess but is it possible your users have discovered and are accessing your content app directly?

Only from the look of that nginx conf, you appear to be using http://mymainapp.fly.dev and that would be public, by default (assuming someone knows that URL e.g it’s appeared in Google or something). I’d recommend looking at the logs of your content app to see what is being requested. If you see lots of requests for e.g /path/to/video.mp4 (or whatever content you are serving) from some IP/user agent, that would suggest someone is accessing it.

If it’s that, it might be worth looking at making your content app private (without an assigned public IP) which would mean your nginx proxy is the only thing able to access it. Take a look at Fly’s internal domains:

Private Networking

That would presumably also mean the data stays within Fly’s network, app->app. I’m not sure what happens billing-wise when one app requests data from another via the name.fly.dev .

[chientrm]

Can I spin up instance using those private domain?
When I ping using those private domain, it hang up.

[greg]

An app can be made “private” by not assigning any public IP (or removing one already allocated - Fly may allocate a shared one by default … not sure).

Since every app automatically gets access to the private networking documented on that page e.g app-name.internal can only be resolved to an IPv6 by other apps within your organisation.

I’d recommend making a new app and proxy to experiment with (e.g edit its fly.toml) before making any change to your production apps. Since it may take some experiments with the hostname and port to get it working how you need.

This is assuming that public-access was the problem to begin with. If not … you’ll still get the same bandwidth usage.

[chientrm]

I still need to switch to private access.

That help me avoid extra outbound network happens when I proxy to http://mymainapp.fly.dev.

[chientrm]

I switched to private network using flycast. So that now my nginx app proxy to .flycast instead.

I also released all public ipv4 and ipv6.

I need a day or two to confirm if it works.