Hi all, I have a question about machines. We are planning to offer a service that will require total isolation between customers, therefore Fly.io machines seem to be a good fit.
Is there a way to prevent a malicious user from establishing something like a Cloudflare tunnel or ngrok or similar and serve files from our machine?
What would be the best way to prevent this kind of abuse?
Thanks in advance!
Hi… The little-known “network policies” can be used to prevent all outgoing network traffic, public or private, and all private incoming things, if what you’re running for them is compute-only. (Or if you yourself mediate all communication with the outside world.)
And if you’re the one creating the Machine, then you control how it interacts with the Fly Proxy (if at all), which is the other way to get incoming traffic. (The network policies don’t prevent Machines from receiving packets that went through the Fly Proxy.)
There are some rough edges to the network policies, or at least there were last I checked, e.g., no way to distinguish .internal from the public Internet, but they would be a good starting point…
Aside: Also be aware of the nuances of Fly-Replay, if all these Machines are in the same Fly organization. They can bounce incoming packets from the Fly Proxy around to each other, across app boundaries, by default.
Aside2: The multi-container Machines might be (indirectly) useful, if you want to give each customer container a sidecar that handles communication. These have some rough edges, too, alas, but are a useful primitive…
Hi! Thanks for your reply. The main issue is that we must allow legitimate outbound traffic, so we cannot rely on an allowlist 
So I am not sure of how to use the network policies in this case.
If the machine is in the same app as one of yours, and the customer has legitimate console access to their machine, they won’t need a tunnel - they can just fetch from your machine using curl or wget over the private network. Would your machine have a listener? If so, it can always reject requests from the private network, or serve an error response.
It’s not so simple. Each tenant must be able to connect to various services over the internet, due to integrations. Also, perhaps I wasn’t clear. The problem is not them needing a tunnel to connect to our app or else. The problem I want to solve is avoiding abuse by malicious users. A malicious user might host some illegal content on their assigned machine, and instantiate a tunnel like ngrok or similar to serve that content over the internet. Part of the features of our tenant apps is agentic, so an LLM needs to be able to connect to third party services.
Ah, okay, that makes sense.
The only technical approach I can think of is setting up a proxy where every IP or domain has to be whitelisted, though you’d have to decide if this was too onerous to look after. Or you could set up an open proxy, but close the connection if a massive amount of traffic flowed through.
You really have a social/trust problem; you can probably resolve it by taking a zero-payment from a card, so that you have details in the event of systems abuse. That would probably resolve 99% of misuse attempts.
The other thing you could do is not solve it, but it depends on your jurisdiction. Some countries hold it to be true (either de facto or de jure) that a service operator is not responsible for subscriber misuse; for example, no-one complains to the telephone company if a miscreant places repeated nuisance calls. Misuse of public Wi-Fi falls often under the same category.
it is very hard to both allow arbitrary traffic (whatever the agent needs to do) and deny arbitrary traffic (someone setting up a reverse tunnel to cloudflare/ngrok/etc).
we, as well as the vast majority of hosting providers, mainly rely on third party reports, with some amount of blocking known and privacy-preserving background scans for things like crypto miners. you can, for instance, block the most well-known reverse proxy tunnel IPs, or detect when your users run a command named cloudflared.
in general this is a hard problem for even nation-states to solve. china/russia, the most well-known countries with restrictive internet access, constantly have a back and forth between blocking and people finding ways around it…
Thanks for the explanation! What happens in case one of my users does abuse the service somehow? Will Fly contact me and give me the time to take some action?
yeah, if we detect or are notified of abuse in a customer machine, we would contact you requesting for the malicious behaviour to be removed. of course I can’t detail our abuse handling practices on a public forum, but we tend to be reasonable with timeframes and such.
Awesome, thanks! Also I am planning on using Cloudflare CSAM Scanning Tool at least to detect child abuse content, since that has higher priority compared to other types of illegal content.
About cloudflare tunnels etc, I was thinking of combining ratelimits at container image level with iptables/nftables with a lightweight service running in the Fly machine that scans outgoing connections and notifies us if it detects use of regular cloudflared, ngrok or similar so we can check what the customer is doing with them. I am going to research also if there is a way to detect with DNS or something else if there are hostnames pointing to the machine with such tunnels with DNS or something else.
So I am going to research more before launching, but it’s comforting to know that Fly will give us some time to rectify any abuse. Thanks again!