Hiding metrics from the public internet with Cloudflare

Hey everyone! I noticed that there was a handful of people asking a question related to an issue I had, but there were no solid answers. I managed to piece together a solution and thought I’d post it to see if anyone saw a glaring issue with it and more importantly, hope that it helps someone else :slight_smile:

The issue: when using cloudflare proxy, the my-app.fly.dev host is still accessible. The preferred way to solve this is to use cloudflared to create a tunnel to your private application, either running it as a sidecar process in your container or as a private service alongside your app. I was having some trouble getting that set up and had other conflicting priorities, so I tried an alternate solution, injecting a secret header with a secret value in requests proxied by cloudflare, using their rewrite rules. My origin server checks for this header in all HTTP requests and returns a forbidden response if not present. I also force https, so that secret header is not exposed. These strategies are described in more detail in these cloudflare docs. Now if you go to the CF domain, you can get our app, protected by CF. If you go to the fly.dev, you will be denied outright.

All good, right? Well, not exactly. Now requests coming from within the fly private network are also rejected because of the secret header authorization. That includes health checks and metrics scraping, so my custom prometheus metrics are no longer flowing :crying_cat_face: Searching through the forums here, I noticed that the fly.io HTTP handler is going to add some headers to any requests coming from outside our private network. That means if the header is not present, then it came from inside the private network. Now I can update my check for the secret CF header and add a condition to allow traffic when that header is not present. Now, our health checks and metrics are working again. All traffic is proxied through CF and I can add rules to block external traffic to our metrics or health check endpoints as well. I think it would be more secure to use cloudflared tunnel, and in some ways simpler, since there’s less application logic to maintain, but it does involve adding another service and managing all that comes with that.

I hope this helps someone. And if I am making any critical mistakes here, please let me know so I can adapt accordingly!

1 Like

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.