Cloudflare 525 error randomly occurs

Makes perfect sense.

There are arguments for and against putting Cloudflare in front (aka “orange cloud”, using its proxy).

For example if you want to know which country your user is in, Cloudflare provides that in a header. Fly does not. So that would be an example where using Cloudflare’s proxy does make sense.

Using Cloudflare’s proxy naturally will always add another network “hop” as a request has to go via its server, and then on to Fly. So you might think that is an argument to not use it, as it increases latency. Which may be the case. But Cloudflare has more edge locations and so a user may (depending on where they are in the world, another variable) get to one of their servers faster. And so even with the extra hop they add, the overall request may end up faster.

Also, Cloudflare supports the very latest networking tricks like HTTP/3. Not sure where Fly is at with that. So again, may be faster … if those make a difference to what your app does.

Do you need a WAF? Cloudflare does do more than DDoS protection and filters out other kinds of attacks. So, again, depends if you need that.

Arguments against? Added complexity, random 525s (like you found), more networks means more to go wrong … and harder to debug when it does.

There isn’t a clear answer. Sadly “it depends” :slight_smile:

2 Likes

I deployed an app to Fly and am getting these 525 errors when trying to make requests from Cloudflare Workers to the Fly app. :confused:

So the only solution is managing TLS in my app?

I didn’t get the 525 errors when using Google Cloud Run. I imagine there is some sort of balancer since you can have multiple instances currently processing requests.

Hmm. Yes, I had the same.

The only way I fixed it was by doing the TLS in-app. Using one of these https://developers.cloudflare.com/ssl/origin-configuration/origin-ca

Is that the only solution? Not sure, but it’s the only thing that worked for me.

2 Likes

Thanks for the link @greg .

So I download the certs from Cloudflare, and then use them in my application.

Then @johan mentioned:

Setup Fly to not terminate TLS at the edge

How is this configured?

1 Like

You can set up an empty handlers = [] section on your [[services.ports]], which basically tells Fly that you want raw connections with no transformations or terminations.

That lets you handle everything in-app.

2 Likes

I got it running, thanks everyone for your help.

These are the exact steps I took:

1 Add your domain to Cloudflare

CF will not give an origin cert for a domain that they do not control. You won’t be able to do this for your Fly dev domain (eg: your-app.fly.dev). Add the domain to CF, change the NS in your registrar’s dashboard, and wait until the NS have propagated.

2 Configure the DNS to point to your Fly app

You will need to add A records so that your domain or subdomain point to your Fly app.

To get the IP of your Fly app do:

fly info

In the DNS management section of CF’s dashboard add the A record. Typically when doing this, you’d want to avoid CF being a proxy, but in this case you must enable the option (the orange cloud) because the CF origin cert is only valid between your app and CF. Browsers will not accept the origin cert as valid.

3 Get the origin certs from CloudFlare

Go to the SSL/origin server section fo the dashboard. Create and download the cert and the key. I used PEM format, but YMMV.

4 Add the certs to your application

Obviously this will change for every app. I’m using Node.js and Fastify and injecting the certs via an env var that comes from two Fly secrets.

To add a cert file to a Fly secret use this:

fly secrets set SSL_KEY=- < key.pem

5 Configure fly to let you handle SSL in your app

In the fly.toml file remove the http and tls handlers for the 443 port.

  [[services.ports]]
    handlers = []
    port = 443

Honestly, I don’t know if this needs to be done for the 80 port too. I think CF acting as a proxy won’t let you access the Fly app via HTTP anyway.

And that’s it. Deploy your app and no more 525 errors!

Took me a couple of hours at first, but now that I understand the whole process and its pitfalls it should be super easy to replicate.

6 Likes

Nice.

Yes it’s a pain to set up but it seems to solve the problem. And their certificate does not expire (or at least not for ages as I recall) so you can just forget about it once done.

Yeah I’ve solved this for the next 15 years :rofl:

2 Likes

Thanks, @pier this an excellent guide!

2 Likes

Thanks for awesome guidance.

I just tried following the steps and change SSL/TLS encryption mode to Full (strict). Now the page is getting Error 525.

I guess after I set SSL_KEY in secrets, in my app i need a function to put the SSL_KEY into a file and place the file in the root? how you accomplish this?

Or

Or the SSL_KEY env will be automatically detected by fly.io as cert key? and I do not need to do anything from my side?

fyi, I am quite new this these things, and i am in confused mode. hehe

I guess after I set SSL_KEY in secrets, in my app i need a function to put the SSL_KEY into a file and place the file in the root? how you accomplish this?

I don’t think you need to save it to a file in order to use it.
You could directly read the env variable in your app code and use it from there.
Steps 2 & 3 from Use Cloudflare Certs to FlyIO - #8 by ignoramous shows you how.

Or the SSL_KEY env will be automatically detected by fly.io as cert key? and I do not need to do anything from my side?

I’m afraid you have to handle TLS in your app :slight_smile:
Fly won’t auto-detect that env as a cert and do anything.

1 Like

Get it! I will try that. Thanks FrequentSolver again :gift_heart:

To read the certs and enable HTTPS will depend on your application, framework, runtime, etc.

For example, in Fastify and other Node servers you simply pass the certs when configuring the server.

https://nodejs.org/dist/latest-v14.x/docs/api/https.html#https_https_createserver_options_requestlistener

I don’t use CloudFlare, but is it really not possible to have them communicate with https://<app>.fly.dev? Because that’s kind of silly.

You probably can setup the CDN with a CNAME record instead of an A record over a Fly subdomain.

My issue was about making requests from Cloudflare Workers to the Fly app while using an A record to the IPv4 of the Fly app.

Yep, if you set that app.fly.dev as a CNAME record, they will pull from that using https.

My issue was getting random 525s (when behind Cloudflare, which I wanted to do to get its geo-ip header). But it did work 99% of the time.

Not sure from @Loonb if they are consistently getting a 525 or randomly, like me. I solved it by handing TLS in-app but given all the changes to the proxy/routing since then, that may not be needed as I’ve not tried reverting back.

2 Likes

Yes randomly and rarely. Currently I leave it as it is. Coz I have no idea how to do the TLS in app using django.

This was really helpful, thank you!

I’ve set up my app to handle SSL and it all works, but I’m still getting random 525 errors that last 5 - 10 minutes.

I’m clutching at straws trying to figure out what I’m missing to stop these errors.

This might be a n00b question, but if I’ve added the origin certs to my app, do I still need the fly created SSL certificate and _acme-challenge DNS record? Could having those in place be causing my random 525 errors?

Thank you so much! :pray:

In step 5 I detailed how to tell Fly that your app will be handling SSL (instead of Fly).

If Cloudflare is trying to communicate with your app, and it receives the Fly certs instead of the Cloudflare issued cert, then you will get the 525 errors.

AFAIK the DNS records are just for verification purposes and shouldn’t interfere with SSL.

1 Like

Perfect, thank you! I’ll delete the Fly SSL certificate and hopefully the 525 errors will stop! :crossed_fingers: