Our production Rails application went down today on Fly.io.
Initially it was just returning a plain text error of “Retry later”. A re-deploy of the same version that was live fixed that.
However it had then lost the RAILS_SERVE_STATIC_FILES secret setting. I believe this was there last week (I can’t guarantee since Friday night but the site was up last week with all images, CSS and JS being served correctly) and I’m the only one with access to remove it (and I definitely didn’t).
Ahhh good point, we use Rack Attack. I think maybe the static thing stopped working, so our users started spamming refresh to try to get the page to load.
Then because Fly doesn’t play nicely with Rack::Request (the IP Rails uses is the one for the router/load balancer in Fly, rather than the one in the HTTP_FLY_CLIENT_IP env header) it considers all the requests to be coming from one IP (the load balancer). I’ve tweaked our config to use that header now, so hopefully we won’t get that error again.