Rails app suddenly went down?

1

Our production Rails application went down today on Fly.io.

Initially it was just returning a plain text error of “Retry later”. A re-deploy of the same version that was live fixed that.

However it had then lost the RAILS_SERVE_STATIC_FILES secret setting. I believe this was there last week (I can’t guarantee since Friday night but the site was up last week with all images, CSS and JS being served correctly) and I’m the only one with access to remove it (and I definitely didn’t).

Anyone else spotting these weirdnesses?

2

“Retry later” isn’t an error our stack would generate. Is it possible you have some rate limiting in the rails app?

Losing the RAILS_SERVE_STATIC_FILES would be surprising. Can you talk through what steps you took to diagnose that?

3

It might actually be that the Fly platform was supposed to be serving the static files instead of Rails. So it may be we never had that setting, because it was never needed. Our fly.toml contains:

[[statics]]
  guest_path = "/app/public"
  url_prefix = "/"

So maybe that subsystem (or route to our container) died?

4

Ahhh good point, we use Rack Attack. I think maybe the static thing stopped working, so our users started spamming refresh to try to get the page to load.

Then because Fly doesn’t play nicely with Rack::Request (the IP Rails uses is the one for the router/load balancer in Fly, rather than the one in the HTTP_FLY_CLIENT_IP env header) it considers all the requests to be coming from one IP (the load balancer). I’ve tweaked our config to use that header now, so hopefully we won’t get that error again.

5

Oh, that [statics] problem would make sense. We’re investigating that right now.

I think there’s a way to configure Rack::Request to use our header. I swear I’ve figured this out before. I’ll see if I can find it.

6

I think I’ve figured the Rack::Request thing:

  Rack::Attack.throttle("requests by ip", limit: 100, period: 60) do |request|
    request.env["HTTP_FLY_CLIENT_IP"]
  end

That works for me.

Good luck on the [statics] problem! We’ve worked around it with that Rails setting anyway.