Redis Reaches Max Limit with Sidekiq in Rails App Before Running Any Jobs

[marrie]

I’ve recently integrated Redis and Sidekiq into a Rails template application but haven’t executed any jobs yet. Surprisingly, Sidekiq seems to be generating a large number of commands, quickly reaching the maximum limit set by my Redis provider, Upstash. The error message I’m encountering is as follows:

nrt [info] 2024-01-31T13:50:31.762Z pid=305 tid=d3x ERROR: ERR max daily request limit exceeded. Limit: 10000, Usage: 10000. See https://upstash.com/docs/redis/troubleshooting/max_daily_request_limit for details

To configure my application, I’ve added the following settings to my fly.toml

• fly.toml

[http_service]
  internal_port = 3000
  force_https = true
  auto_stop_machines = true
  auto_start_machines = true
  min_machines_running = 0
  processes = ["app"]

[processes]
app = "bin/rails server"
worker = "bundle exec sidekiq"

I’ve also installed the sidekiq gem and added the necessary configurations:

• application.rb

config.active_job.queue_adapter = :sidekiq

• config/initializers/sidekiq.rb

Sidekiq.configure_client do |config|
  config.redis = { url: ENV['REDIS_URL'] }
end

Sidekiq.configure_server do |config|
  config.redis = { url: ENV['REDIS_URL'] }
end

• route.rb

require 'sidekiq/web'

Sidekiq::Web.use ActionDispatch::Cookies
Sidekiq::Web.use Rails.application.config.session_store, Rails.application.config.session_options

Rails.application.routes.draw do
  mount Sidekiq::Web, at: '/sidekiq'

  Sidekiq::Web.use(Rack::Auth::Basic) do |user, password|
    [user, password] == [ENV['SIDEKIQ_USERNAME'], ENV['SIDEKIQ_PASSWORD']]
  end
end

Despite not having triggered any Sidekiq jobs, the Redis command limit is quickly reached. Could there be any default or background processes in Sidekiq or Rails that might be causing this issue? How can I investigate and resolve this to prevent hitting the limit without actual job processing?

Any insights or suggestions from the community would be greatly appreciated.

[christopher-fly]

Hi @marrie!

Sidekiq runs a number of background calls that will cause you to exceed the 10,000 commands limit for the free tier of Upstash Redis:

Apart from regular Redis calls directly related to running background jobs, Sidekiq has to make other calls to maintain its functionality.:

• Heartbeat thread makes lots of Redis calls every 5 seconds (or 10 seconds since Sidekiq v7.0.0)
• While checking for queues with jobs to process, Sidekiq makes 1 blocking call per worker every 2 seconds

You can change your Upstash plan with flyctl redis update <yourdb> and select the pay-as-you-go plan to remove the limit.

[christopher-fly]

To follow up on this, I tested this out with a bare Rails app and Sidekiq install. Within a few hours, the 10,000 command limit is exceeded.

[marrie]

Hi, @christopher-fly!

Thank you for letting me know about the Sidekiq background calls. I understand now that it is normal for Sidekiq to execute quite a few commands and that it is inevitable.

I appreciate your help.