Hi, I’m having an issue rate limiting Redis in my Rails application using Sidekiq. At the moment I’ve just deployed it privately and it’s only me using it however after about 5 minutes I start getting innumerable ERR max daily request limit exceeded even when I have just deployed my application and have scarcely used it.
I can’t even imagine what it’s doing to hit the rate limits so quickly. I have this same application deployed in production on Heroku which I’m trying to move to Fly.io and have never seen it hitting rate limits also being on the basic free tiers. This is a fairly low-impact application and only uses Sidekiq for a few trivial tasks.
:verbose: false
:concurrency: 10
# Set timeout to 8 on Heroku, longer if you manage your own systems.
:timeout: 8
# Sidekiq will run this file through ERB when reading it so you can
# even put in dynamic logic, like a host-specific queue.
# http://www.mikeperham.com/2013/11/13/advanced-sidekiq-host-specific-queues/
:queues:
- critical
- default
- <%= `hostname`.strip %>
- mailers
- low
# you can override concurrency based on environment
production:
:concurrency: 2
staging:
:concurrency: 15
Sidekiq will run a few commands every second to check for new jobs. Given the concurrency levels you have set, it makes sense that you’d reach this limit quickly. If you’re not able to upgrade, you could run your own Redis instance as an app inside your organization.
These concurrency settings were there from default. If I set the concurrency to 1 will it change anything? All of the things we use Sidekiq for could be in the background for 5-10 minutes without any issue. How can I set it so it doesn’t run commands every second but maybe every 60 seconds?
Would that be slow enough to live on the free tier? How could I specify this?
We’ve been running the free tier on Heroku for at least a year without any issue using these settings so I was just surprised.
Yeah, setting it to 1 helped but looking thru the logs it appears to poll on the average of every 3-5 seconds. So given that the low tier has a cap of 10,000 commands per 24 hours I will still break that before the day is finished (even at 5 seconds this still 17,280 requests per day!)
Unless I can figure out how to slow it down to 20 or 30 seconds I’ll probably just switch to another service that has a higher cap. ;-(
2022-08-30T13:48:45.296 app[d31d32f0] ams [info] I, [2022-08-30T13:48:45.294632 #515] INFO -- : [461aaab8-285e-464a-af89-180c0753fe53] Started GET "/admin/sidekiq/stats" for 168.220.95.126 at 2022-08-30 13:48:45 +0000
2022-08-30T13:48:47.907 app[d31d32f0] ams [info] I, [2022-08-30T13:48:47.905742 #515] INFO -- : [cd18be0a-98b0-4995-96ae-b95e58d6e06d] Started GET "/admin/sidekiq/stats" for 168.220.95.126 at 2022-08-30 13:48:47 +0000
2022-08-30T13:49:00.405 app[d31d32f0] ams [info] I, [2022-08-30T13:49:00.404804 #515] INFO -- : [893b98ad-55ae-40d6-acfd-7d4efb2d8e82] Started GET "/admin/sidekiq/stats" for 168.220.95.126 at 2022-08-30 13:49:00 +0000
2022-08-30T13:49:02.396 app[d31d32f0] ams [info] I, [2022-08-30T13:49:02.394895 #515] INFO -- : [b5c2f20a-a759-4dfb-a1b2-6a77223ca0a7] Started GET "/admin/sidekiq/stats" for 168.220.95.126 at 2022-08-30 13:49:02 +0000
2022-08-30T13:49:05.648 app[d31d32f0] ams [info] I, [2022-08-30T13:49:05.646651 #515] INFO -- : [76eb9092-a9fa-413e-88f5-ec53a5c0f2c3] Started GET "/admin/sidekiq/stats" for 168.220.95.126 at 2022-08-30 13:49:05 +0000
I looked into this a bit more. Sidekiq blocks on new jobs, but will reconnect after a 2 second timeout when idle. So a single worker, in theory, would need a larger timeout to prevent hitting the limit. You could try something like this in a Rails initializer:
Sidekiq::BasicFetch::TIMEOUT = 15
But the logs you posted suggest another issue on top of this. Those are web requests going to what looks like the sidekiq web interface. Do you have a browser window open that has live polling enabled?
Update, I haven’t made any changes but now I constantly get these errors. I guess I’ll try reducing concurrency back down to 1.
2022-09-02T10:17:20.487 app[dd142973] ams [info] 2022-09-02T10:17:20.487Z pid=513 tid=47mp WARN: Your Redis network connection is performing extremely poorly.
2022-09-02T10:17:20.487 app[dd142973] ams [info] Last RTT readings were [286498, 287021, 286616, 286685, 286494], ideally these should be < 1000.
2022-09-02T10:17:20.487 app[dd142973] ams [info] Ensure Redis is running in the same AZ or datacenter as Sidekiq.
2022-09-02T10:17:20.487 app[dd142973] ams [info] If these values are close to 100,000, that means your Sidekiq process may be
2022-09-02T10:17:20.487 app[dd142973] ams [info] CPU-saturated; reduce your concurrency and/or see https://github.com/mperham/sidekiq/discussions/5039
So just an update. I successfully reduced redis usage by switching to resque and resque scheduler which provided more ability to control how often redis was hit (via INTERVAL and RESQUE_SCHEDULER_INTERVAL env vars)
However, I still hit the 10k command limit for upstash free tier. I’ve concluded that even basic background processing in Rails to not be conducive to this free tier and have switched to Redis Cloud free tier instead which does not have a limit of # of commands but rather memory.
