Rails App with Sidekiq and workers is sending 1000s of requests upstash without running any jobs

Questions / Help
1

I’ve setup a super basic rails app with Sidekiq, and a basic test job. I’ve run through the tutorial and have added the following to the fly.toml file

[processes]
web = "bin/rails server"
worker = "bundle exec sidekiq"

and changed processes = ["app"] to processes = ["web"]

Everything works, but upstash is saying it has received thousands of requests over the span of about 10 minutes with the basic test job only running twice.

Has anyone had a similar experience or know what could be causing the issue?

Screenshot 2023-04-19 at 2.37.51 PM
Screenshot 2023-04-19 at 2.37.51 PM1025×285 25.2 KB

Fly.io Logs

2023-04-19T10:24:12.316 app[22dee4fb] ams [info] I, [2023-04-19T10:24:12.315385 #520] INFO -- : [3875f305-6cfa-4456-a20d-70069dc66e7e] Started GET "/" for 66.241.125.49 at 2023-04-19 10:24:12 +0000
2023-04-19T10:24:12.316 app[22dee4fb] ams [info] I, [2023-04-19T10:24:12.316153 #520] INFO -- : [3875f305-6cfa-4456-a20d-70069dc66e7e] Processing by PagesController#home as HTML
2023-04-19T10:24:12.317 app[22dee4fb] ams [info] I, [2023-04-19T10:24:12.317342 #520] INFO -- : [3875f305-6cfa-4456-a20d-70069dc66e7e] Rendered pages/home.html.erb within layouts/application (Duration: 0.6ms | Allocations: 343)
2023-04-19T10:24:12.318 app[22dee4fb] ams [info] I, [2023-04-19T10:24:12.318009 #520] INFO -- : [3875f305-6cfa-4456-a20d-70069dc66e7e] Rendered layout layouts/application.html.erb (Duration: 1.3ms | Allocations: 893)
2023-04-19T10:24:12.318 app[22dee4fb] ams [info] I, [2023-04-19T10:24:12.318251 #520] INFO -- : [3875f305-6cfa-4456-a20d-70069dc66e7e] Completed 200 OK in 2ms (Views: 1.7ms | Allocations: 1095)
2023-04-19T10:24:14.889 app[22dee4fb] ams [info] I, [2023-04-19T10:24:14.889686 #520] INFO -- : [722a4eff-9bd4-477a-8dc2-55ba19e2ef5c] Started POST "/trigger" for 66.241.125.49 at 2023-04-19 10:24:14 +0000
2023-04-19T10:24:14.890 app[22dee4fb] ams [info] I, [2023-04-19T10:24:14.890598 #520] INFO -- : [722a4eff-9bd4-477a-8dc2-55ba19e2ef5c] Processing by PagesController#trigger as TURBO_STREAM
2023-04-19T10:24:14.890 app[22dee4fb] ams [info] I, [2023-04-19T10:24:14.890680 #520] INFO -- : [722a4eff-9bd4-477a-8dc2-55ba19e2ef5c] Parameters: {"authenticity_token"=>"[FILTERED]"}
2023-04-19T10:24:14.891 app[22dee4fb] ams [info] 2023-04-19T10:24:14.891Z pid=520 tid=f4c INFO: Sidekiq 7.0.9 connecting to Redis with options {:size=>5, :pool_name=>"internal", :url=>"redis://default:REDACTED@eu2-refined-louse-30723.upstash.io:30723"}
2023-04-19T10:24:14.933 app[46148ee7] ams [info] 2023-04-19T10:24:14.932Z pid=520 tid=ga4 class=TestJob jid=1d9e4d9b8342e119b2a44bad INFO: start
2023-04-19T10:24:14.936 app[46148ee7] ams [info] Hello World, from test job
2023-04-19T10:24:14.938 app[46148ee7] ams [info] 2023-04-19T10:24:14.938Z pid=520 tid=ga4 class=TestJob jid=1d9e4d9b8342e119b2a44bad elapsed=0.004 INFO: done
2023-04-19T10:24:14.940 app[22dee4fb] ams [info] I, [2023-04-19T10:24:14.940229 #520] INFO -- : [722a4eff-9bd4-477a-8dc2-55ba19e2ef5c] No template found for PagesController#trigger, rendering head :no_content
2023-04-19T10:24:14.940 app[22dee4fb] ams [info] I, [2023-04-19T10:24:14.940541 #520] INFO -- : [722a4eff-9bd4-477a-8dc2-55ba19e2ef5c] Completed 204 No Content in 50ms (Allocations: 1555)
2023-04-19T10:24:24.431 app[46148ee7] ams [info] 2023-04-19T10:24:24.431Z pid=520 tid=ga4 class=TestJob jid=e19933488579d4f7845402a1 INFO: start
2023-04-19T10:24:24.432 app[46148ee7] ams [info] Hello World, from test job
2023-04-19T10:24:24.432 app[46148ee7] ams [info] 2023-04-19T10:24:24.432Z pid=520 tid=ga4 class=TestJob jid=e19933488579d4f7845402a1 elapsed=0.001 INFO: done
1 Like
2

Sidekiq uses the blocking BRPOP command to fetch jobs, so that’s not the cause.

But it also runs a background thread that polls Redis for scheduled jobs. Retries are considered scheduled jobs. See:

This shouldn’t be a problem, but it’s good to know about, as future pricing plans may be pay-as-you-go based on the number of commands. And libraries like Sidekiq or BullMQ affect that command count.

3

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.