[Rails] Sidekiq suddenly can't connect to Redis

rspeicher · August 8, 2023, 12:11am

I’ve had a Rails app running smoothly on Fly for months, and then suddenly today the machine acting as the Sidekiq worker enters a crash loop due to being unable to connect to Upstash Redis:

2023-08-08T00:08:12Z app[17811772a56789] dfw [info]2023-08-08T00:08:12.918Z pid=255 tid=3dn INFO: Booted Rails 7.0.6 application in production environment
2023-08-08T00:08:12Z app[17811772a56789] dfw [info]2023-08-08T00:08:12.918Z pid=255 tid=3dn INFO: Running in ruby 3.2.1 (2023-02-08 revision 31819e82c8) [x86_64-linux]
2023-08-08T00:08:12Z app[17811772a56789] dfw [info]2023-08-08T00:08:12.918Z pid=255 tid=3dn INFO: See LICENSE and the LGPL-3.0 for licensing details.
2023-08-08T00:08:12Z app[17811772a56789] dfw [info]2023-08-08T00:08:12.918Z pid=255 tid=3dn INFO: Upgrade to Sidekiq Pro for more features and support: https://sidekiq.org
2023-08-08T00:08:12Z app[17811772a56789] dfw [info]2023-08-08T00:08:12.919Z pid=255 tid=3dn INFO: Sidekiq 7.1.2 connecting to Redis with options {:size=>10, :pool_name=>"internal", :url=>"redis://default:REDACTED@fly-REDACTED.upstash.io"}
2023-08-08T00:08:12Z app[17811772a56789] dfw [info]EOFError
2023-08-08T00:08:12Z app[17811772a56789] dfw [info]/rails/vendor/bundle/ruby/3.2.0/gems/redis-client-0.14.1/lib/redis_client/ruby_connection.rb:101:in `rescue in read'
2023-08-08T00:08:12Z app[17811772a56789] dfw [info]/rails/vendor/bundle/ruby/3.2.0/gems/redis-client-0.14.1/lib/redis_client/ruby_connection.rb:92:in `read'
2023-08-08T00:08:12Z app[17811772a56789] dfw [info]/rails/vendor/bundle/ruby/3.2.0/gems/redis-client-0.14.1/lib/redis_client/connection_mixin.rb:51:in `block in call_pipelined'
2023-08-08T00:08:12Z app[17811772a56789] dfw [info]/rails/vendor/bundle/ruby/3.2.0/gems/redis-client-0.14.1/lib/redis_client/connection_mixin.rb:49:in `times'
2023-08-08T00:08:12Z app[17811772a56789] dfw [info]/rails/vendor/bundle/ruby/3.2.0/gems/redis-client-0.14.1/lib/redis_client/connection_mixin.rb:49:in `call_pipelined'
2023-08-08T00:08:12Z app[17811772a56789] dfw [info]/rails/vendor/bundle/ruby/3.2.0/gems/redis-client-0.14.1/lib/redis_client.rb:696:in `block in connect'
2023-08-08T00:08:12Z app[17811772a56789] dfw [info]/rails/vendor/bundle/ruby/3.2.0/gems/redis-client-0.14.1/lib/redis_client/middlewares.rb:16:in `call'
2023-08-08T00:08:12Z app[17811772a56789] dfw [info]/rails/vendor/bundle/ruby/3.2.0/gems/redis-client-0.14.1/lib/redis_client.rb:695:in `connect'
2023-08-08T00:08:12Z app[17811772a56789] dfw [info]/rails/vendor/bundle/ruby/3.2.0/gems/redis-client-0.14.1/lib/redis_client.rb:657:in `raw_connection'
2023-08-08T00:08:12Z app[17811772a56789] dfw [info]/rails/vendor/bundle/ruby/3.2.0/gems/redis-client-0.14.1/lib/redis_client.rb:624:in `ensure_connected'
2023-08-08T00:08:12Z app[17811772a56789] dfw [info]/rails/vendor/bundle/ruby/3.2.0/gems/redis-client-0.14.1/lib/redis_client.rb:209:in `call'
2023-08-08T00:08:12Z app[17811772a56789] dfw [info]/rails/vendor/bundle/ruby/3.2.0/gems/redis-client-0.14.1/lib/redis_client/decorator.rb:26:in `call'
2023-08-08T00:08:12Z app[17811772a56789] dfw [info]/rails/vendor/bundle/ruby/3.2.0/gems/sidekiq-7.1.2/lib/sidekiq/config.rb:145:in `block in redis_info'
2023-08-08T00:08:12Z app[17811772a56789] dfw [info]/rails/vendor/bundle/ruby/3.2.0/gems/sidekiq-7.1.2/lib/sidekiq/config.rb:164:in `block in redis'
2023-08-08T00:08:12Z app[17811772a56789] dfw [info]/rails/vendor/bundle/ruby/3.2.0/gems/connection_pool-2.4.1/lib/connection_pool.rb:110:in `block (2 levels) in with'
2023-08-08T00:08:12Z app[17811772a56789] dfw [info]/rails/vendor/bundle/ruby/3.2.0/gems/connection_pool-2.4.1/lib/connection_pool.rb:109:in `handle_interrupt'
2023-08-08T00:08:12Z app[17811772a56789] dfw [info]/rails/vendor/bundle/ruby/3.2.0/gems/connection_pool-2.4.1/lib/connection_pool.rb:109:in `block in with'
2023-08-08T00:08:12Z app[17811772a56789] dfw [info]/rails/vendor/bundle/ruby/3.2.0/gems/connection_pool-2.4.1/lib/connection_pool.rb:106:in `handle_interrupt'
2023-08-08T00:08:12Z app[17811772a56789] dfw [info]/rails/vendor/bundle/ruby/3.2.0/gems/connection_pool-2.4.1/lib/connection_pool.rb:106:in `with'
2023-08-08T00:08:12Z app[17811772a56789] dfw [info]/rails/vendor/bundle/ruby/3.2.0/gems/sidekiq-7.1.2/lib/sidekiq/config.rb:161:in `redis'
2023-08-08T00:08:12Z app[17811772a56789] dfw [info]/rails/vendor/bundle/ruby/3.2.0/gems/sidekiq-7.1.2/lib/sidekiq/config.rb:144:in `redis_info'
2023-08-08T00:08:12Z app[17811772a56789] dfw [info]/rails/vendor/bundle/ruby/3.2.0/gems/sidekiq-7.1.2/lib/sidekiq/cli.rb:75:in `run'
2023-08-08T00:08:12Z app[17811772a56789] dfw [info]/rails/vendor/bundle/ruby/3.2.0/gems/sidekiq-7.1.2/bin/sidekiq:31:in `<top (required)>'
2023-08-08T00:08:12Z app[17811772a56789] dfw [info]/rails/vendor/bundle/ruby/3.2.0/bin/sidekiq:25:in `load'
2023-08-08T00:08:12Z app[17811772a56789] dfw [info]/rails/vendor/bundle/ruby/3.2.0/bin/sidekiq:25:in `<top (required)>'
2023-08-08T00:08:12Z app[17811772a56789] dfw [info]/rails/vendor/bundle/ruby/3.2.0/gems/bundler-2.4.5/lib/bundler/cli/exec.rb:58:in `load'
2023-08-08T00:08:12Z app[17811772a56789] dfw [info]/rails/vendor/bundle/ruby/3.2.0/gems/bundler-2.4.5/lib/bundler/cli/exec.rb:58:in `kernel_load'
2023-08-08T00:08:12Z app[17811772a56789] dfw [info]/rails/vendor/bundle/ruby/3.2.0/gems/bundler-2.4.5/lib/bundler/cli/exec.rb:23:in `run'
2023-08-08T00:08:12Z app[17811772a56789] dfw [info]/rails/vendor/bundle/ruby/3.2.0/gems/bundler-2.4.5/lib/bundler/cli.rb:491:in `exec'
2023-08-08T00:08:12Z app[17811772a56789] dfw [info]/rails/vendor/bundle/ruby/3.2.0/gems/bundler-2.4.5/lib/bundler/vendor/thor/lib/thor/command.rb:27:in `run'
2023-08-08T00:08:12Z app[17811772a56789] dfw [info]/rails/vendor/bundle/ruby/3.2.0/gems/bundler-2.4.5/lib/bundler/vendor/thor/lib/thor/invocation.rb:127:in `invoke_command'
2023-08-08T00:08:12Z app[17811772a56789] dfw [info]/rails/vendor/bundle/ruby/3.2.0/gems/bundler-2.4.5/lib/bundler/vendor/thor/lib/thor.rb:392:in `dispatch'
2023-08-08T00:08:12Z app[17811772a56789] dfw [info]/rails/vendor/bundle/ruby/3.2.0/gems/bundler-2.4.5/lib/bundler/cli.rb:34:in `dispatch'
2023-08-08T00:08:12Z app[17811772a56789] dfw [info]/rails/vendor/bundle/ruby/3.2.0/gems/bundler-2.4.5/lib/bundler/vendor/thor/lib/thor/base.rb:485:in `start'
2023-08-08T00:08:12Z app[17811772a56789] dfw [info]/rails/vendor/bundle/ruby/3.2.0/gems/bundler-2.4.5/lib/bundler/cli.rb:28:in `start'
2023-08-08T00:08:12Z app[17811772a56789] dfw [info]/rails/vendor/bundle/ruby/3.2.0/gems/bundler-2.4.5/exe/bundle:45:in `block in <top (required)>'
2023-08-08T00:08:12Z app[17811772a56789] dfw [info]/rails/vendor/bundle/ruby/3.2.0/gems/bundler-2.4.5/lib/bundler/friendly_errors.rb:117:in `with_friendly_errors'
2023-08-08T00:08:12Z app[17811772a56789] dfw [info]/rails/vendor/bundle/ruby/3.2.0/gems/bundler-2.4.5/exe/bundle:33:in `<top (required)>'
2023-08-08T00:08:12Z app[17811772a56789] dfw [info]/usr/local/bundle/bin/bundle:25:in `load'
2023-08-08T00:08:12Z app[17811772a56789] dfw [info]/usr/local/bundle/bin/bundle:25:in `<main>'
2023-08-08T00:08:13Z app[17811772a56789] dfw [info] INFO Main child exited normally with code: 1

I’m able to connect to Redis just fine with fly redis connect. I also provisioned a non-free Upstash instance and updated the REDIS_URL secret, to no avail.

Similar error directly in the production Rails console:

irb(main):001:0> require "redis"
=> false
irb(main):002:0> redis = Redis.new(url: "redis://default:REDACTED@fly-REDACTED.upstash.io")
=> #<Redis client v4.8.1 for redis://fly-REDACTED.upstash.io:6379/0>
irb(main):003:0> redis.keys
/rails/vendor/bundle/ruby/3.2.0/gems/redis-4.8.1/lib/redis/client.rb:306:in `rescue in io': Connection lost (ECONNRESET) (Redis::ConnectionError)
/usr/local/lib/ruby/3.2.0/socket.rb:456:in `__read_nonblock': Connection reset by peer (Errno::ECONNRESET)

rspeicher · August 8, 2023, 1:09am

Typical, now that I’ve bothered posting about it it seems to have resolved itself.

nheindev · August 10, 2023, 4:49pm

Would you be able to share any of your sidekiq worker logic or config?

I’m having this issue as well: Redis not connecting to Rails app - #3 by nheindev

rspeicher · August 10, 2023, 6:20pm

@nheindev There’s no Sidekiq configuration in the app beyond config.active_job.queue_adapter = :sidekiq in config/application.rb.

By default Sidekiq is pulling connection info from the REDIS_URL secret set in Fly.

My fly.toml just has:

[processes]
  web = "bin/rails server"
  worker = "bundle exec sidekiq"

So it’s very basic setup/configuration. Sorry that won’t be very helpful.

system · August 17, 2023, 6:21pm

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Rails and Sidekiq issue. How can I view sidekiq logs? Questions / Help rails	4	2575	February 7, 2023
Redis/Upstash outage ?	3	369	September 5, 2022
My App's having a Major Outage: Cannot connect to Redis Questions / Help redis	2	182	January 15, 2024
Upstash: DFW region showing read errors Questions / Help	0	247	August 27, 2022
Sidekiq workers froze, now deployments are stuck Questions / Help	1	408	May 26, 2022

[Rails] Sidekiq suddenly can't connect to Redis

Related Topics