Paid Redis not connecting to Rails Worker

Questions / Help
1

I believe this issue started when I upgraded from a Free Redis to the Paid Redis when making a new app. I copied all config from a previously working Fly V2 backend of mine.

Been trying to resolve this issue for a few days, found these topics, but they were closed to replies:

https://community.fly.io/t/rails-sidekiq-suddenly-cant-connect-to-redis/14661/4

https://community.fly.io/t/redis-not-connecting-to-rails-app/14630

The Sidekiq dashboard shows the correct Redis url.

  • Sidekiq v7.0.9
  • redis://fly-api-x-redis.upstash.io:6379/0

fly redis connect works fine from local.

fly ssh console -C "printenv" also shows the correct url.

You can see the correct url in the dashboard monitoring logs.

You can enqueue items, but they never fire.

I’ve tried dozens of variations of both V1 and V2 style toml configs, super simple or complex and also changing the web process to app process, but seeing no difference in behavior.

Here is latest:

app = "api-x"
primary_region = "mia"
kill_signal = "SIGINT"
kill_timeout = "5s"
console_command = "/rails/bin/rails console"

[experimental]
auto_rollback = true
autoscale = true

[experimental.autoscaling]
min = 1
max = 10

[build]
[build.args]
BUILD_COMMAND = "bin/rails fly:build"
BUNDLER_VERSION = "2.3.18"
NODE_VERSION = "17.2.0"
RUBY_VERSION = "3.1.2"
SERVER_COMMAND = "bin/rails fly:server"

[deploy]
release_command = "bundle exec rails db:migrate"

[env]
PORT = "8080"

[processes]
web = "bin/rails fly:server"
worker = "bundle exec sidekiq --verbose"

[[services]]
protocol = "tcp"
internal_port = 8080
processes = ["web"]

[[services.ports]]
port = 80
handlers = ["http"]
force_https = true

[[services.ports]]
port = 443
handlers = ["tls", "http"]
[services.concurrency]
type = "connections"
hard_limit = 100
soft_limit = 90

[[services.tcp_checks]]
interval = "15s"
timeout = "2s"
grace_period = "1s"
restart_limit = 0

[[statics]]
guest_path = "/app/public"
url_prefix = "/"

and

#sidekiq.rb
require "sidekiq"
require "sidekiq/web"
require "sidekiq-scheduler"
require "sidekiq-scheduler/web"

Sidekiq.configure_server do |config|
  config.redis = { url: ENV["REDIS_URL"] }
end

Sidekiq.configure_client do |config|
  config.redis = { url: ENV["REDIS_URL"] }
end

Any ideas?

2

Can you check your logs with fly logs? That should give you a hint as to what’s going on. I noticed your org registered some OOMs, meaning one of your processes has run out of memory, Maybe that’s related.

3

I scaled up the apps, but not their workers, should they be equal in size? Though apps that haven’t been deployed recently that were using free redis, their sidekiq queues now also show up within the new / paid redis instance, yet they still hang for days on end.

Neither apps can actually get the queue to fire though.

The only way to get 1 job to process is to manually run:

flyctl ssh console -C 'bundle exec sidekiq' -a app-name

But the actual worker itself, seems to never do anything.

Haven’t been able to get logs from the worker, because it never Starts up.

4

I think the best thing here is to check your logs while the instances are being deployed. That should give you some information about what’s happening.

5

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.