Add sidekiq worker to Rails app with sqlite and litefs

[robmc]

Hey there,

I’m working on a new Rails app that will have a lot more DB reads than writes, so wanted to try it with SQLite. Once a day the app needs to collect fresh data to show, which I’ve set up with sidekiq and the sidekiq-scheduler gem.

I’ve followed the docs and the app is running well with SQLite. But I’m unable to get it working with sidekiq.

After getting only the app running with litefs I added a new volume and then mounted it to a sidekiq worker using the fly.toml file (see below). The main app still works, but when the jobs run in sidekiq it can’t write to the database because the table doesn’t exist. The error in the logs is:

2023-06-17T09:18:14.280Z pid=521 tid=1sll WARN: ActiveRecord::StatementInvalid: Could not find table 'companies'

I’m missing something, but not sure what.

Anyone have any ideas?

The worker uses Redis and I’ve set the REDIS_URL secret. My litefs.yml file is the autogenerated one, but I switched the lease type to “static” (I was getting SQLite read only permission errors when it was “consul”).

And my fly.toml file is:

app = "appytheapp"
primary_region = "ams"
console_command = "/rails/bin/rails console"

[processes]
  app = "./bin/rails server"
  sidekiq = "bundle exec sidekiq"

[[mounts]]
  source = "data"
  destination = "/data"
  processes = ["app"]

[[mounts]]
  source = "data_worker"
  destination = "/data"
  processes = ["sidekiq"]

[http_service]
  internal_port = 3000
  force_https = true
  auto_stop_machines = false
  auto_start_machines = false
  min_machines_running = 0
  processes = ["app"]

[[statics]]
  guest_path = "/rails/public"
  url_prefix = "/"

[rubys]

At the moment, sidekiq is not a good match for litefs as all writes need to be done from the same machine. There are some approaches being worked on, such as litefs-ruby, but I wouldn’t recommend them just yet.

I note that you are setting auto_stop_machines to false. If you intending to only run one instance of your application, you can run multiple processes inside a Fly App. Coincidentally, I have been working on dockerfile-rails to allow you to specify a procfile and it will take care of updating your Dockerfile and entrypoint. I haven’t released the changes, but I’m using them myself, and you can update your Gemfile to run dockerfile-rails directly from Github: showcase/Gemfile at 21cc511905182f68d3c9b91e65244e5963c14874 · rubys/showcase · GitHub

Create a file (you can name it what you want, I suggest Procfile.fly), with the following:

web: bin/rails server
sidekiq: bundle exec sidekiq

And then generate a Dockerfile that makes use of this procfile:

bundle update
bin/rails generate dockerfile --procfile=Procfile.fly

[robmc]

Thanks @rubys I appreciate the help!

I don’t have any need to run more than one instance of my app, so I followed your suggestion to use multiple processes inside one Fly app. Your update to dockerfile-rails to specify a procfile worked a treat. My app is up and running and sidekiq is working nicely too.

I’m not sure if it was my setup, but foreman started the app on port 5000 which didn’t work as port 3000 is specified in the Dockerfile and fly.toml. So I added the port to the Procfile, which got it working. In case anyone else has the same problem, my updated Procfile.fly is:

web: bin/rails server -p 3000
sidekiq: bundle exec sidekiq