How to run multiple processes in one machine?

Questions / Help
1

Hey there,

I am using good_job to run multiple worker processes on one fly machine for my rails application.
The author of good_job suggests setting

combined_worker: bundle exec good_job --max-threads=5 & bundle exec good_job --queues="transactional_messages" --max-threads=2 & bundle exec good_job --queues="batch_processing" --max-threads=1 & wait -n

in the procfile, which on fly, I guess, is the [processes] section of the fly.toml file. Running that command on my terminal in my rails root directory works just fine, but when I deploy with

[processes]
  web = "/rails/bin/rails server"
  worker = 'bundle exec good_job --queues="one" --max-threads=8 & bundle exec good_job --queues="two,three" --max-threads=2 & bundle exec good_job --queues="-one,two,three" --max-threads=2 & wait -n'

the logs of the worker machine complains:

ERROR: "good_job start" was called with arguments ["&", "bundle", "exec", "good_job", "&", "bundle", "exec", "good_job", "&", "wait", "-n"]
Usage: "good_job start"

How would I need to adjust the worker process line in my fly.toml? I’d like to have three processes inside the worker machine with the respective set of threads set in that command.

Thanks for any guidance!

2

that’s interesting, I wonder how that command gets parsed. well, instead of backgrounding several processes inline as part of fly.toml, you could add a simple script (or rake task) to wrap all that and call it instead

e.g.

#!/usr/bin/env sh

bundle exec good_job --queues="one" --max-threads=8 & bundle exec good_job --queues="two,three" --max-threads=2 & bundle exec good_job --queues="-one,two,three"

then

[processes]
  web = "/rails/bin/rails server"
  worker = "script/do-work.sh"
3

Thank you for the suggestion. That is exactly what I did now.
But it does not feel like a very elegant solution, and I’d also like to know how the “process” line is parsed.

4

& is a shell feature. If you would like to run a shell and have it run your commands try:

bash -c "foo & bar & wait"

Such an approach tends to get messy quickly with quotes, so that’s why I prefer to create a script and run that.

1 Like
5

Ah, I see. Thank you for the explanation!

6

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.