Ruby on Rails - puma starts too many workers on Fly

Hi,

I am totally lost here. Trying to migrate from Heroku, and would like to use buildpacks.

My issues is that on Fly, when my rails app starts, puma starts 28 workers and runs out of memory.

Here’s on my machine, with Rails prod enviroment:

[3060] Puma starting in cluster mode...
[3060] * Puma version: 5.6.5 (ruby 3.1.2-p20) ("Birdie's Version")
[3060] *  Min threads: 5
[3060] *  Max threads: 5
[3060] *  Environment: production
[3060] *   Master PID: 3060
[3060] *      Workers: 2
[3060] *     Restarts: (✔) hot (✔) phased
[3060] * Listening on http://0.0.0.0:3000
[3060] Use Ctrl-C to stop

Here’s on Fly:

 2022-10-14T06:56:13.108 app[4b6e2f41] fra [info] [520] Puma starting in cluster mode...

2022-10-14T06:56:13.108 app[4b6e2f41] fra [info] [520] * Puma version: 5.6.5 (ruby 3.1.2-p20) ("Birdie's Version")

2022-10-14T06:56:13.108 app[4b6e2f41] fra [info] [520] * Min threads: 5

2022-10-14T06:56:13.108 app[4b6e2f41] fra [info] [520] * Max threads: 5

2022-10-14T06:56:13.108 app[4b6e2f41] fra [info] [520] * Environment: production

2022-10-14T06:56:13.108 app[4b6e2f41] fra [info] [520] * Master PID: 520

2022-10-14T06:56:13.108 app[4b6e2f41] fra [info] [520] * Workers: 28

2022-10-14T06:56:13.108 app[4b6e2f41] fra [info] [520] * Restarts: (✔) hot (✖) phased 
 2022-10-14T06:56:13.108 app[4b6e2f41] fra [info] [520] * Preloading application
2022-10-14T06:56:21.004 app[4b6e2f41] fra [info] [520] * Listening on http://0.0.0.0:3000

2022-10-14T06:56:21.005 app[4b6e2f41] fra [info] [520] Use Ctrl-C to stop 

fly.toml

app = "bsdsec"
kill_signal = "SIGINT"
kill_timeout = 5

[build]
  builder = "heroku/buildpacks:20"
  buildpacks = ["heroku/nodejs", "heroku/ruby"]

  [build.args]
    SECRET_KEY_BASE="asdf"

[deploy]
  release_command = "bundle exec rails db:migrate"

[processes]
  app = "bundle exec puma -C config/puma.rb"

[[services]]
  http_checks = []
  internal_port = 3000
  processes = ["app"]
  protocol = "tcp"
  script_checks = []
  [services.concurrency]
    hard_limit = 25
    soft_limit = 20
    type = "connections"

  [[services.ports]]
    force_https = true
    handlers = ["http"]
    port = 80

  [[services.ports]]
    handlers = ["tls", "http"]
    port = 443

  [[services.tcp_checks]]
    grace_period = "1s"
    interval = "15s"
    restart_limit = 0
    timeout = "2s"

puma config:

# Puma can serve each request in a thread from an internal thread pool.
# The `threads` method setting takes two numbers: a minimum and maximum.
# Any libraries that use thread pools should be configured to match
# the maximum value specified for Puma. Default is set to 5 threads for minimum
# and maximum; this matches the default thread size of Active Record.
#
max_threads_count = ENV.fetch("RAILS_MAX_THREADS") { 5 }
min_threads_count = ENV.fetch("RAILS_MIN_THREADS") { max_threads_count }
threads min_threads_count, max_threads_count

# Specifies the `worker_timeout` threshold that Puma will use to wait before
# terminating a worker in development environments.
#
worker_timeout 3600 if ENV.fetch("RAILS_ENV", "development") == "development"

# Specifies the `port` that Puma will listen on to receive requests; default is 3000.
#
port ENV.fetch("PORT") { 3000 }

# Specifies the `environment` that Puma will run in.
#
environment ENV.fetch("RAILS_ENV") { "development" }

# Specifies the `pidfile` that Puma will use.
pidfile ENV.fetch("PIDFILE") { "tmp/pids/server.pid" }

# Specifies the number of `workers` to boot in clustered mode.
# Workers are forked web server processes. If using threads and workers together
# the concurrency of the application would be max `threads` * `workers`.
# Workers do not work on JRuby or Windows (both of which do not support
# processes).
#
workers ENV.fetch("WEB_CONCURRENCY") { 2 }

# Use the `preload_app!` method when specifying a `workers` number.
# This directive tells Puma to first boot the application and load code
# before forking the application. This takes advantage of Copy On Write
# process behavior so workers use less memory.
#
# preload_app!

# Allow puma to be restarted by `rails restart` command.
plugin :tmp_restart

Never had any issues on Heroku.

What does the trick is to not use config file:

[processes]
  app = "bundle exec puma -t 5:5 -w 0"

Do you have WEB_CONCURRENCY set in your secrets that got pulled over from Heroku? You can check with fly ssh console -C env.

Nope, I don’t. I let it go with 2 defeault ones.

What happens when you set this in your fly.toml file?

[env]
  WEB_CONCURRENCY = "2"

Or if you hard code workers 2 in the config file?

Setting ENV has the same result: 28 workers.
Hardcoding the value works well.

What exact output do you get when you run this? fly ssh console -C env | grep WEB_CONCURRENCY

And this:

fly ssh console
# ruby -e 'puts ENV.fetch("WEB_CONCURRENCY", 2).to_i'

Next thing I’d like for you to try is to call to_i on the ENV var in your config file, so your config file should look like this:

workers ENV.fetch("WEB_CONCURRENCY", 2).to_i

Adding to .to_i still produces negative results.

.fly/bin/flyctl ssh console -C env | grep WEB_CONCURRENCY
Connecting to top1.nearest.of.bsdsec.internal...
.fly/bin/flyctl ssh console
Connecting to top1.nearest.of.bsdsec.internal... complete
# ruby -e 'puts ENV.fetch("WEB_CONCURRENCY", 2).to_i'
2

Also tried actually setting the WEB_CONCURENCY via secrets but no change in any behaviour

This issue was reported in another thread ENVs shown by printenv don't match build envs printed to console - #15 by mobilevet

The common trait is that they’re both buildpack images.

A solution is still unknown, but at least there’s a pattern to 28.

That is weird, because I use many ENV variables, like ie. “TWITTER_ACCESS_TOKEN” which work just fine.

Yeah it’s very weird. I don’t know a lot about build packs, but the folks who do inside Fly say they’re complicated beasts with lots of moving parts.

A workaround would be to use a Ruby Dockerfile instead of a Heroku build pack. I wrote instructions on how to migrate Rails applications from Heroku to Fly with at Migrate from Heroku · Fly Docs that can walk you through the steps. You won’t have the “28” issue if you are able to go down that path.

I don’t use fly.io, but I’m facing a similar issue when building a Rails app using the heroku buildpacks. I use the heroku/ruby buildpack as the main buildpack and the heroku/nodejs buildpack to precompile my assets.

After some investigation it seems that for some reason the heroku/nodejs buildpack automatically sets the WEB_CONCURRENCY environment variable and there is no way to override that. In my case it’s also set to 28.

@hovancik not sure if you’re also using the nodejs buildpack.

Could somebody from fly.io confirm if they use the heroku buildpacks?

By the way, there’s an issue open on the heroku/nodejs buildpack repo: WEB_CONCURRENCY env var is overwritten · Issue #210 · heroku/buildpacks-nodejs · GitHub

Buildpacks are used within Fly, depending on what framework you’re using and what version of flyctl you used to launch and deploy your app initially.

For example, the Ruby (not Rails) launcher uses a build pack: flyctl/ruby.go at b6d7c3cf51614c1a8b881ec752f8e15387fbd7b1 · superfly/flyctl · GitHub

I’m still going back and forth with Heroku on this issue, but and having a difficult time conveying to them that they’re clobbering that variable (they think they check it first).

For now I’m inclined to say if you want to control this variable, use a different ENV var name like PUMA_WORKERS=2

Thanks for looking into this @Brad. Are you talking to somebody directly on Heroku or is there a Github issue I can follow?

It definitely is an issue with the heroku/nodejs buildpack. I’ve created a repo with reproduction steps: GitHub - jordinl/heroku-buildpacks-bug

If it’s of any help, the paketo buildpacks don’t seem to have this bug. Ultimately, I think fly.io should maintain their own set of buildpacks, as opposed to relying on some other company’s ones.

While we continue to support buildpacks our preference is that people use dockerfiles. If you have a puma app (with or without rails) and would like help converting to dockerlet us know.

I opened a ticket within Heroku and advised them to update the Github issues, but they don’t seem to be grasping my ask. You might open a ticket with them from the Heroku dashboard and reference the same Github issues—sometimes when they see a pattern emerge they put more resources on it :smile:

To Sam’s point, you could think of a Buildpack as a Dockerfile, but with a ton of if/else statements that magically detect things in your application. That can lead to surprising behavior, like WEB_CONCURRENCY being 28. Surprising behavior and running production web apps isn’t a great mix.

A point I have seen raised, and maybe this is your sentiment, is that your don’t want to see a Dockerfile in your project directory because you don’t want to deal with it. I feel the same and agree it’s a big ask to throw Dockerfiles in projects, but there’s no plans to change from that approach.

On the plus side, the Fly team does ship some pretty sane Dockerfiles that work for most framework usecases. A huge plus is when you need access to Linux, its much easier to do so in a Dockerfile (once you get past the huge learning curve) to change the distro, add some packages, etc.

Well, yeah. As a developer, I don’t want to learn how to create correct and secure and everything Docker container for my app. I want to write code. I would never feel safe. That’s why we all used heroku and their buildpack: git push heroku. Maybe it is all magic and ifs, but I never had any issues with it :slight_smile:

There is also a second issue on the heroku buildpack, specifically about the 28: WEB_CONCURRENCY is miscalculated on ECS Fargate · Issue #220 · heroku/buildpacks-nodejs · GitHub, FYI.