Ruby on Rails - puma starts too many workers on Fly

What does the trick is to not use config file:

[processes]
  app = "bundle exec puma -t 5:5 -w 0"

Do you have WEB_CONCURRENCY set in your secrets that got pulled over from Heroku? You can check with fly ssh console -C env.

Nope, I don’t. I let it go with 2 defeault ones.

What happens when you set this in your fly.toml file?

[env]
  WEB_CONCURRENCY = "2"

Or if you hard code workers 2 in the config file?

Setting ENV has the same result: 28 workers.
Hardcoding the value works well.

What exact output do you get when you run this? fly ssh console -C env | grep WEB_CONCURRENCY

And this:

fly ssh console
# ruby -e 'puts ENV.fetch("WEB_CONCURRENCY", 2).to_i'

Next thing I’d like for you to try is to call to_i on the ENV var in your config file, so your config file should look like this:

workers ENV.fetch("WEB_CONCURRENCY", 2).to_i

Adding to .to_i still produces negative results.

.fly/bin/flyctl ssh console -C env | grep WEB_CONCURRENCY
Connecting to top1.nearest.of.bsdsec.internal...
.fly/bin/flyctl ssh console
Connecting to top1.nearest.of.bsdsec.internal... complete
# ruby -e 'puts ENV.fetch("WEB_CONCURRENCY", 2).to_i'
2

Also tried actually setting the WEB_CONCURENCY via secrets but no change in any behaviour

This issue was reported in another thread ENVs shown by printenv don't match build envs printed to console - #15 by mobilevet

The common trait is that they’re both buildpack images.

A solution is still unknown, but at least there’s a pattern to 28.

That is weird, because I use many ENV variables, like ie. “TWITTER_ACCESS_TOKEN” which work just fine.

Yeah it’s very weird. I don’t know a lot about build packs, but the folks who do inside Fly say they’re complicated beasts with lots of moving parts.

A workaround would be to use a Ruby Dockerfile instead of a Heroku build pack. I wrote instructions on how to migrate Rails applications from Heroku to Fly with at Migrate from Heroku · Fly Docs that can walk you through the steps. You won’t have the “28” issue if you are able to go down that path.

I don’t use fly.io, but I’m facing a similar issue when building a Rails app using the heroku buildpacks. I use the heroku/ruby buildpack as the main buildpack and the heroku/nodejs buildpack to precompile my assets.

After some investigation it seems that for some reason the heroku/nodejs buildpack automatically sets the WEB_CONCURRENCY environment variable and there is no way to override that. In my case it’s also set to 28.

@hovancik not sure if you’re also using the nodejs buildpack.

Could somebody from fly.io confirm if they use the heroku buildpacks?

By the way, there’s an issue open on the heroku/nodejs buildpack repo: WEB_CONCURRENCY env var is overwritten · Issue #210 · heroku/buildpacks-nodejs · GitHub

Buildpacks are used within Fly, depending on what framework you’re using and what version of flyctl you used to launch and deploy your app initially.

For example, the Ruby (not Rails) launcher uses a build pack: flyctl/ruby.go at b6d7c3cf51614c1a8b881ec752f8e15387fbd7b1 · superfly/flyctl · GitHub

I’m still going back and forth with Heroku on this issue, but and having a difficult time conveying to them that they’re clobbering that variable (they think they check it first).

For now I’m inclined to say if you want to control this variable, use a different ENV var name like PUMA_WORKERS=2

Thanks for looking into this @Brad. Are you talking to somebody directly on Heroku or is there a Github issue I can follow?

It definitely is an issue with the heroku/nodejs buildpack. I’ve created a repo with reproduction steps: GitHub - jordinl/heroku-buildpacks-bug

If it’s of any help, the paketo buildpacks don’t seem to have this bug. Ultimately, I think fly.io should maintain their own set of buildpacks, as opposed to relying on some other company’s ones.

While we continue to support buildpacks our preference is that people use dockerfiles. If you have a puma app (with or without rails) and would like help converting to dockerlet us know.

I opened a ticket within Heroku and advised them to update the Github issues, but they don’t seem to be grasping my ask. You might open a ticket with them from the Heroku dashboard and reference the same Github issues—sometimes when they see a pattern emerge they put more resources on it :smile:

To Sam’s point, you could think of a Buildpack as a Dockerfile, but with a ton of if/else statements that magically detect things in your application. That can lead to surprising behavior, like WEB_CONCURRENCY being 28. Surprising behavior and running production web apps isn’t a great mix.

A point I have seen raised, and maybe this is your sentiment, is that your don’t want to see a Dockerfile in your project directory because you don’t want to deal with it. I feel the same and agree it’s a big ask to throw Dockerfiles in projects, but there’s no plans to change from that approach.

On the plus side, the Fly team does ship some pretty sane Dockerfiles that work for most framework usecases. A huge plus is when you need access to Linux, its much easier to do so in a Dockerfile (once you get past the huge learning curve) to change the distro, add some packages, etc.

Well, yeah. As a developer, I don’t want to learn how to create correct and secure and everything Docker container for my app. I want to write code. I would never feel safe. That’s why we all used heroku and their buildpack: git push heroku. Maybe it is all magic and ifs, but I never had any issues with it :slight_smile:

There is also a second issue on the heroku buildpack, specifically about the 28: WEB_CONCURRENCY is miscalculated on ECS Fargate · Issue #220 · heroku/buildpacks-nodejs · GitHub, FYI.

Yay! Heroku fixed it! Don't overwrite WEB_CONCURRENCY with a different value by joshwlewis · Pull Request #386 · heroku/buildpacks-nodejs · GitHub

1 Like