"Invalid VM group" error when trying to scale worker process

1

I followed this guide to set up sidekiq background workers in a rails app: Sidekiq Background Workers · Fly Docs

The problem I’m having is when I try to scale my app with:

fly scale count web=3 worker=3 -a my_app_name

Then I get this error:

Error: Invalid VM group:

If I run fly status, the process name is still saying app instead of web, and there is no worker process shown:

Instances
ID      	PROCESS	VERSION	REGION	DESIRED	STATUS 	HEALTH CHECKS     	RESTARTS	CREATED
494821df	app    	24     	jnb   	run    	running	1 total, 1 passing	0       	7m49s ago

Here is my toml config file:

app = "quickvoice-staging"
kill_signal = "SIGINT"
kill_timeout = 5

[processes]
web = "bin/rails server"
worker = "bundle exec sidekiq"

[build]
  [build.args]
    BUILD_COMMAND = "bin/rails fly:build"
    SERVER_COMMAND = "bin/rails fly:server"

[deploy]
  release_command = "bin/rails fly:release"

[env]
  PORT = "8080"

[experimental]
  allowed_public_ports = []
  auto_rollback = true

[[services]]
  http_checks = []
  internal_port = 3000
  processes = ["web"]
  protocol = "tcp"
  script_checks = []
  [services.concurrency]
    hard_limit = 25
    soft_limit = 20
    type = "connections"

  [[services.ports]]
    force_https = true
    handlers = ["http"]
    port = 80

  [[services.ports]]
    handlers = ["tls", "http"]
    port = 443

  [[services.tcp_checks]]
    grace_period = "1s"
    interval = "15s"
    restart_limit = 0
    timeout = "2s"

[[statics]]
  guest_path = "/app/public"
  url_prefix = "/"

If ssh into my app and start sidekiq manually, then it processes jobs as expected, so everything else is working, except getting the worker process to scale.

2

Hi @AndrewJ, this looks like what might happen if you hadn’t run fly deploy after editing fly.toml to add process groups.

Deployment applies your changes from fly.toml and should give you a Machine for each process you specified. Then the scale command should work.

1 Like
3

Hi @catflydotio, thanks for the reply. Strangely I deployed multiple times and had the same issue. I then tried specifying the config file path instead of the app name, and that seemed to pull the latest config file changes. However I am running into a different issue now. The deploy is throwing errors, and causes my app to give a 502 error when trying to visit the webpage. It doesn’t seem to like the port 3000 being used. Here is the error when I try visit the page:

2023-05-02T16:29:11.717 proxy[7fc484eb] jnb [warn] Failed to proxy HTTP request (error: no known healthy instances found for route tcp/443. (hint: is your app shutdown? is there an ongoing deployment with a volume or using the 'immediate' strategy? if not, this could be a delayed state issue)). Retrying in 1000 ms (attempt 80)

2023-05-02T16:29:21.759 proxy[7fc484eb] jnb [error] instance refused connection. is your app listening on 0.0.0.0:3000? make sure it is not only listening on 127.0.0.1 (hint: look at your startup logs, servers often print the address they are listening on)

Here are the build logs:

2023-05-02T16:24:45.742 runner[6fa4e00a] jnb [info] Starting instance

2023-05-02T16:24:45.825 runner[6fa4e00a] jnb [info] Configuring virtual machine

2023-05-02T16:24:45.827 runner[6fa4e00a] jnb [info] Pulling container image

2023-05-02T16:24:46.773 runner[8d667e5d] jnb [info] Starting instance

2023-05-02T16:24:46.860 runner[8d667e5d] jnb [info] Configuring virtual machine

2023-05-02T16:24:46.862 runner[8d667e5d] jnb [info] Pulling container image

2023-05-02T16:24:47.000 runner[6fa4e00a] jnb [info] Unpacking image

2023-05-02T16:24:47.009 runner[6fa4e00a] jnb [info] Preparing kernel init

2023-05-02T16:24:47.412 runner[6fa4e00a] jnb [info] Configuring firecracker

2023-05-02T16:24:47.759 runner[8d667e5d] jnb [info] Unpacking image

2023-05-02T16:24:47.766 runner[8d667e5d] jnb [info] Preparing kernel init

2023-05-02T16:24:48.120 runner[8d667e5d] jnb [info] Configuring firecracker

2023-05-02T16:24:49.459 runner[6fa4e00a] jnb [info] Starting virtual machine

2023-05-02T16:24:49.753 runner[8d667e5d] jnb [info] Starting virtual machine

2023-05-02T16:24:49.788 app[6fa4e00a] jnb [info] Starting init (commit: 15c0f38)...

2023-05-02T16:24:49.838 app[6fa4e00a] jnb [info] Preparing to run: `bin/rails server` as root

2023-05-02T16:24:49.926 app[6fa4e00a] jnb [info] 2023/05/02 16:24:49 listening on [fdaa:1:19fd:a7b:d5a6:6fa4:e00a:2]:22 (DNS: [fdaa::3]:53)

2023-05-02T16:24:50.197 app[8d667e5d] jnb [info] Starting init (commit: 15c0f38)...

2023-05-02T16:24:50.263 app[8d667e5d] jnb [info] Preparing to run: `bundle exec sidekiq` as root

2023-05-02T16:24:50.342 app[8d667e5d] jnb [info] 2023/05/02 16:24:50 listening on [fdaa:1:19fd:a7b:d5a6:8d66:7e5d:2]:22 (DNS: [fdaa::3]:53)

2023-05-02T16:24:51.570 health[6fa4e00a] jnb [warn] Health check on port 3000 is in a 'warning' state. Your app may not be responding properly. Services exposed on ports [80, 443] may have intermittent failures until the health check passes.

2023-05-02T16:24:52.566 health[6fa4e00a] jnb [error] Health check on port 3000 has failed. Your app is not responding properly. Services exposed on ports [80, 443] will have intermittent failures until the health check passes.

2023-05-02T16:24:52.964 app[6fa4e00a] jnb [info] => Booting Puma

2023-05-02T16:24:52.964 app[6fa4e00a] jnb [info] => Rails 7.0.4 application starting in production

2023-05-02T16:24:52.964 app[6fa4e00a] jnb [info] => Run `bin/rails server --help` for more startup options

2023-05-02T16:24:54.825 app[6fa4e00a] jnb [info] [520] Puma starting in cluster mode...

2023-05-02T16:24:54.825 app[6fa4e00a] jnb [info] [520] * Puma version: 5.6.5 (ruby 3.0.0-p0) ("Birdie's Version")

2023-05-02T16:24:54.825 app[6fa4e00a] jnb [info] [520] * Min threads: 5

2023-05-02T16:24:54.826 app[6fa4e00a] jnb [info] [520] * Max threads: 5

2023-05-02T16:24:54.826 app[6fa4e00a] jnb [info] [520] * Environment: production

2023-05-02T16:24:54.826 app[6fa4e00a] jnb [info] [520] * Master PID: 520

2023-05-02T16:24:54.826 app[6fa4e00a] jnb [info] [520] * Workers: 4

2023-05-02T16:24:54.826 app[6fa4e00a] jnb [info] [520] * Restarts: (✔) hot (✖) phased

2023-05-02T16:24:54.826 app[6fa4e00a] jnb [info] [520] * Preloading application

2023-05-02T16:24:54.827 app[6fa4e00a] jnb [info] [520] * Listening on http://0.0.0.0:8080

2023-05-02T16:24:54.827 app[6fa4e00a] jnb [info] [520] Use Ctrl-C to stop

2023-05-02T16:24:54.852 app[6fa4e00a] jnb [info] [520] - Worker 0 (PID: 528) booted in 0.02s, phase: 0

2023-05-02T16:24:54.852 app[6fa4e00a] jnb [info] [520] - Worker 1 (PID: 529) booted in 0.02s, phase: 0

2023-05-02T16:24:54.852 app[6fa4e00a] jnb [info] [520] - Worker 3 (PID: 535) booted in 0.01s, phase: 0

2023-05-02T16:24:54.852 app[6fa4e00a] jnb [info] [520] - Worker 2 (PID: 532) booted in 0.01s, phase: 0

2023-05-02T16:24:56.101 app[8d667e5d] jnb [info] 2023-05-02T16:24:56.099Z pid=521 tid=32l INFO: Booted Rails 7.0.4 application in production environment

2023-05-02T16:24:56.101 app[8d667e5d] jnb [info] 2023-05-02T16:24:56.100Z pid=521 tid=32l INFO: Running in ruby 3.0.0p0 (2020-12-25 revision 95aff21468)+jemalloc [x86_64-linux]

2023-05-02T16:24:56.101 app[8d667e5d] jnb [info] 2023-05-02T16:24:56.100Z pid=521 tid=32l INFO: See LICENSE and the LGPL-3.0 for licensing details.

2023-05-02T16:24:56.101 app[8d667e5d] jnb [info] 2023-05-02T16:24:56.100Z pid=521 tid=32l INFO: Upgrade to Sidekiq Pro for more features and support: https://sidekiq.org

2023-05-02T16:24:56.101 app[8d667e5d] jnb [info] 2023-05-02T16:24:56.100Z pid=521 tid=32l INFO: Sidekiq 7.0.9 connecting to Redis with options {:size=>5, :pool_name=>"internal", :url=>"redis://default:REDACTED@fly-quickvoice-staging-redis.upstash.io"}

2023-05-02T16:24:56.148 app[8d667e5d] jnb [info] 2023-05-02T16:24:56.147Z pid=521 tid=32l INFO: Sidekiq 7.0.9 connecting to Redis with options {:size=>5, :pool_name=>"default", :url=>"redis://default:REDACTED@fly-quickvoice-staging-redis.upstash.io"}

2023-05-02T16:24:58.078 runner[ed0bcd1b] jnb [info] Shutting down virtual machine

2023-05-02T16:24:58.089 runner[1832e8b0] jnb [info] Shutting down virtual machine

2023-05-02T16:24:59.874 app[ed0bcd1b] jnb [info] Sending signal SIGINT to main child process w/ PID 520

2023-05-02T16:24:59.883 app[1832e8b0] jnb [info] 2023-05-02T16:24:59.882Z pid=520 tid=32k INFO: Shutting down

2023-05-02T16:24:59.883 app[1832e8b0] jnb [info] 2023-05-02T16:24:59.882Z pid=520 tid=32k INFO: Terminating quiet threads for default capsule

2023-05-02T16:24:59.883 app[1832e8b0] jnb [info] 2023-05-02T16:24:59.882Z pid=520 tid=11oo INFO: Scheduler exiting...

2023-05-02T16:24:59.883 app[1832e8b0] jnb [info] Sending signal SIGINT to main child process w/ PID 520

2023-05-02T16:24:59.895 app[ed0bcd1b] jnb [info] [520] - Gracefully shutting down workers...

2023-05-02T16:25:00.076 app[ed0bcd1b] jnb [info] [520] === puma shutdown: 2023-05-02 16:25:00 +0000 ===

2023-05-02T16:25:00.076 app[ed0bcd1b] jnb [info] [520] - Goodbye!

2023-05-02T16:25:00.076 app[ed0bcd1b] jnb [info] Exiting

2023-05-02T16:25:00.384 app[1832e8b0] jnb [info] 2023-05-02T16:25:00.383Z pid=520 tid=11pc INFO: Pausing to allow jobs to finish...

2023-05-02T16:25:00.911 app[1832e8b0] jnb [info] 2023-05-02T16:25:00.909Z pid=520 tid=32k INFO: Bye!

2023-05-02T16:25:00.930 app[ed0bcd1b] jnb [info] Starting clean up.

2023-05-02T16:25:01.757 app[1832e8b0] jnb [info] Starting clean up.
4

Make sure that [env] PORT and [[services]] internal_port match. Feel free to change them both to 3000 or 8080 or any other free port.

1 Like
5

Thank you! Making the ports both 8080 seems to have fixed it. I don’t know if my toml file was being cached somewhere, but using -a app_name didn’t work. I had to use -c path/to/file for it to work.

One more question. This is what my dashboad looks like now. I still have an “app” process. Is this just a glitch or is there still an app process that is running somewhere? If I run fly status in the console, then I only see 2 processes, the worker and web

Screenshot 2023-05-02 at 20.52.41
Screenshot 2023-05-02 at 20.52.41897×199 11.2 KB

6

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.