Rails app fails: could not find a good candidate within 90 attempts at load balancing

Hi everyone,

I have an issue that probably some folks have already faced however there are no answers to their questions that can help me.

So my question is next: I have a dev organisation with my Rails app and it works like a charm. I’ve created a new organisation for prod, and deployed the app there within the infrastructure(postgres and redis) and necessary secrets. Once I try to reach out to the URL of the app it has 502 error and on the fly side, I have the next logs.

$ fly logs -a elewhite-prod -i 9185e649c44548

Waiting for logs...

2024-04-01T07:55:02.356 runner[9185e649c44548] ams [info] Pulling container image registry.fly.io/elewhite-prod@sha256:0380414fbefcf5e01e5d406006ddfd562e7598790b47bc2f648c2f2f5be63402

2024-04-01T07:55:03.282 runner[9185e649c44548] ams [info] Successfully prepared image registry.fly.io/elewhite-prod@sha256:0380414fbefcf5e01e5d406006ddfd562e7598790b47bc2f648c2f2f5be63402 (925.981518ms)

2024-04-01T07:55:03.297 runner[9185e649c44548] ams [info] Setting up volume 'storage'

2024-04-01T07:55:03.297 runner[9185e649c44548] ams [info] Opening encrypted volume

2024-04-01T07:55:03.536 runner[9185e649c44548] ams [info] Configuring firecracker

2024-04-01T07:55:03.851 app[9185e649c44548] ams [info] [ 0.043060] PCI: Fatal: No config space access function found

2024-04-01T07:55:04.052 app[9185e649c44548] ams [info] INFO Starting init (commit: 88af1610)...

2024-04-01T07:55:04.124 app[9185e649c44548] ams [info] INFO Mounting /dev/vdb at /rails/storage w/ uid: 1000, gid: 1000 and chmod 0755

2024-04-01T07:55:04.127 app[9185e649c44548] ams [info] INFO Resized /rails/storage to 3204448256 bytes

2024-04-01T07:55:04.127 app[9185e649c44548] ams [info] INFO starting statics vsock server

2024-04-01T07:55:04.128 app[9185e649c44548] ams [info] INFO Preparing to run: `/rails/bin/docker-entrypoint ./bin/rails server` as rails

2024-04-01T07:55:04.134 app[9185e649c44548] ams [info] INFO [fly api proxy] listening at /.fly/api

2024-04-01T07:55:04.140 app[9185e649c44548] ams [info] 2024/04/01 07:55:04 listening on [fdaa:9:ea5:a7b:3e:4734:ab61:2]:22 (DNS: [fdaa::3]:53)

2024-04-01T07:55:04.153 runner[9185e649c44548] ams [info] Machine created and started in 1.8s

2024-04-01T07:55:06.414 app[9185e649c44548] ams [info] => Booting Puma

2024-04-01T07:55:06.414 app[9185e649c44548] ams [info] => Rails 7.0.6 application starting in production

2024-04-01T07:55:06.414 app[9185e649c44548] ams [info] => Run `bin/rails server --help` for more startup options

2024-04-01T07:55:06.892 app[9185e649c44548] ams [info] Puma starting in single mode...

2024-04-01T07:55:06.892 app[9185e649c44548] ams [info] * Puma version: 5.6.6 (ruby 3.2.2-p53) ("Birdie's Version")

2024-04-01T07:55:06.892 app[9185e649c44548] ams [info] * Min threads: 5

2024-04-01T07:55:06.892 app[9185e649c44548] ams [info] * Max threads: 5

2024-04-01T07:55:06.892 app[9185e649c44548] ams [info] * Environment: production

2024-04-01T07:55:06.892 app[9185e649c44548] ams [info] * PID: 323

2024-04-01T07:55:06.892 app[9185e649c44548] ams [info] * Listening on http://0.0.0.0:3000

2024-04-01T07:55:06.894 app[9185e649c44548] ams [info] Use Ctrl-C to stop

2024-04-01T07:57:13.512 proxy[9185e649c44548] fra [error] could not find a good candidate within 90 attempts at load balancing

It is hard to understand what it means and why it is in fra and not in ams.

I would very much appreciate it if someone pointed me in the direction of investigating it.

Let me know if I need to post more information about the issue.