"could not find a good candidate within 90 attempts at load balancing" though app can be opened via SSH

[Feodor]

Hello,

Out of nowhere the app, that was working for months, has stopped working and I’m seeing:

could not find a good candidate within 90 attempts at load balancing. last error: no known healthy instances found for route tcp/443. (hint: is your app shut down? is there an ongoing deployment with a volume or are you using the 'immediate' strategy? have your app's instances all reached their hard limit?)

I redeployed the app, but that error remains and the app does not respond on 80/443.

I logged with SSH:

fly ssh console

and I’m able to retrieve index.html page without any issues with the following command:

wget -O- http://0.0.0.0:8000

what could be wrong?

Thanks,
Feodor

[pavel]

Hey @Feodor

Looks like you app has reached the configured concurrency limit.You should be able to see this message in the logs:

Instance <machine> reached hard limit of 25 concurrent connections. This usually indicates your app is not closing connections properly or is not closing them fast enough for the traffic levels it is handling. Scaling resources, number of instances or increasing your hard limit might help.

If you app can support more than 25 concurrent connections, you can set the correct limits as described here - Fly Launch configuration (fly.toml) · Fly Docs

Note that each active websocket connection (even when no data is transferred over it) counts towards the limit.

and I’m able to retrieve index.html page without any issues with the following command:

Yeah, this bypasses fly-proxy, so concurrency limits do not apply.

[Feodor]

Looks like that was it. Thank you for the speedy reply!