Load balancing that does not distribute traffic

Questions / Help
,
1

Hi,

As you can see from the screenshot, there are many machines available in ewr but for some reason most requests hit 2 machines and causes timeout.

Screenshot 2025-04-10 at 10.23.16 AM
Screenshot 2025-04-10 at 10.23.16 AM1251×829 94.3 KB

Here it fly.toml file.:

app = "my-api"
primary_region = "ewr"
kill_signal = "SIGINT"
kill_timeout = "180s"

[experimental]
  auto_rollback = true

[build]
  builder = "heroku/builder:22"

[[services]]
  protocol = "tcp"
  internal_port = 7000
  auto_stop_machines = false
  auto_start_machines = true
  min_machines_running = 5
  processes = ["app"]

  [[services.ports]]
    port = 80
    handlers = ["http"]
    force_https = true

  [[services.ports]]
    port = 443
    handlers = ["tls", "http"]
  [services.concurrency]
    type = "requests"
    soft_limit = 150

  [[services.tcp_checks]]
    interval = "15s"
    timeout = "25s"
    grace_period = "4s"
    restart_limit = 2

This app is an API.

2

Hi Onar,

Load balancing is not strictly round-robin. As long as machines are under their hard limit, the proxy is still allowed and expected to send them requests, possibly unevenly and possibly even if that causes a machine to go over its soft_limit (which is soft precisely for this reason :smiley: )

I recommend setting an explicit hard_limit; if you know your machines can handle, say, 200 concurrent connections, make that the hard limit. The proxy will honor the hard limit strictly and won’t send more connections to a machine that’s at hard_limit already. Don’t wait until the proxy piles 500 connections up on a machine causing it to choke :slight_smile: