Getting a few "instance refused connection" when rails app first loads

[shahaf]

We’re moving a rails app from Heroku to fly.io. We noticed that:

1. The app seems to restart frequently
2. When the app first starts up, we get a handful of these errors: “instance refused connection. is your app listening on 0.0.0.0:3000? make sure it is not only listening on 127.0.0.1 (hint: look at your startup logs, servers often print the address they are listening on)”

I got help from fly.io support to resolve both, so I’ll post the answers here in case it helps others…

[shahaf]

The reason the app restarts frequently is that we have the following in our fly.toml:

  auto_stop_machines = true
  auto_start_machines = true

These settings allow fly.io to automatically spin machines down when load is lower, and spin them back up when there’s a need.

The reason for the “instance refused connection” errors is that the fly.io router begins routing requests to the machine right away, before Puma / Rails gets a chance to start up. The workaround here is to use the following settings in fly.toml:

[[http_service.checks]]
  # Pick an appropriate wait period for your app, and add some buffer.
  grace_period = "10s"

More details:

Fly Launch configuration (fly.toml)

[shahaf]

For reference, I saw the following other threads that may have been about the same issue. Those threads are all closed so I can’t add to them:

Migrating existing Rails app: instance refused connection

I’m trying to deploy an existing Rails app, but I keep getting the error: instance refused connection. is your app listening on 0.0.0.0:3000? make sure it is not only listening on 127.0.0.1 (hint: look at your startup logs, servers often print the address they are listening on) I suspect this will be something simple—like a missing environment variable—but how do I see the appropriate logs? I’m using fly logs to get that error message but that doesn’t appear to show me the Rails logs. How mi…

[error]instance refused connection.

Hi! I have successfully deployed my rails app, but some times i have this errors in my logs that do slow my app. What is it and how i can fix it? otp [error]instance refused connection. is your app listening on 0.0.0.0:3000? make sure it is not only listening on 127.0.0.1 (hint: look at your startup logs, servers often print the address they are listening on) 2023-05-20T11:10:43Z proxy[148e613c715489] otp [error]instance refused connection. is your app listening on 0.0.0.0:3000? make sure it i…

Slow start with failure to connect errors

I moved my Rails app from Heroku to fly.io a few days ago and all seems well except for the startup which takes about 13 seconds to complete. Below are the logs. You will notice two refused connections at 57.41 and 57.43 and finally it will connect. The longest wait period is between 57.45 and 57.53 when it finally kicks in. My app load times in this case are views 41.6ms and ActiveRecord 14.3ms. Please request further info if needed. Any help will be most appreciated. 2024-04-12T12:44:37Z runn…

[kurt]

Wow this is great work. We can probably do better by default for Rails apps here.

[rubys]

@shahaf

The reason for the “instance refused connection” errors is that the fly.io router begins routing requests to the machine right away, before Puma / Rails gets a chance to start up.

I’m not seeing that with other Rails apps, is there anything unusual about your startup that you are aware of?

[kurt]

Quick question: when you get “instance refused connection” in the logs, are you also seeing errors from clients / browsers? The proxy will log that error while it does retries, so if all you’re seeing is logs and no client errors, you can safely ignore those.