We’ve deployed a Postgres app via the CLI but are hating this issue: Connections hard limit when limit is not configured?.
How do we update the config of the app itself? We have no TOML since we used the CLI Postgres sub-command.
Hi @kathrynannetan we tried that but it doesn’t appear to care about app config changes:
❯ f postgres config update -a pdp-preview-db -c ./fly.toml
Error no changes were specified
In the fly.toml we have:
app = "pdp-preview-db"
[[services]]
internal_port = 8080
protocol = "tcp"
[services.concurrency]
hard_limit = 150
soft_limit = 100
[[services.ports]]
force_https = true
handlers = ["http"]
port = 80
[[services.ports]]
handlers = ["tls", "http"]
port = 443
We need the services.concurrency to be updated.
I see you are trying to update your configuration’s concurrency limits through a fly.toml file, but have you used the settings option instead?
fly postgres config update --max-connections 500 --app <pg-app-name>
As specified in the configurations list, max-connections sets the max concurrent connections
How is that related though? We’re not hitting the connection limit of300. We’re hitting the concurrency limit of 25. Error message example:
As you can see “Instance reached connections hard limit of 25”.
So IIUC this has nothing to do with Postgres configuration of max connections.
Oh I see! the app itself and not Postgres configuration then
Yes, and I don’t know how to change the app configuration of a Postgres app deployed through your flyctl postgres abstraction.
How did you add the proxy service? The hard limits are part of that config.
This might be a helpful link to read: instructions on how to apply fly.toml changes to Postgres created with our CLI and see if it works
@kathrynannetan ok we can check that out thanks. Will report back.
@kurt I’m not sure what you mean by “you add the proxy service”. All we did is use flyctl postgres create ... etc. commands.
I think you would have had to run fly ips allocate and then configured your postgres with a service to get those log messages. We don’t default to enabling an IP + proxy for PG databases. I’m trying to figure out how your pg got configured that way!
@kuhrt sorry yes, remembering now that we did configure public access to the database. But do not remember how we did that. We used the docs for sure, since we’re not experts. So I’m pretty sure we would have used this External Connections · Fly Docs.
So we’ve done this now:
# fly.toml file generated for pdp-preview-db on 2023-01-18T10:53:32-05:00
app = "pdp-preview-db"
kill_signal = "SIGINT"
kill_timeout = 5
processes = []
[env]
[experimental]
auto_rollback = true
[[services]]
http_checks = []
internal_port = 8080
processes = ["app"]
protocol = "tcp"
script_checks = []
[services.concurrency]
hard_limit = 100
soft_limit = 75
type = "connections"
[[services.ports]]
force_https = true
handlers = ["http"]
port = 80
[[services.ports]]
handlers = ["tls", "http"]
port = 443
[[services.tcp_checks]]
grace_period = "1s"
interval = "15s"
restart_limit = 0
timeout = "2s"
❯ fly deploy . --app pdp-preview-db --image flyio/postgres:14.4
==> Verifying app config
--> Verified app config
==> Building image
Searching for image 'flyio/postgres:14.4' remotely...
image found: img_wd57v5wx3k7v38o0
? This feature is highly experimental and may produce unexpected results. Proceed? Yes
Deploying with rolling strategy ✓
But the result is that concurrency is NOT updated. Why?
❯ f config show -a pdp-preview-db
{
"env": {},
"experimental": {
"auto_rollback": true
},
"kill_signal": "SIGINT",
"kill_timeout": 5,
"processes": [],
"services": [
{
"concurrency": {
"hard_limit": 25,
"soft_limit": 20,
"type": "connections"
},
"http_checks": [],
"internal_port": 8080,
"ports": [
{
"force_https": true,
"handlers": [
"http"
],
"port": 80
},
{
"handlers": [
"tls",
"http"
],
"port": 443
}
],
"processes": [
"app"
],
"protocol": "tcp",
"script_checks": [],
"tcp_checks": [
{
"grace_period": "1s",
"interval": "15s",
"restart_limit": 0,
"timeout": "2s"
}
]
}
]
}
Any thoughts? This seems like it should be a dead simple thing to resolve.
We’ve moved back our preview database to AWS RDS because of this and are less confident about brining production workloads over in the future.
I’ve just run into the same issue. I’ve increased the connection limit in the fly.toml, did the deploy, confirmed the new config, but I’m still getting dropped connections and the PG machine logs say they are dropping connections because of the old limit.
fly m status 5683d397a79258 -d | grep hard_limit
"hard_limit": 1000,
"hard_limit": 1000,
But from the logs:
2023-12-07T11:58:47Z proxy[5683d397a79258] ams [warn]Instance 5683d397a79258 reached hard limit of 25 concurrent connections. This usually indicates your app is not closing connections properly or is not closing them fast enough for the traffic levels it is handling. Scaling resources, number of instances or increasing your hard limit might help.
And our app servers keep failing to connect.
This happened after we scaled up our app servers.
Well, a fly pg restart seems to have fixed it for some reason 