Increase concurrency limits on Postgres app deployment

We’ve deployed a Postgres app via the CLI but are hating this issue: Connections hard limit when limit is not configured?.

How do we update the config of the app itself? We have no TOML since we used the CLI Postgres sub-command.

Hi @jasonkuhrt !

Kindly check this page for updating your Postgres configuration through the CLI.

Hi @kathrynannetan we tried that but it doesn’t appear to care about app config changes:

❯ f postgres config update -a pdp-preview-db -c ./fly.toml 
Error no changes were specified

In the fly.toml we have:

app = "pdp-preview-db"

[[services]]
  internal_port = 8080
  protocol = "tcp"

  [services.concurrency]
    hard_limit = 150
    soft_limit = 100

  [[services.ports]]
    force_https = true
    handlers = ["http"]
    port = 80

  [[services.ports]]
    handlers = ["tls", "http"]
    port = 443

We need the services.concurrency to be updated.

I see you are trying to update your configuration’s concurrency limits through a fly.toml file, but have you used the settings option instead?

fly postgres config update --max-connections 500 --app <pg-app-name>

As specified in the configurations list, max-connections sets the max concurrent connections

How is that related though? We’re not hitting the connection limit of300. We’re hitting the concurrency limit of 25. Error message example:

As you can see “Instance reached connections hard limit of 25”.

So IIUC this has nothing to do with Postgres configuration of max connections.

Oh I see! the app itself and not Postgres configuration then

Yes, and I don’t know how to change the app configuration of a Postgres app deployed through your flyctl postgres abstraction.

How did you add the proxy service? The hard limits are part of that config.

This might be a helpful link to read: instructions on how to apply fly.toml changes to Postgres created with our CLI and see if it works

@kathrynannetan ok we can check that out thanks. Will report back.

@kurt I’m not sure what you mean by “you add the proxy service”. All we did is use flyctl postgres create ... etc. commands.

I think you would have had to run fly ips allocate and then configured your postgres with a service to get those log messages. We don’t default to enabling an IP + proxy for PG databases. I’m trying to figure out how your pg got configured that way!

@kuhrt sorry yes, remembering now that we did configure public access to the database. But do not remember how we did that. We used the docs for sure, since we’re not experts. So I’m pretty sure we would have used this External Connections · Fly Docs.

So we’ve done this now:

# fly.toml file generated for pdp-preview-db on 2023-01-18T10:53:32-05:00

app = "pdp-preview-db"
kill_signal = "SIGINT"
kill_timeout = 5
processes = []

[env]

[experimental]
  auto_rollback = true

[[services]]
  http_checks = []
  internal_port = 8080
  processes = ["app"]
  protocol = "tcp"
  script_checks = []
  [services.concurrency]
    hard_limit = 100
    soft_limit = 75
    type = "connections"

  [[services.ports]]
    force_https = true
    handlers = ["http"]
    port = 80

  [[services.ports]]
    handlers = ["tls", "http"]
    port = 443

  [[services.tcp_checks]]
    grace_period = "1s"
    interval = "15s"
    restart_limit = 0
    timeout = "2s"
❯ fly deploy . --app pdp-preview-db --image flyio/postgres:14.4
==> Verifying app config
--> Verified app config
==> Building image
Searching for image 'flyio/postgres:14.4' remotely...
image found: img_wd57v5wx3k7v38o0
? This feature is highly experimental and may produce unexpected results. Proceed? Yes
Deploying with rolling strategy ✓

But the result is that concurrency is NOT updated. Why?

❯ f config show -a pdp-preview-db
{
    "env": {},
    "experimental": {
        "auto_rollback": true
    },
    "kill_signal": "SIGINT",
    "kill_timeout": 5,
    "processes": [],
    "services": [
        {
            "concurrency": {
                "hard_limit": 25,
                "soft_limit": 20,
                "type": "connections"
            },
            "http_checks": [],
            "internal_port": 8080,
            "ports": [
                {
                    "force_https": true,
                    "handlers": [
                        "http"
                    ],
                    "port": 80
                },
                {
                    "handlers": [
                        "tls",
                        "http"
                    ],
                    "port": 443
                }
            ],
            "processes": [
                "app"
            ],
            "protocol": "tcp",
            "script_checks": [],
            "tcp_checks": [
                {
                    "grace_period": "1s",
                    "interval": "15s",
                    "restart_limit": 0,
                    "timeout": "2s"
                }
            ]
        }
    ]
}

When I go to the app I see nothing about the activity I just ran:

That seems wrong, is it?

Any thoughts? This seems like it should be a dead simple thing to resolve.

We’ve moved back our preview database to AWS RDS because of this and are less confident about brining production workloads over in the future.