Postgres database down after image update to 14.6

Questions / Help
1

Hello!

I was just following the advice here to update the image of our app’s Postgres instance, in order to update some configuration settings using stolonctl.

I ran flyctl image update --app <app_name> to update the image on our staging server, which worked without issue. Running the same on the production database has brought it down.

It looks like the production image has updated from 14.2 to 14.6, but is no longer responsive. The logs are repetitively showing the following:

ERRO[0255] Error opening connection to database (postgresql://...): dial tcp [...]:5433: connect: connection refused source="postgres_exporter.go:1658"

app[324551e6] ord [info] checking stolon status

ERROR cmd/keeper.go:719 cannot get configured pg parameters {"error": "dial unix /tmp/.s.PGSQL.5433: connect: no such file or directory"}

WARN cmd/sentinel.go:276 no keeper info available {"db": "338c1aa7", "keeper": "9ad9175e2"}

ERROR cmd/sentinel.go:1018 no eligible masters

Anyone know what might be going on, or how to get a functional database back up and running?

2

Hey @jgnieuwhof ,

So it looks like you have 3 volumes tied to your app, but your app is scaled to 1. When you issue a deploy, Nomad will choose an available volume at random, so you want to make sure to keep your volume count in sync with your app count.

I’d recommend scaling your app up to 3 to identify which volume is holding your primary, then work to delete the volumes you no longer need and scale back down.

3

Thanks for getting back so quick!

Increasing scale to 3 has brought the database back to a responsive state.

There’s still some errors there, the root cause might be:

pg_basebackup: error: could not write to file "/data/postgres/base/16386/17177": No space left on device

Does this mean I need to increase the postgres volume size? Or are there logs to clear or something?

4

No problem.

One of your volumes is only 1GB, so that could very well be the reason you’re seeing that issue.

1 Like
5

That’s done it, thank you!