postgresql down

Questions / Help
1

Hi, since today my postgresql database is down. I have not touched this setup for months and it worked reliable. I get the following logs:

2023-01-20T01:39:59.420 runner[6c6eb9c0] fra [info] Shutting down virtual machine
2023-01-20T01:39:59.806 app[6c6eb9c0] fra [info] Sending signal SIGTERM to main child process w/ PID 529
2023-01-20T01:39:59.836 app[6c6eb9c0] fra [info] Got terminated, stopping
2023-01-20T01:39:59.836 app[6c6eb9c0] fra [info] supervisor stopping
2023-01-20T01:39:59.836 app[6c6eb9c0] fra [info] exporter | Stopping interrupt...
2023-01-20T01:39:59.836 app[6c6eb9c0] fra [info] keeper | Stopping interrupt...
2023-01-20T01:39:59.836 app[6c6eb9c0] fra [info] sentinel | Stopping interrupt...
2023-01-20T01:39:59.836 app[6c6eb9c0] fra [info] proxy | Stopping interrupt...
2023-01-20T01:39:59.836 app[6c6eb9c0] fra [info] sentinel | 2023-01-20T01:39:59.807Z INFO cmd/sentinel.go:1816 stopping stolon sentinel
2023-01-20T01:39:59.836 app[6c6eb9c0] fra [info] exporter | signal: interrupt
2023-01-20T01:39:59.836 app[6c6eb9c0] fra [info] sentinel | Process exited 0
2023-01-20T01:39:59.836 app[6c6eb9c0] fra [info] proxy | [NOTICE] 019/013959 (546) : haproxy version is 2.2.9-2+deb11u2
2023-01-20T01:39:59.836 app[6c6eb9c0] fra [info] proxy | [NOTICE] 019/013959 (546) : path to executable is /usr/sbin/haproxy
2023-01-20T01:39:59.836 app[6c6eb9c0] fra [info] proxy | [ALERT] 019/013959 (546) : Current worker #1 (570) exited with code 130 (Interrupt)
2023-01-20T01:39:59.836 app[6c6eb9c0] fra [info] proxy | [WARNING] 019/013959 (546) : All workers exited. Exiting... (130)
2023-01-20T01:39:59.836 app[6c6eb9c0] fra [info] keeper | 2023-01-20 01:39:59.834 UTC [615] LOG: received fast shutdown request
2023-01-20T01:39:59.841 app[6c6eb9c0] fra [info] keeper | 2023-01-20T01:39:59.837Z INFO postgresql/postgresql.go:384 stopping database
2023-01-20T01:39:59.842 app[6c6eb9c0] fra [info] keeper | 2023-01-20 01:39:59.841 UTC [615] LOG: aborting any active transactions
2023-01-20T01:39:59.847 app[6c6eb9c0] fra [info] keeper | 2023-01-20 01:39:59.845 UTC [615] LOG: background worker "logical replication launcher" (PID 622) exited with exit code 1
2023-01-20T01:39:59.847 app[6c6eb9c0] fra [info] keeper | 2023-01-20 01:39:59.846 UTC [617] LOG: shutting down
2023-01-20T01:39:59.848 app[6c6eb9c0] fra [info] proxy | exit status 130
2023-01-20T01:39:59.864 app[6c6eb9c0] fra [info] keeper | waiting for server to shut down....2023-01-20 01:39:59.863 UTC [615] LOG: database system is shut down
2023-01-20T01:39:59.949 app[6c6eb9c0] fra [info] keeper | done
2023-01-20T01:39:59.949 app[6c6eb9c0] fra [info] keeper | server stopped
2023-01-20T01:39:59.955 app[6c6eb9c0] fra [info] keeper | Process exited 0
2023-01-20T01:40:00.808 app[6c6eb9c0] fra [info] Starting clean up.
2023-01-20T01:40:00.824 app[6c6eb9c0] fra [info] Umounting /dev/vdc from /data

And the fly dashboard says the database is pending

I tried to run fly postgres restart ... but all i get is “Error no active leader found”.

Would appreciate some help here

2

Hey,

I ran into this recently too.

You can check what is the problem on your postgres instance(s).
fly checks list -a staging-betafi-postgres

Then fix it, like extending the volume etc.

Then restart the machine that the leader is supposed to be on: flyctl machine restart · Fly Docs

I hope this helps!