Postgres Database Dead, No Volumes Found

Questions / Help
, ,
1

I have hosted a Swift server for a very long time without issue. Today, I learned that my server has failed, and my database is not working.

Attempts to start the Postgres instance that’s attached to the Swift server shows the below logs. When attempting to debug the issue, I ran fly volumes list -a [app name] --all and it returned none. I’d assume that there should be a volume here even if the Postgres database is not on.

2025-02-12T20:13:00.081 proxy[4d891163ae0e28] atl [info] Starting machine
2025-02-12T20:13:00.083 proxy[4d891163ae0e28] atl [error] [PM01] machines API returned an error: "machine still attempting to start"
2025-02-12T20:13:15.789 proxy[4d891163ae0e28] atl [info] Starting machine
2025-02-12T20:13:15.791 proxy[4d891163ae0e28] atl [error] [PM01] machines API returned an error: "machine still attempting to start"
2025-02-12T20:13:37.179 app[4d891163ae0e28] atl [info] repmgrd | [2025-02-12 20:13:37] [INFO] monitoring primary node "fdaa:3:611:a7b:e6:229:da6c:2" (ID: 2030208039) in normal state
2025-02-12T20:17:30.042 app[4d891163ae0e28] atl [info] monitor | Voting member(s): 1, Active: 1, Inactive: 0, Conflicts: 0
2025-02-12T20:18:37.577 app[4d891163ae0e28] atl [info] repmgrd | [2025-02-12 20:18:37] [INFO] monitoring primary node "fdaa:3:611:a7b:e6:229:da6c:2" (ID: 2030208039) in normal state
2025-02-12T20:22:30.052 app[4d891163ae0e28] atl [info] monitor | Voting member(s): 1, Active: 1, Inactive: 0, Conflicts: 0
2025-02-12T20:22:41.595 proxy[4d891163ae0e28] atl [info] Starting machine
2025-02-12T20:22:41.598 proxy[4d891163ae0e28] atl [error] [PM01] machines API returned an error: "machine still attempting to start"
2025-02-12T20:22:45.682 proxy[4d891163ae0e28] atl [info] Starting machine
2025-02-12T20:22:45.684 proxy[4d891163ae0e28] atl [error] [PM01] machines API returned an error: "machine still attempting to start"
2025-02-12T20:23:23.960 proxy[4d891163ae0e28] atl [info] Starting machine
2025-02-12T20:23:23.962 proxy[4d891163ae0e28] atl [error] [PM01] machines API returned an error: "machine still attempting to start"
2025-02-12T20:23:36.016 proxy[4d891163ae0e28] atl [info] Starting machine
2025-02-12T20:23:36.018 proxy[4d891163ae0e28] atl [error] [PM01] machines API returned an error: "machine still attempting to start"
2025-02-12T20:23:37.981 app[4d891163ae0e28] atl [info] repmgrd | [2025-02-12 20:23:37] [INFO] monitoring primary node "fdaa:3:611:a7b:e6:229:da6c:2" (ID: 2030208039) in normal state

I’m also getting errors that the machine is OOM… but the memory has not changed since I started this server. I’m unsure what could be causing these issues or where to start debugging it. I’ve upped the memory from 512, to 1024 with no success…

3

Hi… The Fly.io infrastructure shouldn’t attempt to start a Machine without its volume. I would suggest double-checking that you were using the name of the Postgres app in fly volumes list—and not the Swift app’s name.

Also, when was the last time that the database was definitely known to have been running? (From the context, it sounds like it might not have been looked at in a while…)

4

hi there! The database is used daily. I have a web scraper that’s running on the Vapor server which updates the database every morning. The scraper last ran successfully yesterday morning. Additionally, the database is used to populate my application with data, therefore every time a user enters the application, it’s used.

When I attempt to restart the Postgres database, I get an error that says Error: no active leader found.

Phew, okay, I’ve confirmed now that I’ve used the database instance rather than the Swift app name. That does show a volume.

Update: I increased my memory of my Postgres database from 512 to 1024mb and it appears the server has started.

1 Like
5

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.