No eligible masters and db unavailable after moving primary region and running pg-failover

I’ve moved my db primary region as per What is the correct process to change the postgres leader region? - #2 by shaun, but the new volumes seem to be flapping: running pg-failover is timing out, and I’m seeing no eligible masters. Scaling to 0 and back up, removing and re-adding volumes, and restarting VMs is having no effect.

Not sure whether this is relevant but I’m also constantly seeing this:

 exporter | ERRO[0089] Error opening connection to database (postgresql://flypgadmin:PASSWORD_REMOVED@[ipv6 address]:5433/postgres?sslmode=disable): dial tcp [ipv6 address]:5433: connect: connection refused source="postgres_exporter.go:1658"

Moving back to the original region hasn’t helped – db is still unavailable

Just got an automated email that I still have 16 volumes attached, even though only two are listed. Really hoping that means they’ve only been soft-deleted…

@shugel It looks like you may have deleted the volume that contained your clusters leader. There are three volumes that had been deleted within the last couple days that contain backups. You should be able to see them now when running fly volumes list.

Here’s some additional information on how to issue a restore:

Let me know if you have any other questions on this.

1 Like

Hi @shaun I can see them – however all the recent snapshots seem to be encrypted, meaning they seemingly can’t be used in a postgres create operation?

@shugel Ahh, got it. I just pushed out a change that removes this restriction.

Mind giving it another try?

1 Like

That’s back and healthy. I wasn’t able to attach it to my app directly as the db user already existed, so I manually set DATABASE_URL and related secrets, and DB reads & writes from the app are fine. Is there anything else special about attach that I need to be careful of here?