I’ve been running a machine for quite some time. Got an email the other day about “Some of your apps in AMS are on a host which has suffered a hardware failure and will be down for an extended period.” Then a couple of days ago I got this new email that it won’t recover with a link how to get up again and how to reuse a volume by creating a new volume based on snapshots from old volume [image] Troubleshoot apps when a host is unavailable Documentation and guides from the team at Fly.io. So I try to follow “Apps with one Machine and an attached volume” and it say to fly volumes list fly volumes snapshots list fly volumes create --snapshot-id -s fly scale count 1 So I did all that and at step 3 it creates a volume and at step 4 it deploys alright but when deploying it also creates a new volume rather than using the one I created in step 3 How can I force my app to use the volume I create from a snapshot?

Using the find command I can see matches which are specific to the dataset! It seems that data is there :partying_face: Could our issue be related to the fact that we use the wrong --image-ref ? The first time I created a machine I tried without the --image-ref and it would not start, but then later I had to guess the value…

[image] Tommaso: Could our issue be related to the fact that we use the wrong --image-ref ? Maybe… The cat PG_VERSION above should tell you the major version, at least. Did you definitely create it with fly pg create originally, i.e., last summer?

[image] mayailurus: Did you definitely create it with fly pg create originally, i.e., last summer? Yes, it was exactly that day. If I try cat PG_VERSION I get 15 which does not seem to be one of the versions available https://hub.docker.com/r/flyio/postgres/tags

Same on my side, data is there, there is no version 15 on that repo but I noticed that new instances of pg are using postgres-flex which is found here All versions are here, I will try to restore using this flex version.

[image] Tommaso: Could our issue be related to the fact that we use the wrong --image-ref ? The first time I created a machine I tried without the --image-ref and it would not start, but then later I had to guess the value… I had the same thinking for this one, and also faced connection issues without --image-ref. But now no luck with the flex version flyio/postgres-flex:15, I’m getting this for version 15: WARNING: database "postgres" has a collation version mismatch DETAIL: The database was created using collation version 2.31, but the operating system provides version 2.36. HINT: Rebuild all objects in this database that use the default collation and run ALTER DATABASE postgres REFRESH COLLATION VERSION, or build PostgreSQL with the right library version.

That might actually be progress, though, since this new error is a known, fixable problem: [image]postgres-flex: database "postgres" has a collation version mismatch Just recently a new release of postgres-flex was published. Because the Debian version of the base image has been updated, you may see a message like this: database [DATABASE NAME] has a collation version mismatch. There are a few things you need to do here, but in many cases you should be able to resolve this in the following steps:

Yes but the issue is that we have restored data, so the pg service it’s not starting at all on machine start so I can not connect to the database to run pg commands described in the issue above. FATAL: database "repmgr" does not exist failed post-init: failed to enable repmgr: failed to create repmgr database: ERROR: template database "template1" has a collation version mismatch (SQLSTATE XX000)

Hm… Maybe temporarily try an earlier 15.x release image instead, just so you definitely have your data back. It looks like another user might have had luck with flyio/postgres-flex:15.1… https://community.fly.io/t/cant-recreate-pg-from-snapshot-the-database-was-created-using-collation-version-2-31-but-the-operating-system-provides-version-2-36/19480

THAT IS IT! @mayailurus THANKS a lot for your assistance. Probably it will work for you too @Tommaso So for everyone who lands here with the similar issue this is what worked for me at the end: fly postgres create --snapshot-id YOUR_SNAPSHOT_ID --image-ref flyio/postgres-flex:15.1

Resolved for me too! Thank you so much @mayailurus for the support!

Reuse old volume in new machine after "irrepairable hardware damage"

General

mayailurus August 11, 2024, 6:59pm 29

That might actually be progress, though, since this new error is a known, fixable problem: