I’m having a big issue right now with my production database.
I can’t scale down or scale up, the database is currently unreachable. I figured I would try to redeploy the same image, and I received this message:
Error: failed to update machine configuration for ID_HERE [app]: machine ‘ID_HERE’ requires manual intervention, it can’t be automatically replaced because its volume ‘vol_ID_HERE’ is on an unreachable host
Not quite sure what I can do. No configurations were changed in the recent or not-so-recent past. It just died.
I looked at your DB, and it appears it was created as a development postgres with only one machine. These will go down when the hardware the machine is on fails, and come back up when we recover the hardware.
Right now you have two choices:
You can wait for the host to come back up
You can restore the latest backup to a new database
Either way, if you need max uptime you should select one of the production grade postgres clusters. Production grade PG clusters are high availability, if any one of the members fails it will elect a healthy one as primary.
Your existing DB can upgrade once it becomes available again.
Yeah that was clearly my mistake when creating it. Is there a way I can upgrade it with a fly command? Or do I need to chain a bunch of stuff together?
I absolutely want to maximize uptime on this (not looking forward to tomorrow morning…)
Do we have any idea what kind of timeline we are looking at? I don’t think I can afford to go back to the latest backup as we have operations that would need to be redone in a specific order (let’s just call it a nightmare scenario on my end)
Yeah thanks – the host indeed did go down (the notification popped up maybe 2 minutes after I created this post and reached out via email)
Not much is workable, I mean if I absolutely HAVE to I can recover from a snapshot, but at this point i’d rather take my licks and have a little bit of downtime but not lose a bunch of data in the process. Hopefully the host is operational soon
As far as I’m aware, you’ll need to use individual commands.
Once the original host is back up as Kurt mentioned, you’ll need to do the following:
# get volume id
flyctl vol ls -a <pg_app_name>
# use above volume id here
flyctl vol fork <volume-id> -a <pg_app_name>
# repeat above fork command assuming you are setting up the 3 machines needed for repmgr
flyctl scale count 3 -a <pg_app_name>
If you need to increase the size of the machines, it’s easier to do it before you scale the number of machines since the scaling operation will copy the config.
# get machine's id
flyctl m ls -a <pg_app_name>
# these are just example numbers, you'll want to figure these out
flyctl m update <machine-id> --vm-cpu 2 --vm-mem 2048
If you update the memory, you’ll want to adjust the shared-buffers config to account for it. This defaults to 25% of the available memory when it was originally launched (just note this config is in 8kB units). Assuming you set it to 2048 as the above command, you would want to set it as 65536 (8kB * 65536 = 2048MB * 0.25)
flyctl pg update -a <pg_app_name> --shared-buffers 65536
