Resize database volume size

Questions / Help
, ,
2

Hi Bashar,

The way you scale up and down to create new units in the cluster has changed a bit with the Machines (V2) platform. Here are updated instructions. Please be aware I tested this on a freshly-created database with very little data. So please please do NOT skip step 1.

  1. BACKUP YOUR DATA. Do not skip this step. There are snapshots in case things go wrong. Your backup will help if things go very wrong. You can use fly proxy 5432 -a <pgname> and then pg_dump to grab an SQL dump of your data.
  2. Make sure you did not skip step 1.
  3. Add a new 1GB volume to your Postgres app: fly volumes create pg_data --size 1 --region <your-pg-region> -a <pgname>. Make a note of the VOLUME_ID.
  4. Run fly status -a <pgname>, note which instance is the primary and how many instances are running. Grab the ID (first column) of an existing machine.
  5. Add one more instance as a clone of that machine, attaching it to the previously-created volume: fly machine clone EXISTING_MACHINE_ID --attach-volume VOLUME_ID -a <pgname>. Note the machine ID of this new SMALL_MACHINE_ID.
  6. Clone that machine two more times: fly machines clone SMALL_MACHINE_ID -a <pgname>.
  7. Wait a few minutes, then run fly status to make sure the new instances are in replica state and happy/healthy. Confirm using fly volumes list and fly machines list that the new machines have the small volumes.
  8. Stop the old old replicas with large volumes. The easiest way is to run fly volumes list -a <pgname>, check the ATTACHED VM column for machines with the large volumes, ensure they are not the primary, and fly machine stop those. It’s OK to stop them almost at the same time. Once done, wait for the cluster status to be stable and for the fly logs to be free of errors.
  9. Stop the primary, fly machine stop PRIMARY_ID.
  10. Wait a few minutes for the three remaining replicas to realize the primary is gone, chat among themselves and elect a new primary node.
  11. Once your three small nodes are the only remaining ones, and one of them has been designated the primary, it’s safe to fly machine destroy for the old replicas and the old primary.
  12. In fly volumes list, identify and fly volumes destroy all the old, now-unattached, large volumes.
  13. If anything goes wrong, your cluster is down, or in read-only mode and it’s not obvious how to recover it: Luckily, you have a backup! Nuke the cluster, create a new one with the desired size, and load your data in it. Or you can look into how to restore from a snapshot - but snapshots are taken daily and your backup is only 20 minutes old, isn’t it? :slight_smile:

Let me know if this works!

Regards,

  • Daniel
7 Likes