Fork Yeah! New Volume Forking Feature for Postgres

Fresh Produce
1

As of flyctl version v0.0.530, Volume forks are now available for Postgres!

Use-cases

  • Use volume forking to create identical staging/test environments, troubleshoot issues, analyze data, or test database performance without impacting your production database.
  • Create PR review environments. This is especially useful when a PR contains database migrations that you don’t want to necessarily run against your production database.
  • Etc.

Getting started

To get started, specify the Postgres app you wish to fork using the --fork-from flag on provision.

Example 1:

fly pg create --name my-staging-db --fork-from my-prod-db

Example 2:

fly pg create --name my-staging-db --fork-from my-prod-db:vol_abcdefghij

You above example shows how you can target a specific volume to fork.

How it works

To fork a volume, we first create a new logical volume on the same host as the volume we wish to fork. We then take a snapshot of the target volume using LVM based snapshots. LVM snapshots are implemented using COW “copy-on-write”, making the actual snapshot process virtually instant. Next, we copy the data blocks from the snapshot directly onto the new volume. This approach yields similar results to taking an on-demand backup of the target volume and restoring it into a new application, but it’s significantly faster given we don’t have to send anything over the network. However, it’s worth noting that there are some limitations to this method, which we’ll discuss next.

Limitations

Unfortunately, there’s currently a restriction that only allows forking a volume on the same host, which means that in some cases we may not be able to fulfill fork requests if the host is at or near capacity. This sucks, but this can often be worked around by targeting a volume tied to one of your replicas. In any case, this a limitation that we hope to address soon!

Reasons you may want to fork a specific volume:

  • The host that your primary volume is residing on is at or near capacity, so you may want to manually specify a volume tied to one of your replicas.
  • You may want to fork into a separate region. This can be achieved by specifying a volume tied to one of your out-of-region replicas.

Let us know what you think!

21 Likes
2

Are you guys taking heavy inspiration from Neon? Two of the recent PostgreSQL features ­— scale to zero and volume forking — resemble Neon’s selling points.