`pg create --fork-from` has stopped working on database, both locally and on CI/CD


For quite some time, I had been happily using the --fork-from flag to automatically provision a staging copy of my database for review apps in an automated GitHub Actions context. Unfortunately, about a month ago, some careless scripting on my end led to said workflow destroying all of my apps, the database included. Luckily, I had retained a few of my own dumps so only a less-than-disastrous amount of data was lost, and I thought that simply recreating the app and re-attaching the relevant consumer apps would mean that all would continue just fine.

Unfortunately, since on or about that day, the workflow has been a lot more delicate than I’d like for reasons seemingly out of my control, and since the beginning of August I’ve been completely unable to fork from the production database for any reason or in any context. The most common error message I get is:

Error: failed to fork volume: failed to create volume: source volume not found

Given that I started noticing this error around the time of other platform-related incidents, I originally thought I’d simply have to wait out the storm and see if anything changed. However, I’ve had no luck since; not with changing the FLY_REGISTRY_HOST flag nor any combination of machines and volumes and snapshots and regions. The only thing I’ve admittedly yet to try is tearing down the prod DB app and starting anew, but I’d really prefer to avoid downtime like that, especially if I have no guarantee that I’ll be able to get things running again.

I’ve searched these forums endlessly looking for any similar errors, but haven’t been successful so far. I’m a little nervous about this Discourse’s 7-day necro window, so hopefully I can get some support before the thread closes. I’m of course happy to supply any logs, configs, etc. that are necessary to assist.

There have been some changes in that part of the CLI on use flaps api for volumes #2631.

It seems to me that the --fork-from value isn’t being passed properly after the changes made in flypg/launcher.go.

My diagnosis may not be correct, but if that is the issue, I think it can be fixed quite easily by the owning team.

1 Like

@oliver-daniel Mind sending me the name of your app?

Hey @shaun, happily. Is there a more private method by which I can do that? I don’t have a strong business need to keep the app name private, though I might send it here and then remove my comment once you’ve got it just in case.

Feel free to send it as a DM.

Hi, I’m having a little trouble doing that. I see a bunch of notifications coming in that seem to indicate I have some new features available, but the “Private Message” button yet eludes me. :sweat_smile: Thanks for your help!

Awesome, sent.

Continuing the conversation here :slight_smile:

Hey there,

When you specify --fork-from could you try using the following format:

--fork-from <app-name>:<volume-id>

I should have specified that I have tried this. With each volume shown under fly vol list and the following command, I get the same output:

fly apps destroy <new-app-name> --yes; fly postgres create --name <new-app-name> --fork-from <app-name>:<volume-id> --vm-size shared-cpu-1x

>> Creating postgres cluster in organization personal
Creating app...
Setting secrets on app <new-app-name>...
Provisioning 1 of 1 machines with image registry-1.docker.io/flyio/postgres-flex:15.3@sha256:<digest>
Error: failed to fork volume: failed to create volume: source volume not found

I was able to reproduce this, but what I thought was a problem in the code, was really a feature. So at this point, the error seems to happen in the API, which is beyond my current reach.

I wonder why it doesn’t find the source volume.

@oliver-daniel you could try running the CLI with LOG_LEVEL=debug in front of the command, like LOG_LEVEL=debug fly postgres create ... to see the API requests and responses. But other than that, hope the team helps you solve it.

Sorry I couldn’t be more helpful!

I also tried with a couple more volume IDs from different apps, but same error.
Even though when I use the GET endpoint of the Fly Machines API, it finds the volume with that same volume ID.

We discovered the issue and are working on a fix!


@oliver-daniel Mind giving the fork another go?

1 Like

Hey Shaun, I’m happy to say that it does seem that the forking issue has been resolved. I’m currently wrangling some other Fly-related issues, but I’m not sure how much those are going to be related to this specific API issue.

Thanks for the help and keeping me updated!

1 Like