For quite some time, I had been happily using the --fork-from flag to automatically provision a staging copy of my database for review apps in an automated GitHub Actions context. Unfortunately, about a month ago, some careless scripting on my end led to said workflow destroying all of my apps, the database included. Luckily, I had retained a few of my own dumps so only a less-than-disastrous amount of data was lost, and I thought that simply recreating the app and re-attaching the relevant consumer apps would mean that all would continue just fine.
Unfortunately, since on or about that day, the workflow has been a lot more delicate than I’d like for reasons seemingly out of my control, and since the beginning of August I’ve been completely unable to fork from the production database for any reason or in any context. The most common error message I get is:
Error: failed to fork volume: failed to create volume: source volume not found
Given that I started noticing this error around the time of other platform-related incidents, I originally thought I’d simply have to wait out the storm and see if anything changed. However, I’ve had no luck since; not with changing the FLY_REGISTRY_HOST flag nor any combination of machines and volumes and snapshots and regions. The only thing I’ve admittedly yet to try is tearing down the prod DB app and starting anew, but I’d really prefer to avoid downtime like that, especially if I have no guarantee that I’ll be able to get things running again.
I’ve searched these forums endlessly looking for any similar errors, but haven’t been successful so far. I’m a little nervous about this Discourse’s 7-day necro window, so hopefully I can get some support before the thread closes. I’m of course happy to supply any logs, configs, etc. that are necessary to assist.
Hey @shaun, happily. Is there a more private method by which I can do that? I don’t have a strong business need to keep the app name private, though I might send it here and then remove my comment once you’ve got it just in case.
Hi, I’m having a little trouble doing that. I see a bunch of notifications coming in that seem to indicate I have some new features available, but the “Private Message” button yet eludes me. Thanks for your help!
I was able to reproduce this, but what I thought was a problem in the code, was really a feature. So at this point, the error seems to happen in the API, which is beyond my current reach.
I wonder why it doesn’t find the source volume.
@oliver-daniel you could try running the CLI with LOG_LEVEL=debug in front of the command, like LOG_LEVEL=debug fly postgres create ... to see the API requests and responses. But other than that, hope the team helps you solve it.
I also tried with a couple more volume IDs from different apps, but same error.
Even though when I use the GET endpoint of the Fly Machines API, it finds the volume with that same volume ID.
Hey Shaun, I’m happy to say that it does seem that the forking issue has been resolved. I’m currently wrangling some other Fly-related issues, but I’m not sure how much those are going to be related to this specific API issue.
Thanks for your help!