How to reuse volume in different app?

calamari · August 29, 2023, 11:28am

Heyho.

Since email support is not reacting and my other question is not answered either, I just gave up to fix our production db cluster (that got broken through nomad → machines upgrade). And now I just created a new pg cluster. But I need the data from the old cluster, I though I could do

fly volumes create pg_data --snapshot-id vs_poGM1Ppxxxxxxxx -a xxx-prod-db2 -s 10

to copy that volume, wihch is in app xxx-prod-db and recreate a volumd in xxx-prod-db2, but that does not work, throwing:

Error: failed creating volume: failed to create volume: EOF

What is the best way for getting the data and putting it into my new cluster?

Thanks for any advice

lubien · August 29, 2023, 11:44am

Hi @calamari sorry to hear this EOF happened. I couldn’t find the support email you sent us about this though I could see 3 others already replied.

Would you be able to run LOG_LEVEL=debug fly volumes create pg_data --snapshot-id vs_poGM1Ppxxxxxxxx -a xxx-prod-db2 -s 10 and paste the output (filtering sensitive data if needed) here?

The EOF appears to be coming from one of our APIs.

calamari · August 29, 2023, 11:49am

I did not ask for this, the mail I sent about this had the title: " URGENT: Prod DB Down because of v2 Migration" and was not answered :-/

the debug log is not more useful, I think, here it is:

$  LOG_LEVEL=debug fly volumes create pg_data \
  --snapshot-id vs_poGM1Pplq3KvMIey \
  -a xxx-prod-db2 -s 10
DEBUG Loaded flyctl config from/home/calamari/.fly/config.yml
DEBUG determined hostname: "the-frame"
DEBUG determined working directory: "/home/calamari/work/xxx/fly-config/xxx-prod-db"
DEBUG determined user home directory: "/home/calamari"
DEBUG determined config directory: "/home/calamari/.fly"
DEBUG ensured config directory exists.
DEBUG ensured config directory perms.
DEBUG cache loaded.
DEBUG config initialized.
DEBUG skipped querying for new release
DEBUG client initialized.
DEBUG app config loaded from /home/calamari/work/xxx/fly-config/xxx-prod-db/fly.toml
DEBUG --> POST https://api.fly.io/graphql

DEBUG {
  "query": "query ($appName: String!) { appbasic:app(name: $appName) { id name platformVersion organization { id slug paidPlan } } }",
  "variables": {
    "appName": "xxx-prod-db"
  }
}


DEBUG {}
DEBUG <-- 200 https://api.fly.io/graphql (231.39ms)

DEBUG {
  "data": {
    "appbasic": {
      "id": "xxx-prod-db",
      "name": "xxx-prod-db",
      "platformVersion": "machines",
      "organization": {
        "id": "KbmZpR4wDvml1ImDm1w67zLw6QuK1e",
        "slug": "personal",
        "paidPlan": true
      }
    }
  }
}

DEBUG --> GET https://api.machines.dev/v1/apps/xxx-prod-db2/volumes

DEBUG Starting task manager
DEBUG Config has metrics token

DEBUG --> POST https://api.fly.io/graphql

DEBUG {
  "query": "query { platform { requestRegion regions { name code latitude longitude gatewayAvailable requiresPaidPlan } } }",
  "variables": null
}


DEBUG {}
DEBUG --> POST https://api.fly.io/graphql

DEBUG {
  "query": "query ($appName: String!) { appbasic:app(name: $appName) { id name platformVersion organization { id slug paidPlan } } }",
  "variables": {
    "appName": "xxx-prod-db2"
  }
}


DEBUG {}
DEBUG <-- 200 https://api.fly.io/graphql (132.98ms)

DEBUG {
  "data": {
    "platform": {
      "requestRegion": "fra",
      "regions": [
        {
          "name": "Amsterdam, Netherlands",
          "code": "ams",
          "latitude": 52.374342,
          "longitude": 4.895439,
          "gatewayAvailable": true,
          "requiresPaidPlan": false
        },
        [...]
        {
          "name": "Montreal, Canada",
          "code": "yul",
          "latitude": 45.48647,
          "longitude": -73.75549,
          "gatewayAvailable": false,
          "requiresPaidPlan": false
        },
        {
          "name": "Toronto, Canada",
          "code": "yyz",
          "latitude": 43.644632,
          "longitude": -79.384228,
          "gatewayAvailable": true,
          "requiresPaidPlan": false
        }
      ]
    }
  }
}

DEBUG <-- 200 https://api.fly.io/graphql (148.18ms)

DEBUG {
  "data": {
    "appbasic": {
      "id": "xxx-prod-db2",
      "name": "xxx-prod-db2",
      "platformVersion": "machines",
      "organization": {
        "id": "KbmZpR4wDvml1ImDm1w67zLw6QuK1e",
        "slug": "personal",
        "paidPlan": true
      }
    }
  }
}

DEBUG <-- 200 https://api.machines.dev/v1/apps/xxx-prod-db2/volumes (359.47ms)

DEBUG [
  {
    "id": "vol_6vjz3q9061xxxxxx",
    "name": "pg_data",
    "state": "created",
    "size_gb": 40,
    "region": "fra",
    "zone": "50af",
    "encrypted": true,
    "attached_machine_id": "28650e0fe56578",
    "attached_alloc_id": null,
    "created_at": "2023-08-29T11:14:39.349Z",
    "blocks": 10242360,
    "block_size": 4096,
    "blocks_free": 10218237,
    "blocks_avail": 9689905,
    "fstype": "ext4"
  },
  {
    "id": "vol_149qx9pyejxxxxxx",
    "name": "pg_data",
    "state": "created",
    "size_gb": 40,
    "region": "fra",
    "zone": "82f3",
    "encrypted": true,
    "attached_machine_id": "d8d9996ae9e708",
    "attached_alloc_id": null,
    "created_at": "2023-08-29T11:15:02.872Z",
    "blocks": 10242360,
    "block_size": 4096,
    "blocks_free": 10218236,
    "blocks_avail": 9689904,
    "fstype": "ext4"
  },
  {
    "id": "vol_p4m56272mzxxxxxx",
    "name": "pg_data",
    "state": "created",
    "size_gb": 40,
    "region": "fra",
    "zone": "74e2",
    "encrypted": true,
    "attached_machine_id": "e28659eda62968",
    "attached_alloc_id": null,
    "created_at": "2023-08-29T11:15:27.04Z",
    "blocks": 10239288,
    "block_size": 4096,
    "blocks_free": 10223356,
    "blocks_avail": 9695177,
    "fstype": "ext4"
  }
]


? Select region: Frankfurt, Germany (fra)
DEBUG --> POST https://api.machines.dev/v1/apps/xxx-prod-db2/volumes

DEBUG {
  "name": "pg_data",
  "region": "fra",
  "size_gb": 10,
  "encrypted": true,
  "require_unique_zone": true,
  "machines_only": null,
  "snapshot_id": "vs_poGM1Pplq3KvMIey",
  "source_volume_id": null,
  "compute": null
}

DEBUG {0xc000d862a0}
DEBUG <-- 200 https://api.machines.dev/v1/apps/xxx-prod-db2/volumes (393.92ms)

DEBUG 
DEBUG Task manager done
Error: failed creating volume: failed to create volume: EOF

calamari · August 29, 2023, 11:51am

alternatively, I would also be happy if someone can explain me, how I create a new pg cluster with those volumes. Since I destroyed all broken machines in my prod-db cluster, there is nothing to clone and there seems to be no fly pg create command (that is creating a machine and not a whole cluster)

lubien · August 29, 2023, 12:20pm

Thanks for pasting the output. Indeed, we’ll have to take a look on that EOF.

In the meantime: you can use fly pg create --snapshot-id [string] to create a cluster from a snapshot.

fly pg create --help
Create a new PostgreSQL cluster

Usage:
  flyctl postgres create [flags]

Flags:
      --autostart                  Automatically start a stopped Postgres app when a network request is received
      --consul-url string          Opt into using an existing consul as the backend store by specifying the target consul url.
      --detach                     Return immediately instead of monitoring deployment progress
      --flex                       Create a postgres cluster that's managed by Repmgr (default true)
      --fork-from string           Specify a source Postgres application to fork from. Format: <app-name> or <app-name>:<volume-id>
  -h, --help                       help for create
      --image-ref string           Specify a non-default base image for the Postgres app
      --initial-cluster-size int   Initial cluster size
  -n, --name string                The name of your Postgres app
  -o, --org string                 The target Fly organization
  -p, --password string            The superuser password. The password will be generated for you if you leave this blank
  -r, --region string              The target region (see 'flyctl platform regions')
      --snapshot-id string         Creates the volume with the contents of the snapshot
      --stolon                     Create a postgres cluster that's managed by Stolon
      --vm-size string             the size of the VM
      --volume-size int            The volume size in GB

Global Flags:
  -t, --access-token string   Fly API Access Token
      --debug                 Print additional logs and traces
      --verbose               Verbose output

calamari · August 29, 2023, 12:27pm

Indeed I tried that, but get the same error:

postgres create --snapshot-id vs_poGM1Pplq3KvMIey
? Choose an app name (leave blank to generate one): xxx-prod-db3
? Select Organization: Georg T (personal)
? Select region: Frankfurt, Germany (fra)
? Select configuration: Production (High Availability) - 3 nodes, 2x shared CPUs, 4GB RAM, 40GB disk
Creating postgres cluster in organization personal
Creating app...
Setting secrets on app nioomi-prod-db3...
Restoring 1 of 3 machines with image flyio/postgres-flex:15.3@sha256:02be8dc60aea4fd67587597a44af32af477df92c1e20bd050af2b3bf254fd88e
Error: failed to create volume: failed to create volume: EOF

calamari · August 29, 2023, 12:58pm

Sorry, if I am coming across like I am pleading, but I am , but I really need a way to get to that data and at least to create a new DB cluster with our data.

Is there another way to access our volume/snapshot?

calamari · August 30, 2023, 9:12am

In the end I spent a considerable amount of time basically doing manually what that fly volumes --fork-from script is doing.
In case anyone gets into the same problem, here is what I (roughly) did:

Create a node.js machine in the old and broken DB cluster (can be anything that just runs the machine and gets health checks green)
SSH into that machine to copy the /data folder (via scp) to local machine
Create a local Postgres container and let it use that data dir (here was How To Move a PostgreSQL Data Directory to a New Location on Ubuntu 16.04 | DigitalOcean very helpful)
Use pg_dump to get the data
Create a brand new DB cluster
Use pg_restore with the dumped data to push that into new DB cluster

shaun · August 30, 2023, 2:37pm

@calamari Does fly pg create --name <new-app> --fork-from <source-app> not work for you?

system · September 6, 2023, 2:37pm

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Failing to create a new volume Questions / Help	2	733	December 15, 2022
Error Can't create volume, application is already using 1 of 1 zones in ewr help-me-help-you	2	445	July 1, 2021
flyctl postgres create - Error Can't create volume, application is already using 1 of 1 zones in iad Build debugging	5	622	March 18, 2021
`pg create --fork-from` has stopped working on database, both locally and on CI/CD Build debugging help-me-help-you , postgres , machines	12	499	August 31, 2023
Error Can't create volume, application is already using 1 of 1 zones in fra Questions / Help	4	787	August 26, 2021

How to reuse volume in different app?

Related topics