Volume vol_45lmj7xokynn0xjr cannot mount (IAD) - 502 error on snapshot restore

Hello Fly.io Support team,

I have an application: accounts-manager-ansgt in region IAD.

My volume (vol_45lmj7xokynn0xjr) cannot be mounted — I get:
“insufficient resources to create new machine with existing volume”.

I also tried restoring from snapshot:
vs_A3wzRPwneqVsga8LPyVMZ8g
but I get:
“failed to create volume: 502”.

I cannot create machines, cannot mount the volume, and cannot restore snapshots in region IAD. The data stored in this volume contains a production SQLite database that I need to recover.

Could you please:

  1. Move this volume to a healthy host in IAD,
  2. Or mount it in a temporary machine,
  3. Or export the /data directory (SQLite DB) and provide me with a downloadable archive?

Request IDs from recent attempts:

  • 01K9ZUSCTH7K4G564WEB61J5CWC-iad
  • 01K9Z4DB1C9FSRS9VWA0VN24ZD-iad

Thank you,
David Cuadrado

Hi @david-cuadrado,

I’ll look into that snapshot restore failure.

In the meantime, you should be able to fork the volume onto another host in IAD.

This snapshot ID seems to be invalid and that’s why the snapshot restore was failing. Our error handling for this is bad - you should’ve been told something other than “502”.

Where did that snapshot ID come from? flyctl? Can you confirm it’s not truncated?

We’ve shipped a fix for this. An invalid snapshot ID will return a “not found” error now.

Hi,

Thanks for your help.

I can confirm that this is the correct volume:

  • Volume ID: vs_A3wzRPwneqVsga8LPvYMZ8g

It’s the one that contains my SQLite database for the app.

The problem is that the region where this volume lives no longer has enough capacity. When I try to update or start the machine that uses this volume, I keep getting errors like:

aborted: could not reserve resource for machine: insufficient memory available to fulfill request

Because of this I currently cannot:

  • Start the machine

  • SSH into it

  • Copy or download the database file

  • Deploy a new machine attached to this volume

Even after lowering the machine’s memory to 128MB/256MB in fly machines update, the same “insufficient memory available to fulfill request” error persists in that region, so I’m completely blocked from accessing the data on this volume.

Could you please help me with one of the following?

  1. Temporarily provision enough capacity in that region so I can start a machine attached to vs_A3wzRPwneqVsga8LPvYMZ8g and copy the database, or

  2. Move/clone this volume to a region with capacity so I can attach it to a new machine and recover the data.

My goal is simply to recover the data from that volume without losing the database.

Thanks in advance for your help.

This is the error: failed to update machine 0805095c103478: failed to update VM 0805095c103478: aborted: could not reserve resource for machine: insufficient memory available to fulfill request (Request ID: 01KA0CQK9AM65EB9EDFMPYD2KV-iad) (Trace ID: 2b6d9244a3808b1840fa51489e74f0cd)

Hi, your LLM got it wrong, this is a snapshot ID, not a volume ID.

Looks like another LLM misdiagnostic. What’s happening here is that the host (read: server) on which your volume exists doesn’t have enough capacity to start a machine, even if there’s capacity in the region as a whole. But creating a machine in another host in the same region won’t help because the volume is in the specific host it was created on: volumes are not networked and can only be mounted by a machine on the same host.

You can do this yourself, but you need to use the volume identifier and not the snapshot identifier. The volume identifier starts with vol_xxxxxxxxxxxx.

fly volume fork vol_xxxxxxx --region whatever will create a copy/fork of the volume with its data as it exists in the original, in the given region (it can be the same region or another one, if you’d feel more at ease using a different one). Check the output for the new volume id (vol_yyyy)

fly machine clone original_machine_id --attach-volume vol_yyyy will create a new machine, identical to your old one, attached to the created/forked volume.

You can then fly ssh console --select into the new machine to recover your data as needed.

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.