Error: no space left on device - Can not SSH, resize or backup data on volume

  • can not SSH to app id d7e4617c via fly ssh console
  • don’t know how to check size of the volume
  • can not resize the volume
  • can not backup the volume (which is most important before do something else)

Getting this:

events.js:377
2022-05-12T17:02:20Z app[9e1b7c2f] fra [info]      throw er; // Unhandled 'error' event
2022-05-12T17:02:20Z app[9e1b7c2f] fra [info]      ^
2022-05-12T17:02:20Z app[9e1b7c2f] fra [info]Error: ENOSPC: no space left on device, write
2022-05-12T17:02:20Z app[9e1b7c2f] fra [info]Emitted 'error' event on Logger instance at:
2022-05-12T17:02:20Z app[9e1b7c2f] fra [info]    at WriteStream.onStreamError (/var/lib/ghost/versions/4.38.0/node_modules/bunyan/lib/bunyan.js:627:18)
2022-05-12T17:02:20Z app[9e1b7c2f] fra [info]    at WriteStream.emit (events.js:400:28)
2022-05-12T17:02:20Z app[9e1b7c2f] fra [info]    at emitErrorNT (internal/streams/destroy.js:106:8)
2022-05-12T17:02:20Z app[9e1b7c2f] fra [info]    at emitErrorCloseNT (internal/streams/destroy.js:74:3)
2022-05-12T17:02:20Z app[9e1b7c2f] fra [info]    at processTicksAndRejections (internal/process/task_queues.js:82:21) {
2022-05-12T17:02:20Z app[9e1b7c2f] fra [info]  errno: -28,
2022-05-12T17:02:20Z app[9e1b7c2f] fra [info]  code: 'ENOSPC',
2022-05-12T17:02:20Z app[9e1b7c2f] fra [info]  syscall: 'write'
2022-05-12T17:02:20Z app[9e1b7c2f] fra [info]}
2022-05-12T17:02:20Z app[9e1b7c2f] fra [info]Main child exited normally with code: 1
2022-05-12T17:02:20Z app[9e1b7c2f] fra [info]Starting clean up.
2022-05-12T17:02:20Z app[9e1b7c2f] fra [info]Umounting /dev/vdc from /var/lib/ghost/content

Any options?

Thanks

Can someone please advise me how can I resize volume or backup the data on unaccessible instance? I can create new one with bigger size, but I can not mount two volumes to instance, so HOW can I backup the original volume or transfer files to the new bigger one?

Thank you

As far as I know volumes can not be resized. In which case it will be a case of creating one with a bigger size and then copying the data to it.

Looking through Volumes it doesn’t mention snapshots however I recall Fly saying they take snapshots (aka backups) of volumes :thinking:.

flyctl volumes list

… and then armed with its ID …

flyctl volumes snapshots list volume_id_goes_here

Which should show the snapshots Fly have taken. That doesn’t help you with getting the data off the volume though - that may be one for someone from Fly when they see this.

I would wait for their go-ahead before making any changes as they may be able to sort it at their end.

Thanks for the reply,

yeah I read something about snapshots, but there is no mention of it in the documentation.

I really don’t want to somehow damage the data, so I am looking for recommended way how to solve this… I think it can happen to anyone at any time, because there is no clear way to check or resize volumes, and even small increase in traffic can cause the disk to fill up because of logs etc.

At least if you could mount more than one volume to the instance and then ssh and cp data.

True.

There may be a way (not documented so don’t know) for them to make a new volume from a snapshot. In which case what may be possible is to create a new volume of a bigger size based on that snapshot. No idea given the file system. Would save you copying the data manually.

There may be metrics available for volumes via their API. But even so, you’d still need a way to act on that to resize a volume. Auto-resizing volumes would be even better.

Whole website is down for third day… Is there any support from Fly team or this is just for hobby projects?
Hard to imagin host here some serious project…

Can someone from Fly please advice or help me to fix website (resize volume) or at least backup the data from the volume?

Thank you

I’m taking a look.

Can you tell me the app’s name?

Fly takes snapshots of volumes, but IIRC currently they can only be restored for VMs running Fly’s special Postgres image.

I manually resized your volume to 6GB. There is no automated process for this.

Your app is now back up.

While it was full, I managed to bring your VM back up by overriding it’s cmd with ["sleep", "1000"]. In your fly.toml, that looks like:

[experimental]
cmd = ["sleep", "1000"]

This would’ve allowed you to SSH in and perform maintenance of some kind.

I’m (a little bit) surprised ghost refuses to boot entirely if the disk is full.

2 Likes

Great. Thank you!

Good to know about the “sleep” workaround.