I’ve been trying to track down these errors recently. I’m not sure if it’s possible for users to fix it directly, but I fixed the instance here from the host. Your database is running again now, sorry for the trouble.
Same problem here, @Lillian - is there a way to self serve this or auto-fix?
2025-01-22T15:35:44Z health[e2860e6be63d86] sin [error]Health check on port 3000 has failed. Your app is not responding properly. Services exposed on ports [80, 443] will have intermittent failures until the health check passes.
2025-01-22T15:35:45Z app[e2860e6be63d86] sin [info]2025-01-22T15:35:45.128641039 [01HZQMWMDB8HZ7TW56Z7PYRQZ3:main] Running Firecracker v1.7.0
2025-01-22T15:35:45Z app[e2860e6be63d86] sin [info] INFO Starting init (commit: a6222593)...
2025-01-22T15:35:45Z app[e2860e6be63d86] sin [info][ 0.346624] blk_update_request: I/O error, dev vda, sector 2 op 0x0:(READ) flags 0x1000 phys_seg 1 prio class 0
2025-01-22T15:35:45Z app[e2860e6be63d86] sin [info][ 0.347623] EXT4-fs (vda): unable to read superblock
2025-01-22T15:35:45Z app[e2860e6be63d86] sin [info]ERROR Error: couldn't mount /dev/vda onto /lower/dev/vda, because: EIO: I/O error
2025-01-22T15:35:45Z app[e2860e6be63d86] sin [info][ 0.349152] reboot: Restarting system
2025-01-22T15:35:45Z app[e2860e6be63d86] sin [warn]Virtual machine exited abruptly
2025-01-22T15:35:45Z runner[e2860e6be63d86] sin [info]machine exited with exit code 0, not restarting
2025-01-22T15:38:03Z proxy[e2860e6be63d86] sin [info]Starting machine
2025-01-22T15:38:03Z app[e2860e6be63d86] sin [info]2025-01-22T15:38:03.734337892 [01HZQMWMDB8HZ7TW56Z7PYRQZ3:main] Running Firecracker v1.7.0
2025-01-22T15:38:04Z app[e2860e6be63d86] sin [info] INFO Starting init (commit: a6222593)...
2025-01-22T15:38:04Z app[e2860e6be63d86] sin [info][ 0.366729] blk_update_request: I/O error, dev vda, sector 2 op 0x0:(READ) flags 0x1000 phys_seg 1 prio class 0
2025-01-22T15:38:04Z app[e2860e6be63d86] sin [info][ 0.367684] EXT4-fs (vda): unable to read superblock
2025-01-22T15:38:04Z app[e2860e6be63d86] sin [info]ERROR Error: couldn't mount /dev/vda onto /lower/dev/vda, because: EIO: I/O error
2025-01-22T15:38:04Z app[e2860e6be63d86] sin [info][ 0.369257] reboot: Restarting system
2025-01-22T15:38:04Z app[e2860e6be63d86] sin [warn]Virtual machine exited abruptly
2025-01-22T15:38:04Z runner[e2860e6be63d86] sin [info]machine exited with exit code 0, not restarting
2025-01-22T15:38:08Z proxy[e2860e6be63d86] sin [error][PM03] could not wake up machine due to a timeout requesting from the machines API
2025-01-22T15:38:08Z proxy[e2860e6be63d86] sin [info]Starting machine
2025-01-22T15:38:08Z proxy[e2860e6be63d86] sin [error][PM01] machines API returned an error: "machine still attempting to start"
2025-01-22T15:38:09Z proxy[e2860e6be63d86] sin [info]Starting machine
2025-01-22T15:38:09Z app[e2860e6be63d86] sin [info]2025-01-22T15:38:09.172783540 [01HZQMWMDB8HZ7TW56Z7PYRQZ3:main] Running Firecracker v1.7.0
2025-01-22T15:38:09Z app[e2860e6be63d86] sin [info] INFO Starting init (commit: a6222593)...
2025-01-22T15:38:09Z app[e2860e6be63d86] sin [info][ 0.362223] blk_update_request: I/O error, dev vda, sector 2 op 0x0:(READ) flags 0x1000 phys_seg 1 prio class 0
2025-01-22T15:38:09Z app[e2860e6be63d86] sin [info][ 0.373796] EXT4-fs (vda): unable to read superblock
2025-01-22T15:38:09Z app[e2860e6be63d86] sin [info]ERROR Error: couldn't mount /dev/vda onto /lower/dev/vda, because: EIO: I/O error
2025-01-22T15:38:09Z app[e2860e6be63d86] sin [info][ 0.376588] reboot: Restarting system
2025-01-22T15:38:09Z app[e2860e6be63d86] sin [warn]Virtual machine exited abruptly
2025-01-22T15:38:14Z proxy[e2860e6be63d86] sin [error][PM03] could not wake up machine due to a timeout requesting from the machines API
2025-01-22T15:38:15Z proxy[e2860e6be63d86] sin [info]Starting machine
2025-01-22T15:38:15Z proxy[e2860e6be63d86] sin [error][PM01] machines API returned an error: "machine still attempting to start"
2025-01-22T15:38:17Z proxy[e2860e6be63d86] sin [info]Starting machine
2025-01-22T15:38:17Z proxy[e2860e6be63d86] sin [error][PM01] machines API returned an error: "machine still attempting to start"
2025-01-22T15:38:19Z proxy[e2860e6be63d86] sin [info]Starting machine
2025-01-22T15:38:19Z proxy[e2860e6be63d86] sin [error][PM01] machines API returned an error: "machine still attempting to start"
2025-01-22T15:38:21Z proxy[e2860e6be63d86] sin [info]Starting machine
2025-01-22T15:38:21Z proxy[e2860e6be63d86] sin [error][PM01] machines API returned an error: "machine still attempting to start"
2025-01-22T15:38:23Z proxy[e2860e6be63d86] sin [info]Starting machine
2025-01-22T15:38:23Z proxy[e2860e6be63d86] sin [error][PM01] machines API returned an error: "machine still attempting to start"
2025-01-22T15:38:25Z proxy[e2860e6be63d86] sin [info]Starting machine
2025-01-22T15:38:25Z proxy[e2860e6be63d86] sin [error][PM01] machines API returned an error: "machine still attempting to start"
2025-01-22T15:38:27Z proxy[e2860e6be63d86] sin [info]Starting machine
2025-01-22T15:38:27Z proxy[e2860e6be63d86] sin [error][PM01] machines API returned an error: "machine still attempting to start"
2025-01-22T15:38:29Z proxy[e2860e6be63d86] bom [error][PR04] could not find a good candidate within 21 attempts at load balancing
Following these steps I was able to recover from a backup on fly.io and do a dump on my machine.
Wasn’t able to reattach the application but seems doable.
We have still been investigating this issue affecting a small number of Machines with an unavailable rootfs (/dev/vda) on a few hosts. It does seem that the previous manual fix we applied last week was reverted on the at least the ams host. I applied another fix which should once again unblock those instances (a few dozen total).
We’re hopeful the fix will be permanent this time, but if you continue to have trouble, you should be able to destroy the misbehaving Machine and launch a new one as well.
