Hi all,
One of my sprites started failing yesterday and I’m curious whether others have run into the same thing, and how you deal with it.
What I saw
-
The platform saved a checkpoint named
corrupt-…with the comment “Corrupt filesystem snapshot saved during recovery”. I hadn’t restored or checkpointed anything. -
A few hours later, anything that ran for more than about a minute died: the filesystem remounted read-only and every service on the sprite restarted at once.
-
dmesgwas full of read errors onloop1(mounted at/.sprite/checkpoints/active):I/O error, dev loop1, sector 155631928 op 0x0:(READ) ... EXT4-fs error (device loop1): __ext4_find_entry:1645: inode #4856425: comm chrome_crashpad: reading directory lblock 0 EXT4-fs (loop1): Remounting filesystem read-only -
Disk was 5% used and there was no memory pressure, so it didn’t look like anything in my workload.
In the end I copied off the few files I needed, destroyed the sprite and created a fresh one, which is what I understood others did in #27851 and #27319.
What I’d love to hear from people running sprites for longer-lived work
- Have you seen this, and did it come back after the platform’s own recovery?
- Do you treat a
corrupt-…checkpoint appearing as a signal to replace the sprite straight away? - How do you notice it early? Right now I’m thinking of a periodic
dmesg | grep "I/O error"check. Is there anything better, like an API field or event? - My sprite sleeps and wakes often while a browser writes to disk constantly. Has anyone found ways to reduce the risk, e.g. keeping it awake during heavy writes, checkpointing less often, or putting write-heavy data somewhere else?
Thanks, any experience welcome!
