Sprite doesn't boot/come up anymore for a second time

Questions / Help
1

sprite c -s adsr-handboek-***

Error: failed to start sprite command: failed to access sprite. can’t exec or start it. it contains actual production data so having it back would be great

ty!

3

It’s back but it has dataloss, about a week or 2 of production data hand authored by my users.

This is the second time, a sprite, fails to boot for me, and now it incurs dataloss. (See my last post) I’m not trusting sprites at this moment and their data storage.

4

Hi Emile, thanks for reaching out.

Most of your sprites are on older versions of the storage layer subsystem, it was more prone to filesystem issues which could result in data loss. One thing I can do is update them to the latest version which is more resilient.

We checked on our side and didn’t spot anything that could explain the data loss, which also unfortunately means we didn’t find anything we could directly do to restore the data. My best advice here is for you to check some of the older snapshots you have (take a current snapshot first!) to see if you can salvage some of the data.

I’m sorry to see this happen to you, but let me know if the above helps and ideally the storage upgrade will help prevent further issues - we are doing a lot of work on sprite storage reliability as a high priority.

5

I think it already had duplicate checkpoint errors which maybe meant it just resorted to the last checkpoint? However I do believe that the idea was with Fly’s “hacked up juice fs with sqlite Litestream block replication” (see post) it should just be basically safe?

But so far I’ve a mixed result, maybe have some sort of SLA for sprites?

Would be great if you had some data guarantees because r/n it really isnt a good sign. Also shouldnt any data be continuously be replicated to the underlying storage ? or only when you checkpoint?

6

Hi! Thanks for bringing up those concerns and ideas. The reliability work I mentioned is precisely targeted to address what you were concerned about. The chance of data loss should be reduced dramatically, so you can trust files stored in a Sprite will be safe. We should have something soon.