Two sprites holding real user data stopped waking and have been unreachable ever since. Both ran healthily for weeks (we exec into every sprite on a 3-hour cron, so we have a continuous health record), then each went permanently dead:
- heylife-pool-d2c82cfd (sprite-ffb56ce5-ce6e-4634-aa57-4def32d3d482)
- heylife-pool-66367513 (sprite-c6d1195c-f33e-4b62-b18b-eb7aa86ab89a)
Every 3-hour cycle since those timestamps has failed (dozens of consecutive cycles, 4 retry attempts each). Before this, both sprites only ever showed occasional single-cycle blips that self-healed
Everything we’ve tried:
- JS SDK exec (@fly/sprites execFile/spawn) — fails the WebSocket upgrade immediately with a bare “WebSocket error”. Retried 4x per cycle, every 3h, for days.
- Authenticated GET https://-bsan3.sprites.app/health — hangs ~37s (the wake attempt), then returns 502 with an empty body from server: Fly/945577ce3 — i.e. your edge, not our app.
- GET /v1/sprites/ on api.sprites.dev — returns 200, but reports status:“cold”, last_running_at:null, last_warming_at:null, environment_version:null — as if these machines had never run, despite weeks of successful operation.
- GET /v1/sprites//checkpoints (api.sprites.dev) — times out entirely (>20s, zero bytes received).
- GET /checkpoints and /checkpoints/Current on the sprite gateway — 502.
- POST /v1/services/heylife/restart on the sprite gateway — ~37s hang, then 502 (66367513) / 503 (d2c82cfd).
- Restart from the dashboard — refused:
- Redeploying our runtime — fails, since it depends on the same exec path.
So: your control plane still knows the sprites exist, but every path that touches the actual machine — exec, HTTP wake, checkpoints, service restart, dashboard — fails. We cannot even list checkpoints to export state ourselves.
What we need:
- Recover these two machines, or at minimum restore access to their checkpoints so we can export /home/sprite/state/ (it contains live user data — please don’t discard the volumes).
- Any insight into the cause: we’ve now seen this “permanently cold / unwakeable” mode more than once across our fleet, and we’d like to know whether it’s a known issue and what we can do to prevent or detect it earlier.
Happy to provide our full probe log with timestamps and response headers if useful.
Our workflow, for context — wanted to check whether this is expected usage or an anti-pattern:
We run a fleet of Sprites in two roles: a warm pool of pre-provisioned, unclaimed sprites (so a new signup gets one instantly instead of waiting 2-6min for cold provisioning), and per-user active sprites once claimed. Each sprite runs one long-lived Service that auto-restarts on wake.
Two operations touch the fleet repeatedly:
- Cron that execute some task via on a api
- Redeploy, on-demand when we ship code that runs inside the sprite. Tar the runtime bundle, extract it via exec, POST /v1/services/heylife/restart, then poll /health (up to 60x) until startedAt advances.
- Does this workflow look correct by your standards, or is there an anti-pattern here? In particular: is repeatedly cold-waking an idle sprite many times a day, for days at a stretch, something that’s known to increase the odds of a sprite landing in the “permanently unwakeable” state we’re reporting?
- Is there a better way to write a small file into a sprite than spawning a shell process over exec+stdin? (We don’t want to hold sprites open with the Tasks API just for this — that seems intended for actively-running work, not idle pool sprites — but wanted to check.)
- Any general guidance for a pre-provisioned “warm pool” use case like ours — sprites created ahead of demand and left idle, sometimes for days, until claimed?
Thanks!