Sprites: dev2 unavailable after restart — 503 Storage not ready

Questions / Help
1

Hi Fly team,

Our Sprite dev2 is unavailable, leaving its existing Morty/Mobius Runtime connections offline. Please help investigate and recover access while preserving its current data.

Instance

  • Organization: moxt-ai
  • Sprite: dev2
  • Sprite ID: sprite-615374a5-571c-41d7-bb2b-80d52d561f1e
  • Machine ID returned by restart: 85e714a4902138
  • Environment version: 0.0.4
  • Observed incident window covered by this report: 2026-10-29 07:29–07:47 UTC

Symptoms
Metadata requests succeed, but execution and filesystem access fail. Even a direct POST executing only /bin/echo returns HTTP 503: Storage not ready. Service listing/logs, file reads, and exec session listing time out. Metadata has alternated between cold and running, but successful metadata responses have not meant the environment is usable.

Recovery attempts (all times UTC)

  • 07:37:09: /check reported repaired / restarted stopped machine; execution still failed.
  • 07:38:31: requested /restart. It returned 504 outcome_unknown at 07:39:03; we checked status instead of immediately retrying.
  • 07:40:03: /check reported healthy / machine is running; execution still timed out.
  • 07:42:00: default /upgrade reported already_current and skipped (0.0.4). Storage migration was not enabled.
  • After confirming the Sprite had returned to cold, one bounded restart retry was sent at 07:43:45. It returned HTTP 202 Accepted at 07:43:47 with machine ID 85e714a4902138.
  • Subsequent execution still timed out after 55 seconds; service listing also timed out.
  • The final direct echo attempt in this diagnostic sequence returned 503 Storage not ready at 07:47:36.

Fly request IDs

  • Failed exec at 07:47:36: 01M63A0WR8CZYQRXE45P0C6VMN-nrt
  • Accepted restart: 01M639TVFJXT74Q6BZFCCPHBNH-nrt
  • Earlier restart with unknown outcome: 01M639H8ZQ6902R5MVN3PSDSC8-nrt

Please inspect the Sprite’s SBD/storage startup and host logs, and repair or relocate its backing host while preserving the current volume, files, and configuration. If that is not possible, please help identify the latest recoverable checkpoint and its timestamp before any rollback.

As of the diagnostic window above, no checkpoint restore, deletion, resource-limit change, or storage migration had been performed. Current kernel and memory data could not be collected because execution was blocked. We can provide additional diagnostic details if needed.

Thanks,
Moxt AI Team