Hi Fly team,
Our sprite burst has been unusable since about 23:30 UTC on 5 Nov. Every other sprite in the org works. It holds source code we have no other copy of, so we need it recovered with its filesystem intact.
Sprite
- Org: cerebration-ltd
- Sprite: burst
- ID: sprite-f62a1e98-1e70-4743-ae28-25929caab5e2
- Version: now 0.0.4-beta.4 (was 0.0.2-beta.8)
What happened (UTC)
- 5 Nov, ~23:30: sprite console failed with failed to connect: read tcp …->169.155.48.226:443: i/o timeout. Status showed running, but exec hung, out (request
2b086b3f-9200-434a-a8c7-76d3fc0347e2),RL timed out. - Dashboard restart was rejected: “The uhe restart of burst due to a conflict or
unmet precondition.” The sprite then w - Exec now returns HTTP 503 … Storage nondshake (01M6Q2KHF06Y42FH26J4YKY3D8-lhr).
- 6 Nov, 00:07–00:09: POST /v1/sprites/b.4-beta.4) reported machine_started and
Upgrade complete. On 29 Oct the same ur outage on this sprite. - After the upgrade:
- exec still returns 503 Storage not rXE1FV1-lhr), and retries every 2.5minutes from 00:15 to 00:50 all failed.
- POST /check returns Storage not read
- GET /checkpoints returns Storage not ready or times out (1af08f4d-e673-4f3e-873b-9e4243c7a867).
- A CLI connect at 03:34 stalled after12-8641-696b94bce3c5).
- Latest: 01M6R643ZV2CR3VJ0P2R09VM3F-lhr. It’s still failing as I post.
Other details
- Our other 12 sprites start normally. di/o timeout overnight and recovered byitself at about 03:36.
- We also filed this through the feedbacgx13dyx8e).
What we’d like
- Please recover or move burst to another host while keeping its current filesystem.
- If /tmp has filled up, as in the dev2 fly.io/t/28948), clearing it is fine.
- Please don’t restore a checkpoint without checking with us first. The current disk holds work that’s in no checkpoint and nowhere else.
Thanks!