Sprite unreachable; dashboard restart returns “upstream restart failed”

Questions / Help
1

Hi Fly team,

One of our Sprites is unreachable:

  • `sprite list` works, and the dashboard shows the Sprite as Cold.
  • `sprite exec` fails with `failed to connect: read tcp [redacted]:443: i/o timeout`.
  • HTTP requests time out without receiving a response.
  • Restarting from the dashboard returns `upstream restart failed`.

We tried different local proxy nodes without success. Separately, an application hosted on Fly also recorded timeouts when calling this Sprite. We have not yet determined the cause.

We found similar reports:

Could you check whether this might be related and advise on recovery?

Please preserve all persistent data and contact us before any reset, checkpoint rollback, or action that could discard data.

We can provide the organization, Sprite ID, exact timestamps, and additional diagnostics through a private support channel. Please let us know the best way to share them.

Thanks!

2

Hi there!

I looked at your two organizations and found one sprite in a strange state. I relocated it to a new machine, let me know if this fixes the issue. For reference, md5 hash of the sprite’s name: c61ce2fa50bd7af783be08f9cd1d8b3c (hash the name on your side and if you get this value, it means it was the same sprite).

3

Hi, thanks for looking into this.

We retested after your update, but the affected Sprite is still unavailable:

  • `sprite exec` now returns `failed to start sprite command: service temporarily unavailable, please retry`.
  • An HTTP request to its application endpoint returns a Fly/Sprites “404 Not Found” HTML page.
  • `sprite list` still shows the Sprite, and its public URL has not changed.

The MD5 you provided does not match the affected Sprite’s name. Could you confirm whether the relocated Sprite is the one we reported?

Please let us know how to send you the organization and Sprite ID privately so you can check the correct instance.

Please preserve its persistent data and contact us before any reset or checkpoint rollback.

Thanks!

4

The MD5 you provided does not match the affected Sprite’s name. Could you confirm whether the relocated Sprite is the one we reported?

I cannot because you never reported a sprite name :innocent:

We can communicate via hashes, since you don’t have a lot of organizations and/or sprites. Send me a hash of the organization slug and the sprite name and I’ll look it up.

5

Thanks. Here are the MD5 hashes, computed from the exact strings without a trailing newline:

  • Organization slug: 83a0e90915f2be59fe15df581ec90f13
  • Sprite name: 3e0d74d9a432775b4ec1e3539ff37f1e

Please check this instance and preserve its persistent data during recovery.

6

Thanks, I located the instance that exactly matches those hashes. I checked and it’s apparently working fine, I’m able to hit its API and run commands on it. Can you check again? Let me know and if still unreachable for you I re-assign it to another machine. No worries, data is not touched in this process at all.

7

Confirmed, the issue is resolved now. Thanks for checking the instance and helping us with this!