App stuck in Pending, Volume "Migrated", maybe for a week already

Hello! We have a keydb hot-backup running in fly and today we noticed that our metrics stopped coming in about a week ago (October 28th). Beyond needing to be more aware of that in the future (:sweat_smile:), I’ve had a hard time getting it back up. When I look at the app I see it’s in the “Pending” state. I doesn’t seem like there’s any log of state transitions in the UI anywhere (which would be super helpful by the way :wink:) so I don’t know how long it’s been in this state, but I have to assume it started when the metrics stopped and hasn’t recovered since.

I tried to scale to 0 and back up to 1, but that doesn’t seem to have fixed anything. There’s no logs from the app at all, so I don’t really have a lot of information to go on here. I don’t know if this is related, but the volume we have is in the “migrated” state, which as far as I can tell from other forum posts means it got deleted or something in the past? That was not a thing we did, so I’m not sure if that’s the problem, like if it’s waiting on the volume resource to bind in order to launch, or if it’s unrelated.

So before I delete the volume and rebuild this replica from scratch, is there some debugging that could be done to figure out what’s going on here?

Hello @psycotica0! From the date, it sounds like your replica may have got caught up in this incident; the volume migration would have been part of our recovery from that. From there, though, the recovery of this app did not complete automatically. Sorry about that.

Since some time has gone by, if you have the up-to-date primary, rebuilding the replica is probably, indeed, the way forward here.