I seem to have machines that are zombies

[Chris_Murphy]

I see there is a service interruption:

We are performing emergency maintenance on a host some of your apps instances are running on. Apps may be unavailable until the maintenance is completed.

My one app I did:

fly scale count 0
fly scale count 5

and was back to functional. I tried the same thing on another app and this did not work. I tried to do a deploy and various region changes for scaling, and I cannot get the app working again.

I am seeing 3 machines listed for:
fly m list
but the dashboard is showing 5 machines. With no ability to get the Machines page to load.

fly m destroy e286e30a121d86
Error: could not get machine e286e30a121d86: failed to get VM e286e30a121d86: request returned non-2xx status, 408

It seems like there are dead machines with the traffic routed to them and I can’t find any way to get the new machines to get the requests instead? Is there anything I can do here?

[john-fly]

Hi Chris, I want to write a more thorough response but have to go do something right now. Wanted to get this out quickly though. This:

@Chris_Murphy

seems like there are dead machines with the traffic routed to them

shouldn’t be true. If a host server goes down the Fly Platform should not direct traffic to the Machines on that server nor should you get charged. If you’re seeing evidence to the contrary, that’s really not good and we’d like to know asap.

[john-fly]

Hi Chris, let me give you a more thorough update. At the time you posted, we were still debugging what was going on with the server your e286e30a121d86 Machine was on. We now know that it was damaged, and if you look at the dashboard now you should see:

A server hosting some of your apps has suffered irreparable hardware damage. Please migrate your Fly Machines to other hosts and restore volumes from any backups.

And it looks like you have scaled up some Machines elsewhere, which means you should be good to go. The zombie Machines are the metadata ghosts of the Machines that were on the dead server. We need to update our system to delete them from the backend. That will happen soon, but in the meantime, know that you are NOT being charged for them, nor is the Fly Platform trying to direct traffic to them.