Recently my app went down due to an “irreparable hardware damage” on the host it was running on.
A server hosting some of your apps has suffered irreparable hardware damage. Please migrate your Fly Machines to other hosts and restore volumes from any backups.
I’ve managed to follow the instructions here and service is now restored. However i can’t seem to get rid of the dead machine and volume.
This is also causing the machines and volumes page to not load. and can’t stop or destroy them via the CLI.
Is there a way to clear these things out? As i’d rather they don’t appear on my billing, as it says my apps are running one more machine than they are. which is also causing an issue for clustering via DNS. As the dead machine is showing up in the DNS query.
Hey @erk , that host is indeed dead and not coming back. Not being able to stop or destroy those machines and volumes is a bug in our API, we’re still working on a fix for it. We’ve been running into some new cases with these recent host failures that weren’t accounted for previously, sorry about this.
In the meanwhile though, we’ve checked to make sure that you won’t be billed for those resources that are inaccessible on the dead host. In the event that we’re wrong about this and you do end up seeing charges for these at the end of the month, you can email billing@fly.io and we’ll fix that.