My Postgres's machine in `created` state and I can't stop/restart/kill?

Hi,

My Postgres’s machine is now in the created state and I can’t do anything about it…

Can you help me?

fly status --all
ID            	STATE  	ROLE 	REGION	HEALTH CHECKS	IMAGE                        	CREATED             	UPDATED
3908031f179787	created	error	ewr   	3 total      	flyio/postgres:14.6 (v0.0.34)	2023-01-23T17:16:39Z	2023-01-23T17:16:39Z	
fly machines list
1 machines have been retrieved from app scanner.
View them in the UI here (​https://fly.io/apps/scanner/machines/)

scanner
ID            	NAME        	STATE  	REGION	IMAGE                        	IP ADDRESS                     	VOLUME              	CREATED             	LAST UPDATED
3908031f179787	shy-sun-5188	created	ewr   	flyio/postgres:14.6 (v0.0.34)	fdaa:0:7313:a7b:ab2:4365:b0e3:2	vol_okgj545z8k94y2wz	2023-01-23T17:16:39Z	2023-01-23T17:16:39Z	
fly machines restart 3908031f179787
Restarting machine 3908031f179787
Error failed to restart machine 3908031f179787: could not stop machine 3908031f179787: failed to restart VM 3908031f179787: unable to restart machine, not currently started or stopped

Hi @rickyking ,

Try killing the fly machine and see if that helps.

fly m kill

Thanks,

Tried but it does not work:

fly machines kill 3908031f179787
machine 3908031f179787 was found and is currently in a created state, attempting to kill...
Error could not kill machine 3908031f179787: failed to kill VM 3908031f179787: machine not in known state for signaling, created

Try restarting the PG cluster members

fly pg restart

fly pg restart --force
No leader found, but continuing with restart
Identifying cluster role(s)
  Machine 3908031f179787: error
Restarting machine 3908031f179787
Error could not stop machine 3908031f179787: failed to restart VM 3908031f179787: unable to restart machine, not currently started or stopped

The only option I could think of is to delete that PG cluster and relaunch a new one.

fly m destroy
fly pg create

oh …

I need the data from that volume though…

Oh no. Then let’s wait. But wondering how you got data in that VM in first place when it is stuck in created state.

Can you post the output for

fly m status

I know sshing to that machine may not work. But can you give a try and see if it works?

fly ssh console

fly m status 3908031f179787
Machine ID: 3908031f179787
Instance ID: 01GQFT3HK838ZRCXQ8D5CWPB4Z
State: created

VM
  ID            = 3908031f179787
  Instance ID   = 01GQFT3HK838ZRCXQ8D5CWPB4Z
  State         = created
  Image         = flyio/postgres:14.6 (v0.0.34)
  Name          = shy-sun-5188
  Private IP    = fdaa:0:7313:a7b:ab2:4365:b0e3:2
  Region        = ewr
  Process Group = app
  Memory        = 2048
  CPUs          = 1
  Created       = 2023-01-23T17:16:39Z
  Updated       = 2023-01-23T17:16:39Z
  Command       =
  Volume        = vol_okgj545z8k94y2wz

Event Logs
STATE  	EVENT 	SOURCE	TIMESTAMP                    	INFO
created	launch	user  	2023-01-23T09:16:39.422-08:00	

tried fly ssh console seems not working …

Hey there,

If you can’t get the Machine to stop, I would try the following:

Force remove the Machine
fly machines remove <machine-id> --force --app <app-name>

List your volumes
fly volumes list --app <app-name>

Clone the Machine you deleted and attach it to your volume
fly machines clone <machine-id-you-deleted> --attach-volume <volume-id>

Hi @rickyking ,

As @shaun suggested you can clone a new machine from the existing PG cluster and destroy it once your cloned machine is up and running.

fly m clone 3908031f179787
fly ssh console or fly pg connect (Ensure you are able to connect to your PG cluster)

fly m destroy 3908031f179787

This will only destroy the machine & not the persistent volume. You can destroy the volume if required using

fly vol destroy vol_okgj545z8k94y2wz

But before you destroy this volume ensure you have your data intact in the new running PG cluster.