Postgres down after machine memory update


so I did what is written here Scale Postgres VMs · Fly Docs

I’ve used older version of flyctl and my postgress app status showed me
memory: 1
CPU: 1024

after flyctl update it shows
memory: 1024
CPU: 1

and my db is down :frowning:

any ideas?

Here are some logs

2023-05-01T08:58:13.727 app[1781909b993d08] fra [info] keeper | 2023-05-01T08:58:13.726Z ERROR cmd/keeper.go:1109 db failed to initialize or resync

2023-05-01T08:58:13.796 app[1781909b993d08] fra [info] keeper | pg_basebackup: error: connection to server at "fdaa:1:3c22:a7b:cbb7:10c0:2fc4:2", port 5433 failed: Connection refused

2023-05-01T08:58:13.796 app[1781909b993d08] fra [info] keeper | Is the server running on that host and accepting TCP/IP connections?

2023-05-01T08:58:13.796 app[1781909b993d08] fra [info] keeper | 2023-05-01T08:58:13.796Z ERROR cmd/keeper.go:1364 failed to resync from followed instance {"error": "sync error: exit status 1"}

2023-05-01T08:58:14.485 proxy[1781909b993d08] fra [error] could not proxy TCP data to/from instance: failed to copy (direction=client->server, error=Transport endpoint is not connected (os error 107))

I just wanted do scale up my db a little since my app is getting a lot of traffic lately, and now my app is down :frowning:

1 Like

So I’ve just moved to paid plan to be able to contact support directly, I’ve sent an email to the address provided in the billing secion, but I didn’t get any confirmation or anything in return.

I am not sure if my email hit support inbox or not.

Can someone here let me know if is aware of my issue?

Ok, so it looks like there’s maybe two issues going on here.

  1. There’s only one volume tied to this app with two machines.

  2. Neither machine is attached to the volume that’s currently present.


Use the clone tool to create a new machine and attach it your existing volume.

fly machine clone <existing-machine-id> --attach-volume <volume-id>

Note: You can list volumes with fly volumes list.

Once this is done, you should delete any machines that are not tied an a volume. This should leave you with a single machine holding the primary.

Going forward, any calls to fly machines clone will include the volume and should do what’s expected.

1 Like

thx for reply, what mounting point should I use?

Error: Please specify a mount path on 'vol_53q80vd9e76rgzy6' using <volume_id>:/path/inside/machine

ok, I tried wit /data path, but then

Error: failed to launch VM: volume already claimed by machine <machine-id>

so how can I unmount volume from original machine

Which version of flyctl are you running?

flyctl.exe v0.0.549 windows/amd64 Commit: 59ee5583 BuildDate: 2023-05-01T13:16:47Z

I just updated it few minutes ago

Just to clarify.

This postgres app had only one machine, but since this issue happend I tried to do different things described on fourms, one of those things was to create clone, which I did and this is second machine.
I did fly machine destory <second-machine-id> but it’s still there

so this is option 1, but the issue happened before adding second machine

issue happend after

fly machine update <machine-id> --memory 1024 --app <app-name>

@roszman Taking a closer look. Hold tight.

1 Like

Ok, the machine is back running and attached to the volume. However, there’s no health checks tied to this machine and looks like there never was, which is super confusing to me. Give me a few more minutes.

1 Like

awesome, I can connect now, super thx.

How can I change this postgres app to HA cluster?

1 Like

Looks like you’re good to go.

Go ahead and run fly machines clone <machine-id> and that should spin up a replica.

will this replica have it’s own volume attached to it?

Yep! When things are setup properly, it’ll copy the spec of the target machine, which also includes the attached volume.

great, last question. What can I do to safely scale up postgres and not encounter such issue?

can I do dry run or something? try with clone first?

Are you using an interactive terminal? If so, you should be prompted on machine update with the exact changes that will be applied to your machine.