fly pg create --fork-from stuck in health check loop

Questions / Help
, ,
1

I’m trying to use --fork-from to automate staging forks for PR preview servers.

I got --fork-from to work when using certain other databases, but not the staging database I want. When I try to fork, the CLI gets stuck in the health checks:

 fly pg create --name fork-test-2 --fork-from staging:vol_... --org frdm --vm-size shared-cpu-1x --volume-size 20 --initial-cluster-size 1
Creating postgres cluster in organization frdm
Creating app...
Setting secrets on app fork-test-2...
Provisioning 1 of 1 machines ...
Waiting for machine to start...
Machine 9080ddece16687 is created
==> Monitoring health checks
  Waiting for 9080ddece16687 to become healthy (started, 0/3)

That’s it – the status cycles infinitely between “started” and “healthy”.

The postgres source app has 2 machines, 2 volumes (each 20GB). When I try the same command for a smaller, 1-volume, 1-machine postgres app, it works, so might have something to do with that.

2

And there might also be a difference in Stolon ↔ Repmgr. Glancing at the flyctl source code, it looks like --fork-from should auto-detect the older style, but maybe that went awry this time…

What do the following show you?

fly status -a pg-source-app-name  # particularly IMAGE.
fly logs   -a fork-test-2         # new attempt.
3

The image is the same in all cases (flyio/postgres-flex:15.3 (v0.0.46)), but the logs gave me something to work with:

2024-01-22T18:55:21Z app[1857461f4e1268] sea [info]postgres | Running...
2024-01-22T18:55:22Z app[1857461f4e1268] sea [info]postgres | 2024-01-22 18:55:22.051 UTC [323] FATAL:  could not map anonymous shared memory: Cannot allocate memory
2024-01-22T18:55:22Z app[1857461f4e1268] sea [info]postgres | 2024-01-22 18:55:22.051 UTC [323] HINT:  This error usually means that PostgreSQL's request for a shared memory segment exceeded available memory, swap space, or huge pages. To reduce the request size (currently 587022336 bytes), reduce PostgreSQL's shared memory usage, perhaps by reducing shared_buffers or max_connections.
2024-01-22T18:55:22Z app[1857461f4e1268] sea [info]postgres | 2024-01-22 18:55:22.051 UTC [323] LOG:  database system is shut down
2024-01-22T18:55:22Z app[1857461f4e1268] sea [info]postgres | exit status 1
4

The vm I was using (shared-cpu-1x) had 256MB memory, and Postgres for some reason needs north of 580MB, as the error message indicates here “PostgreSQL’s request for a shared memory segment exceeded available memory, swap space, or huge pages. To reduce the request size (currently 587022336 bytes),”.

I’m not sure why it thinks it needs that much; the shared-buffers value for the source app is 65MB:

flyctl pg config view -a frdm-db-staging | grep shared-buffers
shared-buffers            	65536  	8kB 	Sets the number of shared memory buffers used by the server. (16, 1073741823)              	false

While I was able to do the fork with a shared-cpu-4x (1GB memory), I obviously would like to use the smallest machine possible, since these are really just temporary PR servers I’m trying to run.

EDIT I was able to scale down with a fly machine update 7811ed3b9754d8 --vm-size shared-cpu-1x -a fork-test-4. While slightly annoying to have a couple of extra steps, this will do for now.

5

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.