FATAL: could not map anonymous shared memory: Cannot allocate memory

1

I was trying to increase the shared buffers size for Postgres using

flyctl postgres config update --shared-buffers

After which I restarted using

fly postgres restart

Now it’s just in a loop trying to start and failing:

app[e1baca0d] mia [info] keeper | 2022-11-21T15:34:02.673Z INFO postgresql/postgresql.go:319 starting database
app[e1baca0d] mia [info] keeper | 2022-11-21 15:34:02.690 UTC [2271] FATAL: could not map anonymous shared memory: Cannot allocate memory
app[e1baca0d] mia [info] keeper | 2022-11-21 15:34:02.690 UTC [2271] HINT: This error usually means that PostgreSQL’s request for a shared memory segment exceeded available memory, swap space, or huge pages. To reduce the request size (currently 8989472948224 bytes), reduce PostgreSQL’s shared memory usage, perhaps by reducing shared_buffers or max_connections.
app[e1baca0d] mia [info] keeper | 2022-11-21 15:34:02.691 UTC [2271] LOG: database system is shut down
app[e1baca0d] mia [info] keeper | 2022-11-21T15:34:02.874Z ERROR cmd/keeper.go:1526 failed to start postgres {“error”: “postgres exited unexpectedly”}
app[e1baca0d] mia [info] checking stolon status
app[e1baca0d] mia [info] keeper | 2022-11-21T15:34:02.964Z ERROR cmd/keeper.go:719 cannot get configured pg parameters {“error”: “dial unix /tmp/.s.PGSQL.5433: connect: no such file or directory”}
app[e1baca0d] mia [info] checking stolon status
app[e1baca0d] mia [info] sentinel | 2022-11-21T15:34:04.824Z INFO cmd/sentinel.go:995 master db is failed {“db”: “e9c955d4”, “keeper”: “2cc30d1852”}
app[e1baca0d] mia [info] sentinel | 2022-11-21T15:34:04.824Z INFO cmd/sentinel.go:1006 trying to find a new master to replace failed master
app[e1baca0d] mia [info] sentinel | 2022-11-21T15:34:04.825Z ERROR cmd/sentinel.go:1009 no eligible masters
app[e1baca0d] mia [info] checking stolon status
app[e1baca0d] mia [info] keeper | 2022-11-21T15:34:05.465Z ERROR cmd/keeper.go:719 cannot get configured pg parameters {“error”: “dial unix /tmp/.s.PGSQL.5433: connect: no such file or directory”}
app[e1baca0d] mia [info] checking stolon status
app[e1baca0d] mia [info] exporter | INFO[0330] Established new database connection to “fdaa:0:ddd8:a7b:2cc3:0:d185:2:5433”. source=“postgres_exporter.go:970”
app[e1baca0d] mia [info] checking stolon status
app[e1baca0d] mia [info] exporter | ERRO[0331] Error opening connection to database (postgresql://flypgadmin:PASSWORD_REMOVED@[fdaa:0:ddd8:a7b:2cc3:0:d185:2]:5433/postgres?sslmode=disable): dial tcp [fdaa:0:ddd8:a7b:2cc3:0:d185:2]:5433: connect: connection refused source=“postgres_exporter.go:1658”
app[e1baca0d] mia [info] checking stolon status
app[e1baca0d] mia [info] keeper | 2022-11-21T15:34:07.968Z ERROR cmd/keeper.go:719 cannot get configured pg parameters {“error”: “dial unix /tmp/.s.PGSQL.5433: connect: no such file or directory”}
app[e1baca0d] mia [info] keeper | 2022-11-21T15:34:08.003Z INFO cmd/keeper.go:1504 our db requested role is master

I’ve tried lowering the shared buffers or the max connections, but I always get this in response:

Error can’t get role for fdaa:0:ddd8:a7b:2cc3:0:d185:2: 500: context deadline exceeded

I’ve tried restarting postgres, restarting the app, scaling the app all to no avail. I’m not sure what I should do now to fix the problem.

Thanks for any help you can provide.

2

It seems when I scale the VM, the size of the shared buffers is scaled with it, and so I run into this issue even with larger VM sizes.

3

I was able to follow this answer to update the shared buffers size: How to configure and manage Postgres databases with Stolon? - #2 by greg

4

In the process, I upgraded flyctl from v0.0.431 to v0.0.433, after which I was not able to use flyctl to restart postgres. The only way I could figure how to restart postgres, which I needed to do in order to apply the change I made using stolonctl, was to scale the VM.

5

It seems when I scale the VM, the size of the shared buffers is scaled with it, and so I run into this issue even with larger VM sizes.

@jswanner This is not actually the case. If you’re referring to the configuration output that’s printed on boot, we need to remove that as it’s quite accurate.

You can use the following to see what’s currently configured:

flyctl pg config view

or

# from within your VM.
stolonctl spec

flyctl postgres config update --shared-buffers

What value did you specify here?

6

Could you provide more information on this? Are you hitting a specific error?

7

I was trying to set it to 1GB:

flyctl postgres config update --shared-buffers 1073741824
Error invalid value specified for shared_buffers. (Received: 1073741824, Accepted range: (16, 1073741823)

So I set it to the max 1073741823

flyctl postgres config update --shared-buffers 1073741823

$ flyctl postgres restart 
Error can't get role for e3292b5b-2f9c-2f0e-27b9-cc2eda042a68: 500: context deadline exceeded
8

At the time I was not able to use this to view the config:

flyctl postgres config view
Error 500: context deadline exceeded

Oh, good to know for the future.

Thanks for your help, @shaun.

9

Got it, so shared-buffers doesn’t have a straight 1:1 memory to allocation mapping.

If you look below, you’ll notice the 8kb unit next to shared-buffers. So for every unit, you’ll allocate 8kb.

NAME                      	VALUE  	UNIT	DESCRIPTION                                                                                	PENDING RESTART
log-min-duration-statement	-1     	ms  	Sets the minimum execution time above which all statements will be logged. (-1, 2147483647)	false
log-statement             	none   	    	Sets the type of statements logged. [none, ddl, mod, all]                                  	false
max-connections           	300    	    	Sets the maximum number of concurrent connections. (1, 262143)                             	false
shared-buffers            	8192   	8kB 	Sets the number of shared memory buffers used by the server. (16, 1073741823)              	false
shared-preload-libraries  	       	    	Lists shared libraries to preload into server.                                             	false
wal-level                 	replica	    	Sets the level of information written to the WAL. [minimal, replica, logical]              	false