Rails + Postgres "corrupted shared memory"

[mrjonesbot]

Hi all,

After successfully deploying my Rails application, specific GraphQL queries are returning 500 errors (most queries succeed).

I have a shared-cpu-1x VM, running 2 web processes and 1 worker process.

Both web and worker are scaled to use 1gb memory.

The most expensive query returns a http 500 response:

Server logs:

Full error:
WARNING: terminating connection because of crash of another server process
DETAIL: The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.
HINT: In a moment you should be able to reconnect to the database and repeat your command.

RAILS_MAX_THREADS is set to 10 and WEB_CONCURRENCY to 2, which means each process should have 20 connections (unless I’m misunderstanding what processes are).

“postmaster” clearly indicates some interaction with Postgres, but I’m struggling to find any logs (where are the postgres logs? var/log/postgres is empty) to tell me exactly what the issue is. Any help is greatly appreciated.

My database is a shared-cpu-1x with 1gb memory, average memory usage is 405mb/1gb

Here’s what postgres.conf looks like:

[mrjonesbot]

Today I checked the logs and the previous error message has not been seen, although I’m still experiencing 500 errors.

There is no error message returned from the response, so I suspect it’s not even hitting my graphql controller.

The only difference that I’ve noticed between good and bad requests is the Content-Length header value.

The request that fails has a Content-Length of 3070, whereas successful requests are under 500.

Dumb question: Do fly VMs enforce a content length max on inbound requests?

[mrjonesbot]

Ok rather embarrassing, but it looks like a run of the mill ActiveRecord error.

I forgot I had temporarily disabled Appsignal just to get the deploys working, so I wasn’t seeing any errors thrown in my monitoring dash.

Borderline want to throw my laptop out the window…thanks for reading!