We have an unmanaged Postgres cluster. Today I did a simple deploy of the app moving us to the major version image (we were previously pinned to minor version)
Ever since, we have been getting tons of errors from our connected Rails app being unable to establish a connection. All of these seem to be SSL SYSCALL error: EOF detected issues. Notably, however, it is only a small percentage of all requests.
App 1073 output: [xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx] PG::ConnectionBad (connection to server at "fdaa:xx:xxxx:x:x::2", port 5432 failed: SSL SYSCALL error: EOF detected
No other configuration was changed other than the Postgres-flex tag. Any ideas?
I noticed that this was happening in our staging app as well so have been running experiments there.
We are running a Rails app, and previously our database URL used appname.fly.dev as the hostname and provided no options.
I cleared that database URL and ran fly postgres attach to generate a new one. The new one used appname.flycast and also added the sslmode=disable option to the end.
This (as expected) changed our SSL SYSCALL error to
connection to server at “fdaa:xx:xxxx:x:x::2”, port 5432 failed: server closed the connection unexpectedly (ActiveRecord::ConnectionNotEstablished) This probably means the server terminated abnormally before or while processing the request.
…which I assume was the real underlying issue the whole time, but was masked by a default sslmode=prefer default or something.
Really puzzling, especially since I verified that despite unpinning the Postgres-flex minor version, it still puts us on 17.2, which is what we were pinned to previously
Ultimately, we rolled back to the pinned 17.2 image and that did seem to resolve our connection problems.
In our investigation, all signs seemed to point to the issue lying somewhere in proxy configuration differences between the two images. Postgres configuration seemed basically the same, Postgres wasn’t throwing any errors on its side, and the connection appeared to be dropping at the negotiation stage. Very strange