About 12 hours ago (at the time of writing), postgres stopped allowing write operations (inserts and updates). I’ve checked the volume for the db, and there’s disk space available. I haven’t made any changes to postgres or application code that interacts with postgres during that time.
Specifically, my app is a node.js app that uses sequelize and the exception I’m getting is SequelizeDatabaseError: cannot execute INSERT in a read-only transaction. I haven’t updated sequelize in this time span either.
I have no idea where else to go to debug this short of digging through the sequelize code. I’m hoping someone else here has seen this before.
For any fly employees, the postgres app name is ott-db-prod.
Hi @dyc3—Fly Postgres can set itself read-only to avoid further problems when it detects that it’s unhealthy. For example, it will become read-only when the disk usage reaches 90% (since Postgres can panic and shut down outright if it runs out of space).
To see if anything is wrong, you can view your database’s heath-check output with fly checks list or in the “monitoring” tab for the app in the fly.io web dashboard. (In the interest of transparency—I can see on our end that your DB has some failing checks.)