That proxy health check is basically the problem, it’s not accepting connections.
If you look at fly logs you might see some errors connecting to Consul. The Stolon proxy doesn’t handle these kinds of errors well and won’t recover. We’ve been hesitant to automatically restart databases but the fix is to stop the VM(s) with that error and see if health checks start passing.
We have some improvements to the postgres app coming soon that seem to help with this problem.
One thing you can look into is connecting to the Postgres nodes directly and bypassing the proxy. Getting DB drivers to do this takes a little work, but it would keep you functional when this problem rears its head.
Most drivers need you to build a connection string like this:
postgres://user:password@[fdaa:0:5d2:a7b:7d:0:2cc9:2]:5433,[fdaa:0:5d2:a7b:7d:0:2cc9:3]:5433/database?target_session_attrs=read-write
Those are made up IPs, you’ll want to replace the addresses with whatever is in fly ips private -a premade-db.
The target_session_attrs is a libpq option that tells the driver to connect to both and use the one that’s writable. It works exceptionally well if the driver implements it. If we could figure out how to get all apps to connect this way, we’d drop the proxy (which is really just there for convenience).