Fly Postgres proxy issues and dropped connections(started 2 hours ago)

Questions / Help
1

Hello! My Fly Postgres was working great for a few weeks now, but this proxy issue showed up a few hours ago. The database dropped all connections from internal fly apps and won’t reconnect.

ewr [info] proxy | [WARNING] 350/152927 (562) : Server bk_db/pg1 is DOWN, reason: Layer7 invalid response, info: "HTTP content check did not match", check duration: 7ms. 0 active and 1 backup servers left. Running on backup. 3 sessions active, 0 requeued, 0 remaining in queue.

I ran the Health check command. This is what I am seeing:

❯ flyctl checks list -a <app_name>
Health Checks for <app_name>
  NAME | STATUS   | MACHINE        | LAST UPDATED | OUTPUT
-------*----------*----------------*--------------*----------------------------------------------------------------------------
  pg   | critical | <redacted>     | 2h35m ago    | 500 Internal Server Error
       |          |                |              | failed to connect to proxy: context deadline exceeded
-------*----------*----------------*--------------*----------------------------------------------------------------------------
  role | passing  | <redacted>     | 2h36m ago    | readonly
-------*----------*----------------*--------------*----------------------------------------------------------------------------
  vm   | critical | <redacted>     | 2h35m ago    | 500 Internal Server Error
       |          |                |              | [✗] checkDisk: 98.79 MB (10.0%!)(MISSING) free space on /data/ (127.67µs)
       |          |                |              | [✓] checkLoad: load averages: 0.07 0.06 0.05 (62.02µs)
       |          |                |              | [✓] memory: system spent 0s of the last 60s waiting on memory (25.66µs)
       |          |                |              | [✓] cpu: system spent 924ms of the last 60s waiting on cpu (15.19µs)
       |          |                |              | [✓] io: system spent 24ms of the last 60s waiting on io (14.97µs)
-------*----------*----------------*--------------*----------------------------------------------------------------------------

I am not able to connect to the database also, or even restart it.

❯ fly pg connect --app <app_name> --database <database_name>
Error no active leader found

❯ fly pg restart --app <app_name>         
Error no active leader found

CLI version

❯ fly version
fly v0.0.441 darwin/amd64 Commit: 7a9ec7be BuildDate: 2022-12-16T17:47:35Z

This was deployed on November 22, 2022. I deployed it using the guide here.

I will appreciate any suggestion here. Thanks in advance!

2

Hi @zafe, it looks like your database disk is nearly full, from the checkDisk warning you have there in the vm health check.

Similar: Unable to connect to postgres via fly postgres connect, or proxy. - #2 by catflydotio

3

Thank you! That fixed it!
I extended the volume and restarted the machine using the flyctl machine command as described here.
I then restarted all my fly apps that depended on the db.
I was thrown off by the fact that the db wasn’t completely full and thought that the issue was somewhere else.

1 Like