postgres database not reachable

1

Just noticed a down period of maybe 20 minutes or so for the postgres database machine. Here are some logs which might be relevant. Is this something I could fix myself if this happens again?


2023-08-09T13:49:05.075 app[9080543a6e1668] ams [info] proxy | [WARNING] 220/134905 (280) : parsing [/fly/haproxy.cfg:38]: Missing LF on last line, file might have been truncated at position 96. This will become a hard error in HAProxy 2.3.

....

2023-08-09T13:49:06.132 app[9080543a6e1668] ams [info] keeper | 2023-08-09T13:49:06.132Z FATAL cmd/keeper.go:2118 cannot create keeper: cannot create store: cannot create kv store: Put "https://consul-fra.fly-shared.net/v1/catalog/register?wait=5000ms": dial tcp [2a09:8280:1::3:c1f]:443: connect: connection refused

2023-08-09T13:49:06.133 app[9080543a6e1668] ams [info] keeper | exit status 1

2023-08-09T13:49:06.133 app[9080543a6e1668] ams [info] keeper | restarting in 5s [attempt 1]

2023-08-09T13:49:06.133 app[9080543a6e1668] ams [info] sentinel | 2023-08-09T13:49:06.132Z FATAL cmd/sentinel.go:2030 cannot create sentinel: cannot create store: cannot create kv store: Put "https://consul-fra.fly-shared.net/v1/catalog/register?wait=5000ms": dial tcp [2a09:8280:1::3:c1f]:443: connect: connection refused

2023-08-09T13:49:06.134 app[9080543a6e1668] ams [info] sentinel | exit status 1

2023-08-09T13:49:06.134 app[9080543a6e1668] ams [info] sentinel | restarting in 3s [attempt 1]

2023-08-09T13:49:06.137 app[9080543a6e1668] ams [info] panic: error checking stolon status: cannot create kv store: Put "https://consul-fra.fly-shared.net/v1/catalog/register?wait=5000ms": dial tcp [2a09:8280:1::3:c1f]:443: connect: connection refused
2

On my Fly dashboard there is a message that a host is down, affected one of my postgres instances in AMS region. Maybe your instance was on the same host. Check your dashboard

image
image1032×506 23.1 KB

3

still can’t deploy, when trying to run migrations database connections are timing out as part of fly deploy.

  ** (DBConnection.ConnectionError) connection not available and request was dropped from queue after 2977ms. This means requests are coming in and your connection pool cannot serve them fast enough. You can address this by:

not sure if related

 Configuring firecracker
  [    0.035394] PCI: Fatal: No config space access function found

note: The app itself is running and is able to access the db.

4

Still can’t run migrations - is there anything I can do about it? Or is this a broader platform issue?

5

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.