Postgres instance randomly dies and fails to restart

1

Startup fails and keeps spamming these logs until I run fly machine restart

2022-12-29T20:40:30.428 app[5683072c61118e] fra [info] github.com/fly-examples/postgres-ha/pkg/check.(*Check).RawResult(0xc000093c00, 0x8b5c88, 0xc00002e000)
2022-12-29T20:40:30.428 app[5683072c61118e] fra [info] /go/src/github.com/fly-examples/postgres-ha/pkg/check/check.go:60 +0x60
2022-12-29T20:40:30.428 app[5683072c61118e] fra [info] github.com/fly-examples/postgres-ha/pkg/check.(*CheckSuite).RawResult(0xc0002b0a00, 0x44126e, 0xc000199860)
2022-12-29T20:40:30.428 app[5683072c61118e] fra [info] /go/src/github.com/fly-examples/postgres-ha/pkg/check/check_suite.go:84 +0x93
2022-12-29T20:40:30.428 app[5683072c61118e] fra [info] github.com/fly-examples/postgres-ha/pkg/flycheck.handleCheckResponse(0x919308, 0xc0002c20e0, 0xc0002b0a00, 0x91a801)
2022-12-29T20:40:30.428 app[5683072c61118e] fra [info] /go/src/github.com/fly-examples/postgres-ha/pkg/flycheck/checks.go:89 +0x51
2022-12-29T20:40:30.428 app[5683072c61118e] fra [info] github.com/fly-examples/postgres-ha/pkg/flycheck.runRoleCheck(0x919308, 0xc0002c20e0, 0xc000430600)
2022-12-29T20:40:30.428 app[5683072c61118e] fra [info] /go/src/github.com/fly-examples/postgres-ha/pkg/flycheck/checks.go:79 +0x1ba
2022-12-29T20:40:30.428 app[5683072c61118e] fra [info] net/http.HandlerFunc.ServeHTTP(0x8b5390, 0x919308, 0xc0002c20e0, 0xc000430600)
2022-12-29T20:40:30.428 app[5683072c61118e] fra [info] /usr/local/go/src/net/http/server.go:2050 +0x44
2022-12-29T20:40:30.428 app[5683072c61118e] fra [info] net/http.(*ServeMux).ServeHTTP(0xc000082cc0, 0x919308, 0xc0002c20e0, 0xc000430600)
2022-12-29T20:40:30.428 app[5683072c61118e] fra [info] /usr/local/go/src/net/http/server.go:2429 +0x1ad
2022-12-29T20:40:30.428 app[5683072c61118e] fra [info] github.com/go-chi/chi/v5.(*Mux).Mount.func1(0x919308, 0xc0002c20e0, 0xc000430600)
2022-12-29T20:40:30.428 app[5683072c61118e] fra [info] /go/pkg/mod/github.com/go-chi/chi/v5@v5.0.7/mux.go:314 +0x176
2022-12-29T20:40:30.428 app[5683072c61118e] fra [info] net/http.HandlerFunc.ServeHTTP(0xc00005c9a0, 0x919308, 0xc0002c20e0, 0xc000430600)
2022-12-29T20:40:30.428 app[5683072c61118e] fra [info] /usr/local/go/src/net/http/server.go:2050 +0x44
2022-12-29T20:40:30.428 app[5683072c61118e] fra [info] github.com/go-chi/chi/v5.(*Mux).routeHTTP(0xc00005f1a0, 0x919308, 0xc0002c20e0, 0xc000430600)
2022-12-29T20:40:30.428 app[5683072c61118e] fra [info] /go/pkg/mod/github.com/go-chi/chi/v5@v5.0.7/mux.go:442 +0x2a9
2022-12-29T20:40:30.428 app[5683072c61118e] fra [info] net/http.HandlerFunc.ServeHTTP(0xc00001faf0, 0x919308, 0xc0002c20e0, 0xc000430600)
2022-12-29T20:40:30.428 app[5683072c61118e] fra [info] /usr/local/go/src/net/http/server.go:2050 +0x44
2022-12-29T20:40:30.428 app[5683072c61118e] fra [info] github.com/go-chi/chi/v5.(*Mux).ServeHTTP(0xc00005f1a0, 0x919308, 0xc0002c20e0, 0xc000430500)
2022-12-29T20:40:30.428 app[5683072c61118e] fra [info] /go/pkg/mod/github.com/go-chi/chi/v5@v5.0.7/mux.go:88 +0x310
2022-12-29T20:40:30.428 app[5683072c61118e] fra [info] net/http.serverHandler.ServeHTTP(0xc000136000, 0x919308, 0xc0002c20e0, 0xc000430500)
2022-12-29T20:40:30.428 app[5683072c61118e] fra [info] /usr/local/go/src/net/http/server.go:2868 +0xa3
2022-12-29T20:40:30.428 app[5683072c61118e] fra [info] net/http.(*conn).serve(0xc0001350e0, 0x91a880, 0xc000372740)
2022-12-29T20:40:30.428 app[5683072c61118e] fra [info] /usr/local/go/src/net/http/server.go:1933 +0x8cd
2022-12-29T20:40:30.428 app[5683072c61118e] fra [info] created by net/http.(*Server).Serve
2022-12-29T20:40:30.428 app[5683072c61118e] fra [info] /usr/local/go/src/net/http/server.go:2994 +0x39b
2022-12-29T20:40:31.396 app[5683072c61118e] fra [info] sentinel | 2022-12-29T20:40:31.395Z WARN cmd/sentinel.go:276 no keeper info available {"db": "fa0a5306", "keeper": "23c670f515e62"}
2022-12-29T20:40:31.398 app[5683072c61118e] fra [info] sentinel | 2022-12-29T20:40:31.398Z ERROR cmd/sentinel.go:1018 no eligible masters
2022-12-29T20:40:36.254 app[5683072c61118e] fra [info] 2022/12/29 20:40:36 http: panic serving [fdaa:0:cdd4:a7b:23c6:70f5:15e6:2]:35712: runtime error: invalid memory address or nil pointer dereference
2022-12-29T20:40:36.254 app[5683072c61118e] fra [info] goroutine 311298 [running]:
2022-12-29T20:40:36.254 app[5683072c61118e] fra [info] net/http.(*conn).serve.func1(0xc000135180)
2022-12-29T20:40:36.254 app[5683072c61118e] fra [info] /usr/local/go/src/net/http/server.go:1805 +0x153
2022-12-29T20:40:36.254 app[5683072c61118e] fra [info] panic(0x8270a0, 0xb41c70)
2022-12-29T20:40:36.254 app[5683072c61118e] fra [info] /usr/local/go/src/runtime/panic.go:971 +0x499
2022-12-29T20:40:36.254 app[5683072c61118e] fra [info] github.com/fly-examples/postgres-ha/pkg/check.(*Check).RawResult(0xc000093e30, 0x8b5c88, 0xc00002e000)
2022-12-29T20:40:36.254 app[5683072c61118e] fra [info] /go/src/github.com/fly-examples/postgres-ha/pkg/check/check.go:60 +0x60
2022-12-29T20:40:36.254 app[5683072c61118e] fra [info] github.com/fly-examples/postgres-ha/pkg/check.(*CheckSuite).RawResult(0xc0002d05f0, 0x44126e, 0xc000199860)
2022-12-29T20:40:36.254 app[5683072c61118e] fra [info] /go/src/github.com/fly-examples/postgres-ha/pkg/check/check_suite.go:84 +0x93
2022-12-29T20:40:36.254 app[5683072c61118e] fra [info] github.com/fly-examples/postgres-ha/pkg/flycheck.handleCheckResponse(0x919308, 0xc0002c2380, 0xc0002d05f0, 0x91a801)
2022-12-29T20:40:36.254 app[5683072c61118e] fra [info] /go/src/github.com/fly-examples/postgres-ha/pkg/flycheck/checks.go:89 +0x51
2022-12-29T20:40:36.254 app[5683072c61118e] fra [info] github.com/fly-examples/postgres-ha/pkg/flycheck.runRoleCheck(0x919308, 0xc0002c2380, 0xc000430900)
2022-12-29T20:40:36.254 app[5683072c61118e] fra [info] /go/src/github.com/fly-examples/postgres-ha/pkg/flycheck/checks.go:79 +0x1ba
2022-12-29T20:40:36.254 app[5683072c61118e] fra [info] net/http.HandlerFunc.ServeHTTP(0x8b5390, 0x919308, 0xc0002c2380, 0xc000430900)
2022-12-29T20:40:36.254 app[5683072c61118e] fra [info] /usr/local/go/src/net/http/server.go:2050 +0x44
2022-12-29T20:40:36.254 app[5683072c61118e] fra [info] net/http.(*ServeMux).ServeHTTP(0xc000082cc0, 0x919308, 0xc0002c2380, 0xc000430900)
2022-12-29T20:40:36.254 app[5683072c61118e] fra [info] /usr/local/go/src/net/http/server.go:2429 +0x1ad
2022-12-29T20:40:36.254 app[5683072c61118e] fra [info] github.com/go-chi/chi/v5.(*Mux).Mount.func1(0x919308, 0xc0002c2380, 0xc000430900)
2022-12-29T20:40:36.254 app[5683072c61118e] fra [info] /go/pkg/mod/github.com/go-chi/chi/v5@v5.0.7/mux.go:314 +0x176
2022-12-29T20:40:36.254 app[5683072c61118e] fra [info] net/http.HandlerFunc.ServeHTTP(0xc00005c9a0, 0x919308, 0xc0002c2380, 0xc000430900)
2022-12-29T20:40:36.254 app[5683072c61118e] fra [info] /usr/local/go/src/net/http/server.go:2050 +0x44
2022-12-29T20:40:36.254 app[5683072c61118e] fra [info] github.com/go-chi/chi/v5.(*Mux).routeHTTP(0xc00005f1a0, 0x919308, 0xc0002c2380, 0xc000430900)
2022-12-29T20:40:36.254 app[5683072c61118e] fra [info] /go/pkg/mod/github.com/go-chi/chi/v5@v5.0.7/mux.go:442 +0x2a9
2022-12-29T20:40:36.254 app[5683072c61118e] fra [info] net/http.HandlerFunc.ServeHTTP(0xc00001faf0, 0x919308, 0xc0002c2380, 0xc000430900)
2022-12-29T20:40:36.254 app[5683072c61118e] fra [info] /usr/local/go/src/net/http/server.go:2050 +0x44
2022-12-29T20:40:36.254 app[5683072c61118e] fra [info] github.com/go-chi/chi/v5.(*Mux).ServeHTTP(0xc00005f1a0, 0x919308, 0xc0002c2380, 0xc000430800)
2022-12-29T20:40:36.254 app[5683072c61118e] fra [info] /go/pkg/mod/github.com/go-chi/chi/v5@v5.0.7/mux.go:88 +0x310
2022-12-29T20:40:36.254 app[5683072c61118e] fra [info] net/http.serverHandler.ServeHTTP(0xc000136000, 0x919308, 0xc0002c2380, 0xc000430800)
2022-12-29T20:40:36.254 app[5683072c61118e] fra [info] /usr/local/go/src/net/http/server.go:2868 +0xa3
2022-12-29T20:40:36.254 app[5683072c61118e] fra [info] net/http.(*conn).serve(0xc000135180, 0x91a880, 0xc0003732c0)
2022-12-29T20:40:36.254 app[5683072c61118e] fra [info] /usr/local/go/src/net/http/server.go:1933 +0x8cd
2022-12-29T20:40:36.254 app[5683072c61118e] fra [info] created by net/http.(*Server).Serve
2022-12-29T20:40:36.254 app[5683072c61118e] fra [info] /usr/local/go/src/net/http/server.go:2994 +0x39b
2022-12-29T20:40:36.616 app[5683072c61118e] fra [info] 2022/12/29 20:40:36 http: panic serving [fdaa:0:cdd4:a7b:23c6:70f5:15e6:2]:35714: runtime error: invalid memory address or nil pointer dereference
2022-12-29T20:40:36.616 app[5683072c61118e] fra [info] goroutine 311308 [running]:
2022-12-29T20:40:36.616 app[5683072c61118e] fra [info] net/http.(*conn).serve.func1(0xc000135220)
2022-12-29T20:40:36.616 app[5683072c61118e] fra [info] /usr/local/go/src/net/http/server.go:1805 +0x153
2022-12-29T20:40:36.616 app[5683072c61118e] fra [info] panic(0x8270a0, 0xb41c70)
2022-12-29T20:40:36.616 app[5683072c61118e] fra [info] /usr/local/go/src/runtime/panic.go:971 +0x499
2022-12-29T20:40:36.616 app[5683072c61118e] fra [info] github.com/fly-examples/postgres-ha/pkg/check.(*Check).RawResult(0xc000340070, 0x8b5c88, 0xc00002e000)
2022-12-29T20:40:36.616 app[5683072c61118e] fra [info] /go/src/github.com/fly-examples/postgres-ha/pkg/check/check.go:60 +0x60
2022-12-29T20:40:36.616 app[5683072c61118e] fra [info] github.com/fly-examples/postgres-ha/pkg/check.(*CheckSuite).RawResult(0xc0002d1d60, 0x44126e, 0xc0001bd860)
2022-12-29T20:40:36.616 app[5683072c61118e] fra [info] /go/src/github.com/fly-examples/postgres-ha/pkg/check/check_suite.go:84 +0x93
2022-12-29T20:40:36.616 app[5683072c61118e] fra [info] github.com/fly-examples/postgres-ha/pkg/flycheck.handleCheckResponse(0x919308, 0xc0002c2620, 0xc0002d1d60, 0x91a801)
2022-12-29T20:40:36.616 app[5683072c61118e] fra [info] /go/src/github.com/fly-examples/postgres-ha/pkg/flycheck/checks.go:89 +0x51
2022-12-29T20:40:36.616 app[5683072c61118e] fra [info] github.com/fly-examples/postgres-ha/pkg/flycheck.runRoleCheck(0x919308, 0xc0002c2620, 0xc000430c00)
2022-12-29T20:40:36.616 app[5683072c61118e] fra [info] /go/src/github.com/fly-examples/postgres-ha/pkg/flycheck/checks.go:79 +0x1ba
2022-12-29T20:40:36.616 app[5683072c61118e] fra [info] net/http.HandlerFunc.ServeHTTP(0x8b5390, 0x919308, 0xc0002c2620, 0xc000430c00)
2022-12-29T20:40:36.616 app[5683072c61118e] fra [info] /usr/local/go/src/net/http/server.go:2050 +0x44
2022-12-29T20:40:36.616 app[5683072c61118e] fra [info] net/http.(*ServeMux).ServeHTTP(0xc000082cc0, 0x919308, 0xc0002c2620, 0xc000430c00)
2022-12-29T20:40:36.616 app[5683072c61118e] fra [info] /usr/local/go/src/net/http/server.go:2429 +0x1ad
2022-12-29T20:40:36.616 app[5683072c61118e] fra [info] github.com/go-chi/chi/v5.(*Mux).Mount.func1(0x919308, 0xc0002c2620, 0xc000430c00)
2022-12-29T20:40:36.616 app[5683072c61118e] fra [info] /go/pkg/mod/github.com/go-chi/chi/v5@v5.0.7/mux.go:314 +0x176
2022-12-29T20:40:36.616 app[5683072c61118e] fra [info] net/http.HandlerFunc.ServeHTTP(0xc00005c9a0, 0x919308, 0xc0002c2620, 0xc000430c00)
2022-12-29T20:40:36.616 app[5683072c61118e] fra [info] /usr/local/go/src/net/http/server.go:2050 +0x44
2022-12-29T20:40:36.616 app[5683072c61118e] fra [info] github.com/go-chi/chi/v5.(*Mux).routeHTTP(0xc00005f1a0, 0x919308, 0xc0002c2620, 0xc000430c00)
2022-12-29T20:40:36.616 app[5683072c61118e] fra [info] /go/pkg/mod/github.com/go-chi/chi/v5@v5.0.7/mux.go:442 +0x2a9
2022-12-29T20:40:36.616 app[5683072c61118e] fra [info] net/http.HandlerFunc.ServeHTTP(0xc00001faf0, 0x919308, 0xc0002c2620, 0xc000430c00)
2022-12-29T20:40:36.616 app[5683072c61118e] fra [info] /usr/local/go/src/net/http/server.go:2050 +0x44
2022-12-29T20:40:36.616 app[5683072c61118e] fra [info] github.com/go-chi/chi/v5.(*Mux).ServeHTTP(0xc00005f1a0, 0x919308, 0xc0002c2620, 0xc000430b00)
2022-12-29T20:40:36.616 app[5683072c61118e] fra [info] /go/pkg/mod/github.com/go-chi/chi/v5@v5.0.7/mux.go:88 +0x310
2022-12-29T20:40:36.616 app[5683072c61118e] fra [info] net/http.serverHandler.ServeHTTP(0xc000136000, 0x919308, 0xc0002c2620, 0xc000430b00)
2022-12-29T20:40:36.616 app[5683072c61118e] fra [info] /usr/local/go/src/net/http/server.go:2868 +0xa3
2022-12-29T20:40:36.616 app[5683072c61118e] fra [info] net/http.(*conn).serve(0xc000135220, 0x91a880, 0xc000373e80)
2022-12-29T20:40:36.616 app[5683072c61118e] fra [info] /usr/local/go/src/net/http/server.go:1933 +0x8cd
2022-12-29T20:40:36.616 app[5683072c61118e] fra [info] created by net/http.(*Server).Serve
2022-12-29T20:40:36.616 app[5683072c61118e] fra [info] /usr/local/go/src/net/http/server.go:2994 +0x39b
2022-12-29T20:40:39.512 app[5683072c61118e] fra [info] sentinel | 2022-12-29T20:40:39.512Z WARN cmd/sentinel.go:276 no keeper info available {"db": "fa0a5306", "keeper": "23c670f515e62"}
2022-12-29T20:40:39.514 app[5683072c61118e] fra [info] sentinel | 2022-12-29T20:40:39.514Z ERROR cmd/sentinel.go:1018 no eligible masters

machine specs are:

  Region        = fra                               
  Process Group =                                   
  Memory        = 1024                              
  CPUs          = 1

Is there anything I can do to fix it?

2

Can someone help me investigate this problem?
machine id is 5683072c61118e
volume id is vol_wod56vjqpko4ny30

3

Hi @ppraisethesum, did you find a solution already? I see the machine is running ok and sql queries are executing fine according to logs.

4

No, I did not. I had to restart the machine manually. It does not start unless I do that.

5

Not sure I understand then, what do you mean by “startup fails … until the machine is restarted” ?

6

It just keeps spamming the logs from the post and my app cannot connect to it until I restart the machine