BackendUnavailableError when trying to connect to Fly.io EdgeDB instance

  • EdgeDB Version: EdgeDB CLI 2.0.5+53f9c9b
  • OS Version: Mac OS & Amazon Linux 2

Steps to Reproduce:

  1. Try to connect to EdgeDB instance setup on Fly.io, that worked successfully a while before. Nothing has changed since then.
  2. Put in the command
edgedb instance link \
  --trust-tls-cert \
  --dsn $DSN \
  --non-interactive \
  fly

with the proper DSN variable. It tries to connect and spits out the following error.

edgedb error: BackendUnavailableError: cannot fetch database version: Postgres is not available: Connection is lost, please check server log for the reason.
  Server traceback:
      Traceback (most recent call last):
        File "edb/server/protocol/binary.pyx", line 1364, in edb.server.protocol.binary.EdgeConnection.main
        File "edb/server/protocol/binary.pyx", line 1270, in execute
        File "edb/server/protocol/binary.pyx", line 1073, in _execute
        File "edb/server/protocol/binary.pyx", line 258, in get_pgcon
        File "edb/server/protocol/binary.pyx", line 252, in edb.server.protocol.binary.EdgeConnection.get_pgcon
        File "/usr/lib/x86_64-linux-gnu/edgedb-server-2/lib/python3.10/site-packages/edb/server/server.py", line 552, in acquire_pgcon
          raise errors.BackendUnavailableError(
      edb.errors.BackendUnavailableError: Postgres is not available: Connection is lost, please check server log for the reason.

Running fly checks list -a for both the fly.io EdgeDB app and postgres app attached to it. All checkmarks are there meaning it should be good to go.

Not sure what the issue is and how to fix it.

What do the edgedb logs look like? flyctl logs --app $EDB_APP

This is what I get running flyctl logs --app $EDB_POSTGRES

2022-09-20T15:30:25Z app[a9a9d6d8] iad [info]sentinel | 2022-09-20T15:30:25.088Z	ERROR	cmd/sentinel.go:1880	cannot update sentinel info	{"error": "Unexpected response code: 500 (leadership lost while committing log)"}
2022-09-20T15:30:25Z app[a9a9d6d8] iad [info]keeper   | 2022-09-20T15:30:25.488Z	ERROR	cmd/keeper.go:870	failed to update keeper info	{"error": "Unexpected response code: 500 (leadership lost while committing log)"}
2022-09-20T15:30:32Z app[a9a9d6d8] iad [info]sentinel | 2022-09-20T15:30:32.326Z	ERROR	cmd/sentinel.go:102	election loop error	{"error": "Unexpected response code: 500 (No cluster leader)"}
2022-09-20T15:30:32Z app[a9a9d6d8] iad [info]keeper   | 2022-09-20T15:30:32.743Z	ERROR	cmd/keeper.go:1041	error retrieving cluster data	{"error": "Unexpected response code: 500"}
2022-09-20T15:30:53Z app[a9a9d6d8] iad [info]sentinel | panic: close of closed channel
2022-09-20T15:30:53Z app[a9a9d6d8] iad [info]sentinel |
2022-09-20T15:30:53Z app[a9a9d6d8] iad [info]sentinel | goroutine 783128 [running]:
2022-09-20T15:30:53Z app[a9a9d6d8] iad [info]sentinel | github.com/superfly/leadership.(*Candidate).initLock(0xc000468000)
2022-09-20T15:30:53Z app[a9a9d6d8] iad [info]sentinel | 	/go/pkg/mod/github.com/superfly/leadership@v0.2.1/candidate.go:98 +0x2e
2022-09-20T15:30:53Z app[a9a9d6d8] iad [info]sentinel | github.com/superfly/leadership.(*Candidate).campaign(0xc000468000)
2022-09-20T15:30:53Z app[a9a9d6d8] iad [info]sentinel | 	/go/pkg/mod/github.com/superfly/leadership@v0.2.1/candidate.go:124 +0xc6
2022-09-20T15:30:53Z app[a9a9d6d8] iad [info]sentinel | created by github.com/superfly/leadership.(*Candidate).RunForElection
2022-09-20T15:30:53Z app[a9a9d6d8] iad [info]sentinel | 	/go/pkg/mod/github.com/superfly/leadership@v0.2.1/candidate.go:60 +0xbd
2022-09-20T15:30:53Z app[a9a9d6d8] iad [info]sentinel | exit status 2
2022-09-20T15:30:53Z app[a9a9d6d8] iad [info]sentinel | restarting in 3s [attempt 1]
2022-09-20T15:30:56Z app[a9a9d6d8] iad [info]sentinel | Running...
2022-09-20T16:37:38Z app[a9a9d6d8] iad [info]sentinel | 2022-09-20T16:37:38.078Z	WARN	cmd/sentinel.go:276	no keeper info available	{"db": "d9cfc7e0", "keeper": "9d36196dc2"}
2022-09-20T16:53:51Z app[a9a9d6d8] iad [info]sentinel | 2022-09-20T16:53:51.704Z	WARN	cmd/sentinel.go:276	no keeper info available	{"db": "d9cfc7e0", "keeper": "9d36196dc2"}

But, when I run flyctl logs --app $EDB_APP nothing shows up. I just ctrl-c after a few minutes

does restarting the EdgeDB app change anything? flyctl restart $EDB_APP

Found the fix. Needed to scale the postgres cluster up to dedicated-cpu-1x

1 Like

Awesome!

Wait, the minimum requirements for hosting the PG app for an EdgeDB instance is dedicated-cpu-1x ($31/month just to start)? :astonished:

Ya. It sucks, but it is what it is. The guys are working on their own service which I think will launch with 3.0 (in around 6 months from now?).