database stuck after I moved the org

I moved one my app from my personal account into an org acccount (darisini-711), the app name is darisini-supertokens-staging-db

2022-12-28T10:32:58.386 app[0e2861d4fe6986] sin [info] exporter | INFO[0012] Established new database connection to "fdaa:0:3eb6:a7b:80:e227:4b64:2:5433". source="postgres_exporter.go:970"

Update 1

Previously it says no active leader found, I tried to restart the machine. Turns out this postgres already using apps v2 (machine).

Update 2

(base) muhajir@Muhammads-MacBook-Pro-2 darisini-com % fly machine status 0e2861d4fe6986
Machine ID: 0e2861d4fe6986
Instance ID: 01GG7GJS2JRRK8R4FD9D2JF4J7
State: started

VM
  ID            = 0e2861d4fe6986
  Instance ID   = 01GG7GJS2JRRK8R4FD9D2JF4J7
  State         = started
  Image         = flyio/postgres:14.4 (v0.0.32)
  Name          = still-butterfly-205
  Private IP    = fdaa:0:3eb6:a7b:80:e227:4b64:2
  Region        = sin
  Process Group =
  Memory        = 256
  CPUs          = 1
  Created       = 2022-10-25T12:06:24Z
  Updated       = 2022-12-28T11:46:48Z
  Command       =
  Volume        = vol_2n0l9vlkg6pr635d

Event Logs
STATE   	EVENT  	SOURCE	TIMESTAMP                    	INFO
started 	start  	flyd  	2022-12-28T18:46:48.423+07:00
started 	start  	flyd  	2022-12-28T18:46:48.027+07:00
starting	restart	flyd  	2022-12-28T18:46:47.579+07:00
stopped 	exit   	flyd  	2022-12-28T18:46:46.667+07:00
stopping	restart	user  	2022-12-28T18:46:44.322+07:00

(base) muhajir@Muhammads-MacBook-Pro-2 darisini-com % fly pg connect -a darisini-supertokens-staging-db
2022-12-28T10:32:58.411 app[0e2861d4fe6986] sin [info] exporter | INFO[0012] Semantic Version Changed on "fdaa:0:3eb6:a7b:80:e227:4b64:2:5433": 0.0.0 -> 14.4.0 source="postgres_exporter.go:1539"

2022-12-28T10:32:58.439 app[0e2861d4fe6986] sin [info] exporter | INFO[0012] Established new database connection to "fdaa:0:3eb6:a7b:80:e227:4b64:2:5433". source="postgres_exporter.go:970"

2022-12-28T10:32:58.465 app[0e2861d4fe6986] sin [info] exporter | INFO[0012] Semantic Version Changed on "fdaa:0:3eb6:a7b:80:e227:4b64:2:5433": 0.0.0 -> 14.4.0 source="postgres_exporter.go:1539"

2022-12-28T10:33:03.760 app[0e2861d4fe6986] sin [info] proxy | [WARNING] 361/103303 (563) : Backup Server bk_db/pg is UP, reason: Layer7 check passed, code: 200, check duration: 8ms. 0 active and 1 backup servers online. Running on backup. 0 sessions requeued, 0 total in queue.

here’s the error log, now the database stuck

I tried to restart, but it’s not working

Update 1

  • Previously it says “no active leader found”. I tried to restart the machine, turns out this database is using apps v2 (machine)

Update 2

  • It’s now stuck when trying to connect to database
(base) muhajir@Muhammads-MacBook-Pro-2 darisini-com % fly machine status 0e2861d4fe6986
Machine ID: 0e2861d4fe6986
Instance ID: 01GG7GJS2JRRK8R4FD9D2JF4J7
State: started

VM
  ID            = 0e2861d4fe6986
  Instance ID   = 01GG7GJS2JRRK8R4FD9D2JF4J7
  State         = started
  Image         = flyio/postgres:14.4 (v0.0.32)
  Name          = still-butterfly-205
  Private IP    = fdaa:0:3eb6:a7b:80:e227:4b64:2
  Region        = sin
  Process Group =
  Memory        = 256
  CPUs          = 1
  Created       = 2022-10-25T12:06:24Z
  Updated       = 2022-12-28T11:46:48Z
  Command       =
  Volume        = vol_2n0l9vlkg6pr635d

Event Logs
STATE   	EVENT  	SOURCE	TIMESTAMP                    	INFO
started 	start  	flyd  	2022-12-28T18:46:48.423+07:00
started 	start  	flyd  	2022-12-28T18:46:48.027+07:00
starting	restart	flyd  	2022-12-28T18:46:47.579+07:00
stopped 	exit   	flyd  	2022-12-28T18:46:46.667+07:00
stopping	restart	user  	2022-12-28T18:46:44.322+07:00

(base) muhajir@Muhammads-MacBook-Pro-2 darisini-com % fly pg connect -a darisini-supertokens-staging-db

Connecting to fdaa:0:3eb6:a7b:80:e227:4b64:2...⣯Error error connecting to SSH server: connect tcp [fdaa:0:3eb6:a7b:80:e227:4b64:2]:22: operation timed out


Connecting to fdaa:0:3eb6:a7b:80:e227:4b64:2... complete

Hello. Your postgres app looks healthy, I tried to connect to and it worked fine

~$ export FLY_APP=darisini-supertokens-staging-db
~$ fly status
ID              STATE   ROLE    REGION  HEALTH CHECKS           IMAGE                           CREATED                 UPDATED
0e2861d4fe6986  started leader  sin     3 total, 3 passing      flyio/postgres:14.4 (v0.0.33)   2022-10-25T12:06:24Z    2022-12-28T12:44:00Z
e784e5ea210983  started replica sin     3 total, 3 passing      flyio/postgres:14.4 (v0.0.33)   2022-12-28T12:17:10Z    2022-12-28T12:38:57Z


~$ fly pg connect
Connecting to fdaa:1:140e:a7b:80:d907:571e:2... complete
psql (14.4 (Debian 14.4-1.pgdg110+1))
Type "help" for help.

postgres=#

This is the interesting part:

that IP isn’t listed in my case for machines in your app:

~$ fly ssh console -s
? Select VM:  [Use arrows to move, type to filter]
> sin: e784e5ea210983 fdaa:1:140e:a7b:f0f:2a23:1523:2 sparkling-feather-5741
  sin: 0e2861d4fe6986 fdaa:1:140e:a7b:80:d907:571e:2 still-butterfly-205