At 1:20 EST, my pg database was unexpectedly updated by the “Fly Admin Bot”
Release by Fly Admin Bot 41 minutes
Released v2
Secret by Fly Admin Bot 41 minutes Set secret FLY_CONSUL_URL
Releaseby Fly Admin Bot 41 minutes
Released v1
Since then it keeps going suspended and restarting over and over. Logs show
2023-03-09T18:56:28Z app[732874c15de859] iad [info]Starting clean up.
2023-03-09T18:56:28Z app[732874c15de859] iad [info]Umounting /dev/vdb from /data
2023-03-09T18:56:29Z app[732874c15de859] iad [info][ 3.115585] reboot: Restarting system
2023-03-09T18:57:09Z app[732874c15de859] iad [info]Starting init (commit: 08b4c2b)…
2023-03-09T18:57:09Z app[732874c15de859] iad [info]Mounting /dev/vdb at /data w/ uid: 0, gid: 0 and chmod 0755
2023-03-09T18:57:09Z app[732874c15de859] iad [info]Preparing to run: docker-entrypoint.sh start as root
2023-03-09T18:57:09Z app[732874c15de859] iad [info]2023/03/09 18:57:09 listening on [fdaa:0:6643:a7b:ab9:d07c:33de:2]:22 (DNS: [fdaa::3]:53)
2023-03-09T18:57:10Z app[732874c15de859] iad [info]cluster spec filename /fly/cluster-spec.json
2023-03-09T18:57:10Z app[732874c15de859] iad [info]system memory: 256mb vcpu count: 1
2023-03-09T18:57:10Z app[732874c15de859] iad [info]{
2023-03-09T18:57:10Z app[732874c15de859] iad [info] “initMode”: “existing”,
2023-03-09T18:57:10Z app[732874c15de859] iad [info] “existingConfig”: {
2023-03-09T18:57:10Z app[732874c15de859] iad [info] “keeperUID”: “ab926aa32”
2023-03-09T18:57:10Z app[732874c15de859] iad [info] },
2023-03-09T18:57:10Z app[732874c15de859] iad [info] “pgParameters”: {
2023-03-09T18:57:10Z app[732874c15de859] iad [info] “archive_command”: “if [ $ENABLE_WALG ]; then /usr/local/bin/wal-g wal-push "%p"; fi”,
2023-03-09T18:57:10Z app[732874c15de859] iad [info] “archive_mode”: “on”,
2023-03-09T18:57:10Z app[732874c15de859] iad [info] “archive_timeout”: “60”,
2023-03-09T18:57:10Z app[732874c15de859] iad [info] “effective_cache_size”: “192MB”,
2023-03-09T18:57:10Z app[732874c15de859] iad [info] “effective_io_concurrency”: “200”,
2023-03-09T18:57:10Z app[732874c15de859] iad [info] “maintenance_work_mem”: “64MB”,
2023-03-09T18:57:10Z app[732874c15de859] iad [info] “max_connections”: “300”,
2023-03-09T18:57:10Z app[732874c15de859] iad [info] “max_parallel_workers”: “8”,
2023-03-09T18:57:10Z app[732874c15de859] iad [info] “max_parallel_workers_per_gather”: “2”,
2023-03-09T18:57:10Z app[732874c15de859] iad [info] “max_worker_processes”: “8”,
2023-03-09T18:57:10Z app[732874c15de859] iad [info] “random_page_cost”: “1.1”,
2023-03-09T18:57:10Z app[732874c15de859] iad [info] “shared_buffers”: “64MB”,
2023-03-09T18:57:10Z app[732874c15de859] iad [info] “wal_compression”: “on”,
2023-03-09T18:57:10Z app[732874c15de859] iad [info] “work_mem”: “4MB”
2023-03-09T18:57:10Z app[732874c15de859] iad [info] },
2023-03-09T18:57:10Z app[732874c15de859] iad [info] “maxStandbysPerSender”: 50,
2023-03-09T18:57:10Z app[732874c15de859] iad [info] “deadKeeperRemovalInterval”: “1h”
2023-03-09T18:57:10Z app[732874c15de859] iad [info]}
2023-03-09T18:57:10Z app[732874c15de859] iad [info]generated new config
2023-03-09T18:57:10Z app[732874c15de859] iad [info]keeper | Running…
2023-03-09T18:57:10Z app[732874c15de859] iad [info]sentinel | Running…
2023-03-09T18:57:10Z app[732874c15de859] iad [info]proxy | Running…
2023-03-09T18:57:10Z app[732874c15de859] iad [info]exporter | Running…
2023-03-09T18:57:10Z app[732874c15de859] iad [info]exporter | INFO[0000] Starting Server: :9187 source=“postgres_exporter.go:1837”
2023-03-09T18:57:10Z app[732874c15de859] iad [info]proxy | [WARNING] 067/185710 (540) : parsing [/fly/haproxy.cfg:38]: Missing LF on last line, file might have been truncated at position 96. This will become a hard error in HAProxy 2.3.
2023-03-09T18:57:10Z app[732874c15de859] iad [info]proxy | [NOTICE] 067/185710 (540) : New worker #1 (562) forked
2023-03-09T18:57:10Z app[732874c15de859] iad [info]proxy | [WARNING] 067/185710 (562) : bk_db/pg1 changed its IP from (none) to fdaa:0:6643:a7b:ab9:d07c:33de:2 by flydns/dns1.
2023-03-09T18:57:10Z app[732874c15de859] iad [info]proxy | [WARNING] 067/185710 (562) : Server bk_db/pg1 (‘iad.seth-db.internal’) is UP/READY (resolves again).
2023-03-09T18:57:10Z app[732874c15de859] iad [info]proxy | [WARNING] 067/185710 (562) : Server bk_db/pg1 administratively READY thanks to valid DNS answer.
2023-03-09T18:57:10Z app[732874c15de859] iad [info]keeper | 2023-03-09T18:57:10.454Z ERROR cmd/keeper.go:811 error retrieving cluster data {“error”: “invalid character ‘\x1e’ in string literal”}
2023-03-09T18:57:10Z app[732874c15de859] iad [info]keeper | 2023-03-09T18:57:10.471Z ERROR cmd/keeper.go:719 cannot get configured pg parameters {“error”: “dial unix /tmp/.s.PGSQL.5433: connect: no such file or directory”}
2023-03-09T18:57:10Z app[732874c15de859] iad [info]sentinel | 2023-03-09T18:57:10.494Z ERROR cmd/sentinel.go:1852 error retrieving cluster data {“error”: “invalid character ‘\x1e’ in string literal”}
2023-03-09T18:57:10Z app[732874c15de859] iad [info]keeper | 2023-03-09T18:57:10.496Z ERROR cmd/keeper.go:1041 error retrieving cluster data {“error”: “invalid character ‘\x1e’ in string literal”}
2023-03-09T18:57:11Z app[732874c15de859] iad [info]checking stolon status
2023-03-09T18:57:11Z app[732874c15de859] iad [info]panic: error checking stolon status: cannot get cluster data: invalid character ‘\x1e’ in string literal
2023-03-09T18:57:11Z app[732874c15de859] iad [info]: exit status 1
2023-03-09T18:57:11Z app[732874c15de859] iad [info]goroutine 9 [running]:
2023-03-09T18:57:11Z app[732874c15de859] iad [info]main.main.func2(0xc0000d0000, 0xc000084a00)
2023-03-09T18:57:11Z app[732874c15de859] iad [info] /go/src/github.com/fly-examples/postgres-ha/cmd/start/main.go:81 +0x72c
2023-03-09T18:57:11Z app[732874c15de859] iad [info]created by main.main
2023-03-09T18:57:11Z app[732874c15de859] iad [info] /go/src/github.com/fly-examples/postgres-ha/cmd/start/main.go:72 +0x43b
2023-03-09T18:57:11Z app[732874c15de859] iad [info]Starting clean up.
2023-03-09T18:57:11Z app[732874c15de859] iad [info]Umounting /dev/vdb from /data
2023-03-09T18:57:12Z app[732874c15de859] iad [info][ 3.120499] reboot: Restarting system
Fly status page shows no issues. Any suggestions?