flyctl deploy does not seem to update the DB nor the deployment version

doingitnow · October 31, 2022, 11:37am

Hello I am new to fly.io and I am trying to set up an external connection to the DB. I am running into some issues where i try to flyctl deploy my postgres DB and it seems to have worked but nothing has changed in the DB and the deployment version has not been updated. I will give an extensive summary of what I am trying to achieve and steps to replicate my issue

I am trying to perform a db script in Python on my laptop (no fly commands) and it seems I need to setup and IP address and open the external ports.

Doco I am referring to: Connecting to Apps · Fly Docs.

Flyctl Version: flyctl v0.0.425 darwin/amd64 Commit: a1d61bdc BuildDate: 2022-10-29T20:24:10Z

The steps I am doing is as follows

Run fly pg create -n "myapp-db" I am based in australia but I choose Tokyo Region, Development DB
Run fly ips allocate-v4 -a myapp-db. This is to create the IPv4 address for the external connection
fly config save --app myapp-db to get the toml file
Update this toml file to look something like this. This is based from the doco above where you set the external port you want to use to access the DB

app = "myapp-db"
kill_signal = "SIGINT"
kill_timeout = 5
processes = []

[env]

[experimental]
  allowed_public_ports = []
  auto_rollback = true

[[services]]
  http_checks = []
  internal_port = 5432 # Postgres instance
  protocol = "tcp"
  script_checks = []
  [services.concurrency]
    hard_limit = 25
    soft_limit = 20
    type = "connections"

  # Open port 10000 for plaintext connections.
  [[services.ports]]
    handlers = []
    port = 10000

  [[services.ports]]
    force_https = true
    handlers = ["http"]
    port = 80

  [[services.ports]]
    handlers = ["tls", "http"]
    port = 443

  [[services.tcp_checks]]
    grace_period = "1s"
    interval = "15s"
    restart_limit = 0
    timeout = "2s"

Run fly deploy . --app myapp-db --image flyio/postgres:14

When i do the fly deploy I see in the monitoring tab it is doing some work. However, when I do fly info -a myapp-db I get this

App
  Name     = myapp-db
  Owner    = personal
  Version  = 0
  Status   = deployed
  Hostname = myapp-db.fly.dev

Services
PROTOCOL PORTS

The version is not updated and the port configs set is not done.

Upon looking at the logs I see a bunch of errors that do not make sense to me. Googling various errors has not helped me figure out what I have done wrong.

The log trace when it was performing the deploy

2022-10-31T11:27:21.621 app[39080775bd4d87] nrt [info] keeper | Running...

2022-10-31T11:27:21.623 app[39080775bd4d87] nrt [info] sentinel | Running...

2022-10-31T11:27:21.623 app[39080775bd4d87] nrt [info] proxy | Running...

2022-10-31T11:27:21.628 app[39080775bd4d87] nrt [info] exporter | Running...

2022-10-31T11:27:21.779 app[39080775bd4d87] nrt [info] exporter | INFO[0000] Starting Server: :9187 source="postgres_exporter.go:1837"

2022-10-31T11:27:21.798 app[39080775bd4d87] nrt [info] proxy | [WARNING] 303/112721 (538) : parsing [/fly/haproxy.cfg:38]: Missing LF on last line, file might have been truncated at position 96. This will become a hard error in HAProxy 2.3.

2022-10-31T11:27:21.814 app[39080775bd4d87] nrt [info] proxy | [NOTICE] 303/112721 (538) : New worker #1 (560) forked

2022-10-31T11:27:21.826 app[39080775bd4d87] nrt [info] proxy | [WARNING] 303/112721 (560) : bk_db/pg1 changed its IP from (none) to fdaa:0:cb85:a7b:db52:369d:8984:2 by flydns/dns1.

2022-10-31T11:27:21.828 app[39080775bd4d87] nrt [info] proxy | [WARNING] 303/112721 (560) : Server bk_db/pg1 ('nrt.myapp-db.internal') is UP/READY (resolves again).

2022-10-31T11:27:21.828 app[39080775bd4d87] nrt [info] proxy | [WARNING] 303/112721 (560) : Server bk_db/pg1 administratively READY thanks to valid DNS answer.

2022-10-31T11:27:22.254 app[39080775bd4d87] nrt [info] keeper | 2022-10-31T11:27:22.251Z ERROR cmd/keeper.go:719 cannot get configured pg parameters {"error": "dial unix /tmp/.s.PGSQL.5433: connect: no such file or directory"}

2022-10-31T11:27:22.621 app[39080775bd4d87] nrt [info] checking stolon status

2022-10-31T11:27:23.110 app[39080775bd4d87] nrt [info] keeper is healthy, db is healthy, role: master

2022-10-31T11:27:24.752 app[39080775bd4d87] nrt [info] keeper | 2022-10-31T11:27:24.752Z ERROR cmd/keeper.go:719 cannot get configured pg parameters {"error": "dial unix /tmp/.s.PGSQL.5433: connect: no such file or directory"}

2022-10-31T11:27:27.254 app[39080775bd4d87] nrt [info] keeper | 2022-10-31T11:27:27.253Z ERROR cmd/keeper.go:719 cannot get configured pg parameters {"error": "dial unix /tmp/.s.PGSQL.5433: connect: no such file or directory"}

2022-10-31T11:27:28.110 app[39080775bd4d87] nrt [info] error connecting to local postgres context deadline exceeded

2022-10-31T11:27:28.110 app[39080775bd4d87] nrt [info] checking stolon status

2022-10-31T11:27:28.377 app[39080775bd4d87] nrt [info] keeper is healthy, db is healthy, role: master

2022-10-31T11:27:28.619 app[39080775bd4d87] nrt [info] proxy | [WARNING] 303/112728 (560) : Backup Server bk_db/pg is DOWN, reason: Layer7 timeout, check duration: 5000ms. 1 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.

2022-10-31T11:27:28.800 app[39080775bd4d87] nrt [info] proxy | [WARNING] 303/112728 (560) : Server bk_db/pg1 is DOWN, reason: Layer7 timeout, check duration: 5000ms. 0 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.

2022-10-31T11:27:28.800 app[39080775bd4d87] nrt [info] proxy | [NOTICE] 303/112728 (560) : haproxy version is 2.2.9-2+deb11u3

2022-10-31T11:27:28.800 app[39080775bd4d87] nrt [info] proxy | [NOTICE] 303/112728 (560) : path to executable is /usr/sbin/haproxy

2022-10-31T11:27:28.800 app[39080775bd4d87] nrt [info] proxy | [ALERT] 303/112728 (560) : backend 'bk_db' has no server available!

2022-10-31T11:27:29.754 app[39080775bd4d87] nrt [info] keeper | 2022-10-31T11:27:29.754Z ERROR cmd/keeper.go:719 cannot get configured pg parameters {"error": "dial unix /tmp/.s.PGSQL.5433: connect: no such file or directory"}

2022-10-31T11:27:32.256 app[39080775bd4d87] nrt [info] keeper | 2022-10-31T11:27:32.255Z ERROR cmd/keeper.go:719 cannot get configured pg parameters {"error": "dial unix /tmp/.s.PGSQL.5433: connect: no such file or directory"}

2022-10-31T11:27:32.922 app[39080775bd4d87] nrt [info] keeper | 2022-10-31 11:27:32.921 UTC [588] LOG: starting PostgreSQL 14.4 (Debian 14.4-1.pgdg110+1) on x86_64-pc-linux-gnu, compiled by gcc (Debian 10.2.1-6) 10.2.1 20210110, 64-bit

2022-10-31T11:27:32.922 app[39080775bd4d87] nrt [info] keeper | 2022-10-31 11:27:32.921 UTC [588] LOG: listening on IPv6 address "fdaa:0:cb85:a7b:db52:369d:8984:2", port 5433

2022-10-31T11:27:32.923 app[39080775bd4d87] nrt [info] keeper | 2022-10-31 11:27:32.922 UTC [588] LOG: listening on Unix socket "/tmp/.s.PGSQL.5433"

2022-10-31T11:27:32.926 app[39080775bd4d87] nrt [info] keeper | 2022-10-31 11:27:32.925 UTC [589] LOG: database system was shut down at 2022-10-31 11:27:18 UTC

2022-10-31T11:27:32.930 app[39080775bd4d87] nrt [info] keeper | 2022-10-31 11:27:32.930 UTC [588] LOG: database system is ready to accept connections

2022-10-31T11:27:33.378 app[39080775bd4d87] nrt [info] error connecting to local postgres context deadline exceeded

2022-10-31T11:27:33.378 app[39080775bd4d87] nrt [info] checking stolon status

2022-10-31T11:27:33.713 app[39080775bd4d87] nrt [info] keeper is healthy, db is healthy, role: master

2022-10-31T11:27:33.718 app[39080775bd4d87] nrt [info] configuring operator

2022-10-31T11:27:33.729 app[39080775bd4d87] nrt [info] operator password does not match config, changing

2022-10-31T11:27:33.732 app[39080775bd4d87] nrt [info] operator ready!

2022-10-31T11:27:33.732 app[39080775bd4d87] nrt [info] configuring repluser

2022-10-31T11:27:33.733 app[39080775bd4d87] nrt [info] repluser password does not match config, changing

2022-10-31T11:27:33.735 app[39080775bd4d87] nrt [info] replication ready!

2022-10-31T11:27:39.638 app[39080775bd4d87] nrt [info] proxy | [WARNING] 303/112739 (560) : Backup Server bk_db/pg is UP, reason: Layer7 check passed, code: 200, check duration: 6ms. 0 active and 1 backup servers online. Running on backup. 0 sessions requeued, 0 total in queue.

2022-10-31T11:27:39.819 app[39080775bd4d87] nrt [info] proxy | [WARNING] 303/112739 (560) : Server bk_db/pg1 is UP, reason: Layer7 check passed, code: 200, check duration: 6ms. 1 active and 1 backup servers online. 0 sessions requeued, 0 total in queue.

This is extremely frustrating to figure out the root cause of the issue. I am essentially creating a fresh postgres DB and doing a flyctl deploy to update the config and nothing is happening.

Furthermore doing a

flyctl secrets unset MY_SECRET -a myapp-db

Also shows very similar to exact logs. I can set secrets but I cannot unset.

Any help is appreciated. I am unsure if I updated the fly.toml incorrectly or I am missing some steps (not in the doc given) that is needed to do any DB deploys/update

dangra · October 31, 2022, 12:49pm

Hello. No need to open the postgres port to the world. The recommended way to do this is by running

fly proxy 5432:5432 -a myapp-db

then point your python script to localhost:5432

doingitnow · October 31, 2022, 1:42pm

Ahhh i see. I will try that out. Regardless of that, I can’t seem to update DB with the fly.toml file. Is that something I should be able to do or not

dangra · October 31, 2022, 1:54pm

Hey, there is something fishy for sure because you should be able to update the app from a fly.toml. I think it is related to the recent change to the backend we use to run VMs. Let me know if you need to change anything else, I will take a look at fly.toml for postgres in a bit.

doingitnow · October 31, 2022, 2:13pm

Thanks! I will check your suggestion tomorrow and see if it works. If you can try to replicate the issue with the steps I gave that be good. Wondering if it my config issue or something on the backend

doingitnow · November 2, 2022, 7:37am

Just an update. The proxy command works as expected thanks. I still cannot remove secrets from the DB but that is not a big deal.

Still an issue that probably should be noted somewhere

Topic		Replies	Views
Help with fly db external connection	1	203	July 12, 2023
Can't get Postgres external connection working Questions / Help	7	628	February 20, 2023
Solved. "Push" a fly.toml to postgres created with fly pg create	0	630	August 20, 2022
External Connections to Fly Postgres (Update: Where's my data???) Questions / Help postgres , troubleshooting	7	1521	May 21, 2023
Problem exposing PostgreSQL database to external world postgres	15	668	February 21, 2023

flyctl deploy does not seem to update the DB nor the deployment version

Related Topics