db creation problems

1

I’ve been having problems all day yesterday and now today trying to get even the simplest phoenix app deployed.

So I’ve attempted with a fresh 1.6.7 app and the latest problem is that with this fresh app, Postgres db creation fails during the fly launch script.

I’ve had a few different errors, both a failure to attach, and this

1 desired, 1 placed, 0 healthy, 0 unhealthy [health checks: 3 total, 2 critical]
v0 running - Deployment is running
Failed Instances

==> Failure #1

Instance
  ID            = 0edb61d8                
  Process       =                         
  Version       = 0                       
  Region        = sjc                     
  Desired       = run                     
  Status        = running (failed to co)  
  Health Checks = 3 total, 2 critical     
  Restarts      = 0                       
  Created       = 24s ago                 

Recent Events
TIMESTAMP            TYPE       MESSAGE                 
2022-04-24T23:13:03Z Received   Task received by client 
2022-04-24T23:13:03Z Task Setup Building Task Directory 
2022-04-24T23:13:07Z Started    Task started by client  

Recent Logs
2022-04-24T23:13:26.000 [info] keeper   | 2022-04-24T23:13:26.299Z      INFO    postgresql/postgresql.go:319    starting database
2022-04-24T23:13:26.000 [info] keeper   | 2022-04-24 23:13:26.312 UTC [689] LOG:  starting PostgreSQL 14.2 (Debian 14.2-1.pgdg110+1) on x86_64-pc-linux-gnu, compiled by gcc (Debian 10.2.1-6) 10.2.1 20210110, 64-bit
2022-04-24T23:13:26.000 [info] keeper   | 2022-04-24 23:13:26.312 UTC [689] LOG:  listening on IPv6 address "fdaa:0:5d96:a7b:ad1:0:d6a8:2", port 5433
2022-04-24T23:13:26.000 [info] keeper   | 2022-04-24 23:13:26.313 UTC [689] LOG:  listening on Unix socket "/tmp/.s.PGSQL.5433"
2022-04-24T23:13:26.000 [info] keeper   | 2022-04-24 23:13:26.315 UTC [690] LOG:  database system was shut down at 2022-04-24 23:13:26 UTC
2022-04-24T23:13:26.000 [info] keeper   | 2022-04-24 23:13:26.317 UTC [689] LOG:  database system is ready to accept connections
2022-04-24T23:13:26.000 [info] keeper   | 2022-04-24T23:13:26.507Z      INFO    cmd/keeper.go:1542      already master
2022-04-24T23:13:26.000 [info] keeper   | 2022-04-24T23:13:26.527Z      INFO    cmd/keeper.go:1675      postgres parameters not changed
2022-04-24T23:13:26.000 [info] keeper   | 2022-04-24T23:13:26.527Z      INFO    cmd/keeper.go:1702      postgres hba entries not changed
2022-04-24T23:13:26.000 [info] checking stolon status
2022-04-24T23:13:26.000 [info] sentinel | 2022-04-24T23:13:26.966Z      WARN    cmd/sentinel.go:287     received db state for unexpected db uid        {"receivedDB": "", "db": "9603ce5a", "keeper": "ad10d6a82"}
2022-04-24T23:13:26.000 [info] sentinel | 2022-04-24T23:13:26.967Z      INFO    cmd/sentinel.go:837     waiting for db{"db": "9603ce5a", "keeper": "ad10d6a82"}
2022-04-24T23:13:27.000 [info] keeper is healthy, db is healthy, role: master
2022-04-24T23:13:27.000 [info] configuring operator
2022-04-24T23:13:27.000 [info] operator user does not exist, creating
2022-04-24T23:13:27.000 [info] operator is not a superuser, fixing
2022-04-24T23:13:27.000 [info] operator password does not match config, changing
2022-04-24T23:13:27.000 [info] operator ready!
2022-04-24T23:13:30.000 [info] exporter | INFO[0022] Established new database connection to "fdaa:0:5d96:a7b:ad1:0:d6a8:2:5433".  source="postgres_exporter.go:970"
2022-04-24T23:13:30.000 [info] exporter | INFO[0022] Semantic Version Changed on "fdaa:0:5d96:a7b:ad1:0:d6a8:2:5433": 0.0.0 -> 14.2.0  source="postgres_exporter.go:1539"
2022-04-24T23:13:31.000 [info] keeper   | 2022-04-24T23:13:31.634Z      INFO    cmd/keeper.go:1504      our db requested role is master
2022-04-24T23:13:31.000 [info] keeper   | 2022-04-24T23:13:31.635Z      INFO    cmd/keeper.go:1542      already master
2022-04-24T23:13:31.000 [info] keeper   | 2022-04-24T23:13:31.653Z      INFO    cmd/keeper.go:1675      postgres parameters not changed
2022-04-24T23:13:31.000 [info] keeper   | 2022-04-24T23:13:31.653Z      INFO    cmd/keeper.go:1702      postgres hba entries not changed
2022-04-24T23:13:32.000 [info] sentinel | 2022-04-24T23:13:32.561Z      INFO    cmd/sentinel.go:823     db initialized{"db": "9603ce5a", "keeper": "ad10d6a82"}
2022-04-24T23:13:33.000 [info] proxy    | [WARNING] 113/231333 (564) : Server bk_db/pg1 is UP, reason: Layer7 check passed, code: 200, check duration: 6ms. 1 active and 0 backup servers online. 0 sessions requeued, 0 total in queue.
2022-04-24T23:13:36.000 [info] keeper   | 2022-04-24T23:13:36.767Z      INFO    cmd/keeper.go:1504      our db requested role is master
2022-04-24T23:13:36.000 [info] keeper   | 2022-04-24T23:13:36.768Z      INFO    cmd/keeper.go:1542      already master
2022-04-24T23:13:36.000 [info] keeper   | 2022-04-24T23:13:36.787Z      INFO    cmd/keeper.go:1675      postgres parameters not changed
2022-04-24T23:13:36.000 [info] keeper   | 2022-04-24T23:13:36.787Z      INFO    cmd/keeper.go:1702      postgres hba entries not changed
Error failed creating the Postgres cluster withered-sunset-7289-db: 1 error occurred:
        * Post "https://api.fly.io/graphql": dial tcp: lookup api.fly.io on 192.168.86.1:53: write udp 192.168.86.163:63396->192.168.86.1:53: write: no route to host
2

Total guess, but since the final error there mentions udp in its error, can’t hurt to try this. It uses a websocket tcp instead of udp. That can fix some network issues (depending where they are):

fly version update (get the latest version, or however you prefer to do that)
fly wireguard websockets enable
fly launch (or the command you want to run)

If that doesn’t help, someone from Fly will be able to assist.

3

Thanks for the tip.

I came back and tried same thing again and it worked this time :man_shrugging:

1 Like