Issues creating database cluster in Singapore

Gabriel · April 18, 2021, 5:25pm

I’m trying to create a database cluster following the docs

Ran fly postgres create

Selected sin as the location.

After taking 11 minutes, the process stopped running. I checked it with fly postgres list and noticed it’s “pending”. Trying fly restart divine-wind-7241 causes it to go into “suspended” mode.

Here’s the logs if it helps:

$ flyctl --app divine-wind-7241 logs
2021-04-18T16:51:13.775Z cc8facaf sin [info] proxy             | Restarting...
2021-04-18T16:51:13.775Z cc8facaf sin [info] proxy             | Exited
2021-04-18T16:51:13.787Z cc8facaf sin [info] keeper            | Exited
2021-04-18T16:51:13.788Z cc8facaf sin [info] postgres_exporter | Interrupting...
2021-04-18T16:51:13.794Z cc8facaf sin [info] sentinel          | 2021-04-18T16:51:13.792Z       INFO    cmd/sentinel.go:2000       sentinel uid    {"uid": "33d805a1"}
2021-04-18T16:51:13.798Z cc8facaf sin [info] sentinel          | 2021-04-18T16:51:13.795Z       FATAL   cmd/sentinel.go:2021       cannot create sentinel: cannot create store: cannot create kv store: Put "http://127.0.0.1:8500/v1/catalog/register?wait=5000ms": dial tcp 127.0.0.1:8500: connect: connection refused
2021-04-18T16:51:13.801Z cc8facaf sin [info] proxy             | 2021-04-18T16:51:13.798Z       FATAL   cmd/proxy.go:433   cannot create cluster checker: cannot create store: cannot create kv store: Put "http://127.0.0.1:8500/v1/catalog/register?wait=5000ms": dial tcp 127.0.0.1:8500: connect: connection refused
2021-04-18T16:51:13.873Z cc8facaf sin [info] proxy             | Restarted with pid 589...
2021-04-18T16:51:13.873Z cc8facaf sin [info] postgres_exporter | Exited
2021-04-18T16:51:13.874Z cc8facaf sin [info] proxy             | Exited
2021-04-18T16:51:13.875Z cc8facaf sin [info] sentinel          | Restarted with pid 587...
2021-04-18T16:51:13.875Z cc8facaf sin [info] sentinel          | Exited
2021-04-18T16:51:15.456Z cc8facaf sin [info] Main child exited normally with code: 1
2021-04-18T16:51:15.457Z cc8facaf sin [info] Starting clean up.
2021-04-18T16:51:15.457Z cc8facaf sin [info] Reaped child process with pid: 554, exit code: 0
2021-04-18T16:51:15.471Z cc8facaf sin [info] Umounting /dev/vdc from /data
2021-04-18T16:51:19.147Z cc8facaf sin [info] Starting instance
2021-04-18T16:51:19.174Z cc8facaf sin [info] Configuring virtual machine
2021-04-18T16:51:19.175Z cc8facaf sin [info] Pulling container image
2021-04-18T16:51:21.517Z cc8facaf sin [info] Unpacking image
2021-04-18T16:51:21.523Z cc8facaf sin [info] Preparing kernel init
2021-04-18T16:51:21.610Z cc8facaf sin [info] Setting up volume 'pg_data'
2021-04-18T16:51:21.837Z cc8facaf sin [info] Configuring firecracker
2021-04-18T16:51:22.074Z cc8facaf sin [info] Starting virtual machine
2021-04-18T16:51:22.196Z cc8facaf sin [info] Starting init (commit: 0512da4)...
2021-04-18T16:51:22.210Z cc8facaf sin [info] Mounting /dev/vdc at /data
2021-04-18T16:51:22.213Z cc8facaf sin [info] Running: `docker-entrypoint.sh /fly/start.sh` as root
2021-04-18T16:51:22.229Z cc8facaf sin [info] 2021/04/18 16:51:22 listening on [fdaa:0:21ca:a7b:f0f:0:18c3:2]:22 (DNS: [fdaa::3]:53)
2021-04-18T16:51:22.306Z cc8facaf sin [info] system            | Tmux socket name: overmind-fly-ymDQ7eYFxSozhcBQG6cQJh
2021-04-18T16:51:22.307Z cc8facaf sin [info] system            | Tmux session ID: fly
2021-04-18T16:51:22.308Z cc8facaf sin [info] system            | Listening at ./.overmind.sock
2021-04-18T16:51:22.407Z cc8facaf sin [info] keeper            | Started with pid 555...
2021-04-18T16:51:22.409Z cc8facaf sin [info] sentinel          | Started with pid 558...
2021-04-18T16:51:22.410Z cc8facaf sin [info] proxy             | Started with pid 560...
2021-04-18T16:51:22.411Z cc8facaf sin [info] postgres_exporter | Started with pid 563...
2021-04-18T16:51:22.415Z cc8facaf sin [info] postgres_exporter | INFO[0000] Starting Server: :9187                        source="postgres_exporter.go:1837"
2021-04-18T16:51:22.513Z cc8facaf sin [info] sentinel          | 2021-04-18T16:51:22.509Z       INFO    cmd/sentinel.go:2000       sentinel uid    {"uid": "683f7dd3"}
2021-04-18T16:51:22.523Z cc8facaf sin [info] proxy             | 2021-04-18T16:51:22.520Z       FATAL   cmd/proxy.go:433   cannot create cluster checker: cannot create store: cannot create kv store: Put "http://127.0.0.1:8500/v1/catalog/register?wait=5000ms": dial tcp 127.0.0.1:8500: connect: connection refused
2021-04-18T16:51:22.528Z cc8facaf sin [info] keeper            | 2021-04-18T16:51:22.524Z       FATAL   cmd/keeper.go:2081 cannot create keeper: cannot create store: cannot create kv store: Put "http://127.0.0.1:8500/v1/catalog/register?wait=5000ms": dial tcp 127.0.0.1:8500: connect: connection refused
2021-04-18T16:51:22.531Z cc8facaf sin [info] sentinel          | 2021-04-18T16:51:22.528Z       FATAL   cmd/sentinel.go:2021       cannot create sentinel: cannot create store: cannot create kv store: Put "http://127.0.0.1:8500/v1/catalog/register?wait=5000ms": dial tcp 127.0.0.1:8500: connect: connection refused
2021-04-18T16:51:22.608Z cc8facaf sin [info] sentinel          | Exited
2021-04-18T16:51:22.609Z cc8facaf sin [info] proxy             | Exited
2021-04-18T16:51:22.609Z cc8facaf sin [info] sentinel          | Restarting...
2021-04-18T16:51:22.610Z cc8facaf sin [info] keeper            | Exited
2021-04-18T16:51:22.610Z cc8facaf sin [info] proxy             | Restarting...
2021-04-18T16:51:22.622Z cc8facaf sin [info] sentinel          | 2021-04-18T16:51:22.621Z       INFO    cmd/sentinel.go:2000       sentinel uid    {"uid": "13ae81fd"}
2021-04-18T16:51:22.628Z cc8facaf sin [info] sentinel          | 2021-04-18T16:51:22.625Z       FATAL   cmd/sentinel.go:2021       cannot create sentinel: cannot create store: cannot create kv store: Put "http://127.0.0.1:8500/v1/catalog/register?wait=5000ms": dial tcp 127.0.0.1:8500: connect: connection refused
2021-04-18T16:51:22.635Z cc8facaf sin [info] proxy             | 2021-04-18T16:51:22.632Z       FATAL   cmd/proxy.go:433   cannot create cluster checker: cannot create store: cannot create kv store: Put "http://127.0.0.1:8500/v1/catalog/register?wait=5000ms": dial tcp 127.0.0.1:8500: connect: connection refused
2021-04-18T16:51:22.708Z cc8facaf sin [info] postgres_exporter | Interrupting...
2021-04-18T16:51:22.709Z cc8facaf sin [info] proxy             | Exited
2021-04-18T16:51:22.709Z cc8facaf sin [info] proxy             | Restarted with pid 588...
2021-04-18T16:51:22.710Z cc8facaf sin [info] sentinel          | Restarted with pid 586...
2021-04-18T16:51:22.711Z cc8facaf sin [info] postgres_exporter | Exited
2021-04-18T16:51:22.711Z cc8facaf sin [info] sentinel          | Exited
2021-04-18T16:51:24.306Z cc8facaf sin [info] Reaped child process with pid: 554, exit code: 0
2021-04-18T16:51:24.306Z cc8facaf sin [info] Main child exited normally with code: 1
2021-04-18T16:51:24.307Z cc8facaf sin [info] Starting clean up.
2021-04-18T16:51:24.321Z cc8facaf sin [info] Umounting /dev/vdc from /data
2021-04-18T16:51:27.710Z cc8facaf sin [info] Starting instance
2021-04-18T16:51:27.734Z cc8facaf sin [info] Configuring virtual machine
2021-04-18T16:51:27.735Z cc8facaf sin [info] Pulling container image
2021-04-18T16:51:30.084Z cc8facaf sin [info] Unpacking image
2021-04-18T16:51:30.092Z cc8facaf sin [info] Preparing kernel init
2021-04-18T16:51:30.199Z cc8facaf sin [info] Setting up volume 'pg_data'
2021-04-18T16:51:30.467Z cc8facaf sin [info] Configuring firecracker
2021-04-18T16:51:30.706Z cc8facaf sin [info] Starting virtual machine
2021-04-18T16:51:30.810Z cc8facaf sin [info] Starting init (commit: 0512da4)...
2021-04-18T16:51:30.824Z cc8facaf sin [info] Mounting /dev/vdc at /data
2021-04-18T16:51:30.827Z cc8facaf sin [info] Running: `docker-entrypoint.sh /fly/start.sh` as root
2021-04-18T16:51:30.841Z cc8facaf sin [info] 2021/04/18 16:51:30 listening on [fdaa:0:21ca:a7b:f0f:0:18c3:2]:22 (DNS: [fdaa::3]:53)
2021-04-18T16:51:30.918Z cc8facaf sin [info] system            | Tmux socket name: overmind-fly-WYtqL-xcA1K6ptTFQ3MMrV
2021-04-18T16:51:30.919Z cc8facaf sin [info] system            | Tmux session ID: fly
2021-04-18T16:51:30.920Z cc8facaf sin [info] system            | Listening at ./.overmind.sock
2021-04-18T16:51:31.019Z cc8facaf sin [info] keeper            | Started with pid 555...
2021-04-18T16:51:31.020Z cc8facaf sin [info] sentinel          | Started with pid 558...
2021-04-18T16:51:31.022Z cc8facaf sin [info] proxy             | Started with pid 560...
2021-04-18T16:51:31.023Z cc8facaf sin [info] postgres_exporter | Started with pid 563...
2021-04-18T16:51:31.024Z cc8facaf sin [info] postgres_exporter | INFO[0000] Starting Server: :9187                        source="postgres_exporter.go:1837"
2021-04-18T16:51:31.121Z cc8facaf sin [info] sentinel          | 2021-04-18T16:51:31.117Z       INFO    cmd/sentinel.go:2000       sentinel uid    {"uid": "f768a714"}
2021-04-18T16:51:31.129Z cc8facaf sin [info] proxy             | 2021-04-18T16:51:31.125Z       FATAL   cmd/proxy.go:433   cannot create cluster checker: cannot create store: cannot create kv store: Put "http://127.0.0.1:8500/v1/catalog/register?wait=5000ms": dial tcp 127.0.0.1:8500: connect: connection refused
2021-04-18T16:51:31.137Z cc8facaf sin [info] keeper            | 2021-04-18T16:51:31.134Z       FATAL   cmd/keeper.go:2081 cannot create keeper: cannot create store: cannot create kv store: Put "http://127.0.0.1:8500/v1/catalog/register?wait=5000ms": dial tcp 127.0.0.1:8500: connect: connection refused
2021-04-18T16:51:31.140Z cc8facaf sin [info] sentinel          | 2021-04-18T16:51:31.137Z       FATAL   cmd/sentinel.go:2021       cannot create sentinel: cannot create store: cannot create kv store: Put "http://127.0.0.1:8500/v1/catalog/register?wait=5000ms": dial tcp 127.0.0.1:8500: connect: connection refused
2021-04-18T16:51:31.220Z cc8facaf sin [info] proxy             | Exited
2021-04-18T16:51:31.221Z cc8facaf sin [info] keeper            | Exited
2021-04-18T16:51:31.221Z cc8facaf sin [info] proxy             | Restarting...
2021-04-18T16:51:31.222Z cc8facaf sin [info] sentinel          | Restarting...
2021-04-18T16:51:31.222Z cc8facaf sin [info] sentinel          | Exited
2021-04-18T16:51:31.240Z cc8facaf sin [info] proxy             | 2021-04-18T16:51:31.237Z       FATAL   cmd/proxy.go:433   cannot create cluster checker: cannot create store: cannot create kv store: Put "http://127.0.0.1:8500/v1/catalog/register?wait=5000ms": dial tcp 127.0.0.1:8500: connect: connection refused
2021-04-18T16:51:31.242Z cc8facaf sin [info] sentinel          | 2021-04-18T16:51:31.240Z       INFO    cmd/sentinel.go:2000       sentinel uid    {"uid": "c7a564f3"}
2021-04-18T16:51:31.246Z cc8facaf sin [info] sentinel          | 2021-04-18T16:51:31.243Z       FATAL   cmd/sentinel.go:2021       cannot create sentinel: cannot create store: cannot create kv store: Put "http://127.0.0.1:8500/v1/catalog/register?wait=5000ms": dial tcp 127.0.0.1:8500: connect: connection refused
2021-04-18T16:51:31.320Z cc8facaf sin [info] postgres_exporter | Interrupting...
2021-04-18T16:51:31.321Z cc8facaf sin [info] sentinel          | Exited
2021-04-18T16:51:31.321Z cc8facaf sin [info] sentinel          | Restarted with pid 590...
2021-04-18T16:51:31.322Z cc8facaf sin [info] proxy             | Restarted with pid 588...
2021-04-18T16:51:31.323Z cc8facaf sin [info] proxy             | Exited
2021-04-18T16:51:31.422Z cc8facaf sin [info] postgres_exporter | Exited
2021-04-18T16:51:32.927Z cc8facaf sin [info] Reaped child process with pid: 554, exit code: 0
2021-04-18T16:51:32.927Z cc8facaf sin [info] Main child exited normally with code: 1
2021-04-18T16:51:32.928Z cc8facaf sin [info] Starting clean up.
2021-04-18T16:51:32.941Z cc8facaf sin [info] Umounting /dev/vdc from /data
Error 404 Not Found

michael · April 18, 2021, 5:31pm

We just deployed a fix for this. If you delete that app and create a new one you should be good to go.

Gabriel · April 18, 2021, 5:33pm

Trying that now! Thanks Michael!!

Any idea what went wrong? Wondering if there’s any way I could debug issues with setting up PG clusters in future.

michael · April 18, 2021, 5:35pm

This was a bug on our end that wasn’t configuring consul correctly, not much you could do.

Gabriel · April 18, 2021, 5:38pm

Oh no. Gotcha, thanks for resolving this so quickly! You’re awesome!

Just got it deployed in 2m55s.

Can’t wait to get this connected with my other fly apps!

2 quick questions though:

can I use the same cluster for multiple, non-related fly apps?
when scaling the PG cluster, does it replicate data over? (any downtime when doing so?)

michael · April 18, 2021, 5:49pm

Glad it worked!

Yes you can share one or more databases in the cluster across multiple fly apps

Yes, everything replicates to secondaries. Unless the primary fails over (restart from scaling memory/cpu, failure, etc) you won’t have downtime.

Topic		Replies	Views
After `fly pg create` stuck in "Pending"	6	274	March 11, 2023
Postgres having issues: `cannot create kv store` postgres , consul	5	257	December 14, 2023
Postgresql "cannot create sentinel: cannot create store: cannot create kv store" postgres	9	626	October 24, 2022
Timeout on CreatePostgresCluster.app Questions / Help	16	966	April 24, 2022
"Cannot create kv store" when creating a new Postgres Database Questions / Help	2	405	October 6, 2021

Issues creating database cluster in Singapore

Related topics