Timescale (data directory mount issue lost+found)

1

Naively trying to get timescale to deploy, I’ve tried with no volume and with a volume mounted to /var/lib/postgresql/data, both have errors.

The base config is:

app = "timescale"

kill_signal = "SIGINT"
kill_timeout = 5

[experimental]
private_network = true

[[services]]
  internal_port = 5432
  protocol = "tcp"

  [services.concurrency]
    hard_limit = 25
    soft_limit = 20

  [[services.tcp_checks]]
    grace_period = "1s"
    interval = "10s"
    restart_limit = 2
    timeout = "6s"

I get:

2021-03-08T05:17:26.540Z b056efd1 lax [info] Starting instance
2021-03-08T05:17:26.574Z b056efd1 lax [info] Configuring virtual machine
2021-03-08T05:17:26.576Z b056efd1 lax [info] Pulling container image
2021-03-08T05:17:30.588Z b056efd1 lax [info] Unpacking image
2021-03-08T05:17:30.602Z b056efd1 lax [info] Preparing kernel init
2021-03-08T05:17:30.764Z b056efd1 lax [info] Setting up volume 'timescale_data'
2021-03-08T05:17:30.769Z b056efd1 lax [info] Opening encrypted volume
2021-03-08T05:17:31.105Z b056efd1 lax [info] Configuring firecracker
2021-03-08T05:17:31.183Z b056efd1 lax [info] Starting virtual machine
2021-03-08T05:17:31.370Z b056efd1 lax [info] Starting init (commit: c1dec69)...
2021-03-08T05:17:31.391Z b056efd1 lax [info] Mounting /dev/vdc at /var/lib/postgresql/data
2021-03-08T05:17:31.393Z b056efd1 lax [info] directory /var/lib/postgresql/data already exists
2021-03-08T05:17:31.401Z b056efd1 lax [info] Running: `docker-entrypoint.sh postgres` as root
2021-03-08T05:17:31.411Z b056efd1 lax [info] 2021/03/08 05:17:31 listening on [fdaa:0:17e2:a7b:85:0:1124:2]:22 (DNS: [fdaa::3]:53)
2021-03-08T05:17:31.495Z b056efd1 lax [info] The files belonging to this database system will be owned by user "postgres".
2021-03-08T05:17:31.496Z b056efd1 lax [info] This user must also own the server process.
2021-03-08T05:17:31.499Z b056efd1 lax [info] The database cluster will be initialized with locale "en_US.utf8".
2021-03-08T05:17:31.501Z b056efd1 lax [info] The default database encoding has accordingly been set to "UTF8".
2021-03-08T05:17:31.502Z b056efd1 lax [info] The default text search configuration will be set to "english".
2021-03-08T05:17:31.503Z b056efd1 lax [info] Data page checksums are disabled.
2021-03-08T05:17:31.504Z b056efd1 lax [info] initdb: directory "/var/lib/postgresql/data" exists but is not empty
2021-03-08T05:17:31.505Z b056efd1 lax [info] If you want to create a new database system, either remove or empty
2021-03-08T05:17:31.506Z b056efd1 lax [info] the directory "/var/lib/postgresql/data" or run initdb
2021-03-08T05:17:31.507Z b056efd1 lax [info] with an argument other than "/var/lib/postgresql/data".
2021-03-08T05:17:32.410Z b056efd1 lax [info] Main child exited normally with code: 1
2021-03-08T05:17:32.424Z b056efd1 lax [info] Umounting /dev/vdc from /var/lib/postgresql/data
2021-03-08T05:17:52.705Z bbe1bb9c lax [info] Starting instance
2021-03-08T05:17:52.750Z bbe1bb9c lax [info] Configuring virtual machine
2021-03-08T05:17:52.751Z bbe1bb9c lax [info] Pulling container image
2021-03-08T05:17:58.145Z bbe1bb9c lax [info] Unpacking image
2021-03-08T05:17:58.160Z bbe1bb9c lax [info] Preparing kernel init
2021-03-08T05:17:58.315Z bbe1bb9c lax [info] Setting up volume 'timescale_data'
2021-03-08T05:17:58.320Z bbe1bb9c lax [info] Opening encrypted volume
2021-03-08T05:17:58.625Z bbe1bb9c lax [info] Configuring firecracker
2021-03-08T05:17:58.703Z bbe1bb9c lax [info] Starting virtual machine
2021-03-08T05:17:58.902Z bbe1bb9c lax [info] Starting init (commit: c1dec69)...
2021-03-08T05:17:58.928Z bbe1bb9c lax [info] Mounting /dev/vdc at /data
2021-03-08T05:17:58.939Z bbe1bb9c lax [info] Running: `docker-entrypoint.sh postgres` as root
2021-03-08T05:17:58.948Z bbe1bb9c lax [info] 2021/03/08 05:17:58 listening on [fdaa:0:17e2:a7b:85:0:1124:2]:22 (DNS: [fdaa::3]:53)
2021-03-08T05:17:59.038Z bbe1bb9c lax [info] The files belonging to this database system will be owned by user "postgres".
2021-03-08T05:17:59.039Z bbe1bb9c lax [info] This user must also own the server process.
2021-03-08T05:17:59.042Z bbe1bb9c lax [info] The database cluster will be initialized with locale "en_US.utf8".
2021-03-08T05:17:59.043Z bbe1bb9c lax [info] The default database encoding has accordingly been set to "UTF8".
2021-03-08T05:17:59.044Z bbe1bb9c lax [info] Data page checksums are disabled.
2021-03-08T05:17:59.044Z bbe1bb9c lax [info] The default text search configuration will be set to "english".
2021-03-08T05:17:59.046Z bbe1bb9c lax [info] fixing permissions on existing directory /var/lib/postgresql/data ... ok
2021-03-08T05:17:59.047Z bbe1bb9c lax [info] creating subdirectories ... ok
2021-03-08T05:17:59.080Z bbe1bb9c lax [info] selecting default max_connections ... 100
2021-03-08T05:17:59.106Z bbe1bb9c lax [info] selecting default shared_buffers ... 128MB
2021-03-08T05:17:59.250Z bbe1bb9c lax [info] selecting default timezone ... UTC
2021-03-08T05:17:59.251Z bbe1bb9c lax [info] selecting dynamic shared memory implementation ... posix
2021-03-08T05:17:59.256Z bbe1bb9c lax [info] creating configuration files ... ok
2021-03-08T05:17:59.418Z bbe1bb9c lax [info] running bootstrap script ... ok
2021-03-08T05:17:59.588Z bbe1bb9c lax [info] performing post-bootstrap initialization ... sh: locale: not found
2021-03-08T05:17:59.589Z bbe1bb9c lax [info] 2021-03-08 05:17:59.575 UTC [545] WARNING:  no usable system locales were found
2021-03-08T05:18:00.043Z bbe1bb9c lax [info] ok
2021-03-08T05:18:00.138Z bbe1bb9c lax [info] syncing data to disk ... ok
2021-03-08T05:18:00.139Z bbe1bb9c lax [info] WARNING: enabling "trust" authentication for local connections
2021-03-08T05:18:00.140Z bbe1bb9c lax [info] --auth-local and --auth-host, the next time you run initdb.
2021-03-08T05:18:00.140Z bbe1bb9c lax [info] You can change this by editing pg_hba.conf or using the option -A, or
2021-03-08T05:18:00.142Z bbe1bb9c lax [info] Success. You can now start the database server using:
2021-03-08T05:18:00.143Z bbe1bb9c lax [info]     pg_ctl -D /var/lib/postgresql/data -l logfile start
2021-03-08T05:18:00.160Z bbe1bb9c lax [info] waiting for server to start....2021-03-08 05:18:00.145 UTC [550] LOG:  listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
2021-03-08T05:18:00.173Z bbe1bb9c lax [info] 2021-03-08 05:18:00.158 UTC [551] LOG:  database system was shut down at 2021-03-08 05:18:00 UTC
2021-03-08T05:18:00.176Z bbe1bb9c lax [info] 2021-03-08 05:18:00.161 UTC [550] LOG:  database system is ready to accept connections
2021-03-08T05:18:00.179Z bbe1bb9c lax [info] 2021-03-08 05:18:00.165 UTC [557] LOG:  TimescaleDB background worker launcher connected to shared catalogs
2021-03-08T05:18:00.252Z bbe1bb9c lax [info] server started
2021-03-08T05:18:00.252Z bbe1bb9c lax [info]  done
2021-03-08T05:18:00.516Z bbe1bb9c lax [info] CREATE DATABASE
2021-03-08T05:18:00.518Z bbe1bb9c lax [info] /usr/local/bin/docker-entrypoint.sh: sourcing /docker-entrypoint-initdb.d/000_install_timescaledb.sh
2021-03-08T05:18:00.663Z bbe1bb9c lax [info] 2021-03-08 05:18:00.649 UTC [565] WARNING:
2021-03-08T05:18:00.663Z bbe1bb9c lax [info] 	WELCOME TO
2021-03-08T05:18:00.664Z bbe1bb9c lax [info] 	|_   _(_)                             | |    |  _  \ ___ \
2021-03-08T05:18:00.664Z bbe1bb9c lax [info] 	 _____ _                               _     ____________
2021-03-08T05:18:00.665Z bbe1bb9c lax [info] 	  | |  _ _ __ ___   ___  ___  ___ __ _| | ___| | | | |_/ /
2021-03-08T05:18:00.666Z bbe1bb9c lax [info] 	  | | | |  _ ` _ \ / _ \/ __|/ __/ _` | |/ _ \ | | | ___ \
2021-03-08T05:18:00.667Z bbe1bb9c lax [info] 	  | | | | | | | | |  __/\__ \ (_| (_| | |  __/ |/ /| |_/ /
2021-03-08T05:18:00.668Z bbe1bb9c lax [info] 	  |_| |_|_| |_| |_|\___||___/\___\__,_|_|\___|___/ \____/
2021-03-08T05:18:00.668Z bbe1bb9c lax [info] 	               Running version 1.7.2
2021-03-08T05:18:00.669Z bbe1bb9c lax [info] 	For more information on TimescaleDB, please visit the following links:
2021-03-08T05:18:00.670Z bbe1bb9c lax [info] 	 1. Getting started: https://docs.timescale.com/getting-started
2021-03-08T05:18:00.671Z bbe1bb9c lax [info] 	 2. API reference documentation: https://docs.timescale.com/api
2021-03-08T05:18:00.672Z bbe1bb9c lax [info] 	 3. How TimescaleDB is designed: https://docs.timescale.com/introduction/architecture
2021-03-08T05:18:00.673Z bbe1bb9c lax [info] 	Note: TimescaleDB collects anonymous reports to better understand and assist our users.
2021-03-08T05:18:00.674Z bbe1bb9c lax [info] 	For more information and how to disable, please see our docs https://docs.timescaledb.com/using-timescaledb/telemetry.
2021-03-08T05:18:00.676Z bbe1bb9c lax [info] 2021-03-08 05:18:00.649 UTC [565] CONTEXT:  PL/pgSQL function inline_code_block line 23 at RAISE
2021-03-08T05:18:00.677Z bbe1bb9c lax [info] psql:/tmp/tmp.dkmKBg:1: WARNING:
2021-03-08T05:18:00.678Z bbe1bb9c lax [info]  _____ _                               _     ____________
2021-03-08T05:18:00.678Z bbe1bb9c lax [info] WELCOME TO
2021-03-08T05:18:00.679Z bbe1bb9c lax [info] |_   _(_)                             | |    |  _  \ ___ \
2021-03-08T05:18:00.680Z bbe1bb9c lax [info]   | |  _ _ __ ___   ___  ___  ___ __ _| | ___| | | | |_/ /
2021-03-08T05:18:00.681Z bbe1bb9c lax [info]   | | | | | | | | |  __/\__ \ (_| (_| | |  __/ |/ /| |_/ /
2021-03-08T05:18:00.681Z bbe1bb9c lax [info]   | | | |  _ ` _ \ / _ \/ __|/ __/ _` | |/ _ \ | | | ___ \
2021-03-08T05:18:00.682Z bbe1bb9c lax [info]   |_| |_|_| |_| |_|\___||___/\___\__,_|_|\___|___/ \____/
2021-03-08T05:18:00.683Z bbe1bb9c lax [info] For more information on TimescaleDB, please visit the following links:
2021-03-08T05:18:00.683Z bbe1bb9c lax [info]                Running version 1.7.2
2021-03-08T05:18:00.684Z bbe1bb9c lax [info]  1. Getting started: https://docs.timescale.com/getting-started
2021-03-08T05:18:00.685Z bbe1bb9c lax [info]  2. API reference documentation: https://docs.timescale.com/api
2021-03-08T05:18:00.686Z bbe1bb9c lax [info]  3. How TimescaleDB is designed: https://docs.timescale.com/introduction/architecture
2021-03-08T05:18:00.687Z bbe1bb9c lax [info] Note: TimescaleDB collects anonymous reports to better understand and assist our users.
2021-03-08T05:18:00.689Z bbe1bb9c lax [info] For more information and how to disable, please see our docs https://docs.timescaledb.com/using-timescaledb/telemetry.
2021-03-08T05:18:00.695Z bbe1bb9c lax [info] CREATE EXTENSION
2021-03-08T05:18:00.869Z bbe1bb9c lax [info]   | | | |  _ ` _ \ / _ \/ __|/ __/ _` | |/ _ \ | | | ___ \
2021-03-08T05:18:00.870Z bbe1bb9c lax [info]   | | | | | | | | |  __/\__ \ (_| (_| | |  __/ |/ /| |_/ /
2021-03-08T05:18:00.871Z bbe1bb9c lax [info]                Running version 1.7.2
2021-03-08T05:18:00.871Z bbe1bb9c lax [info]   |_| |_|_| |_| |_|\___||___/\___\__,_|_|\___|___/ \____/
2021-03-08T05:18:00.872Z bbe1bb9c lax [info] For more information on TimescaleDB, please visit the following links:
2021-03-08T05:18:00.873Z bbe1bb9c lax [info]  1. Getting started: https://docs.timescale.com/getting-started
2021-03-08T05:18:00.874Z bbe1bb9c lax [info]  2. API reference documentation: https://docs.timescale.com/api
2021-03-08T05:18:00.875Z bbe1bb9c lax [info]  3. How TimescaleDB is designed: https://docs.timescale.com/introduction/architecture
2021-03-08T05:18:00.876Z bbe1bb9c lax [info] Note: TimescaleDB collects anonymous reports to better understand and assist our users.
2021-03-08T05:18:00.877Z bbe1bb9c lax [info] For more information and how to disable, please see our docs https://docs.timescaledb.com/using-timescaledb/telemetry.
2021-03-08T05:18:00.884Z bbe1bb9c lax [info] CREATE EXTENSION
2021-03-08T05:18:00.884Z bbe1bb9c lax [info] 2021-03-08 05:18:00.865 UTC [572] ERROR:  TimescaleDB background worker connected to template database, exiting
2021-03-08T05:18:00.886Z bbe1bb9c lax [info] 2021-03-08 05:18:00.866 UTC [571] WARNING:  telemetry could not connect to "telemetry.timescale.com"
2021-03-08T05:18:00.887Z bbe1bb9c lax [info] 2021-03-08 05:18:00.866 UTC [571] DETAIL:  SSL error want read
2021-03-08T05:18:00.892Z bbe1bb9c lax [info] 2021-03-08 05:18:00.869 UTC [550] LOG:  background worker "TimescaleDB Background Worker Scheduler" (PID 572) exited with exit code 1
2021-03-08T05:18:01.028Z bbe1bb9c lax [info] 2021-03-08 05:18:01.014 UTC [574] WARNING:
2021-03-08T05:18:01.029Z bbe1bb9c lax [info] 	WELCOME TO
2021-03-08T05:18:01.030Z bbe1bb9c lax [info] 	|_   _(_)                             | |    |  _  \ ___ \
2021-03-08T05:18:01.030Z bbe1bb9c lax [info] 	 _____ _                               _     ____________
2021-03-08T05:18:01.031Z bbe1bb9c lax [info] 	  | |  _ _ __ ___   ___  ___  ___ __ _| | ___| | | | |_/ /
2021-03-08T05:18:01.032Z bbe1bb9c lax [info] 	  | | | |  _ ` _ \ / _ \/ __|/ __/ _` | |/ _ \ | | | ___ \
2021-03-08T05:18:01.033Z bbe1bb9c lax [info] 	  |_| |_|_| |_| |_|\___||___/\___\__,_|_|\___|___/ \____/
2021-03-08T05:18:01.033Z bbe1bb9c lax [info] 	  | | | | | | | | |  __/\__ \ (_| (_| | |  __/ |/ /| |_/ /
2021-03-08T05:18:01.034Z bbe1bb9c lax [info] 	               Running version 1.7.2
2021-03-08T05:18:01.035Z bbe1bb9c lax [info] 	For more information on TimescaleDB, please visit the following links:
2021-03-08T05:18:01.036Z bbe1bb9c lax [info] 	 2. API reference documentation: https://docs.timescale.com/api
2021-03-08T05:18:01.036Z bbe1bb9c lax [info] 	 1. Getting started: https://docs.timescale.com/getting-started
2021-03-08T05:18:01.037Z bbe1bb9c lax [info] 	 3. How TimescaleDB is designed: https://docs.timescale.com/introduction/architecture
2021-03-08T05:18:01.039Z bbe1bb9c lax [info] 	Note: TimescaleDB collects anonymous reports to better understand and assist our users.
2021-03-08T05:18:01.040Z bbe1bb9c lax [info] 	For more information and how to disable, please see our docs https://docs.timescaledb.com/using-timescaledb/telemetry.
2021-03-08T05:18:01.041Z bbe1bb9c lax [info] 2021-03-08 05:18:01.014 UTC [574] CONTEXT:  PL/pgSQL function inline_code_block line 23 at RAISE
2021-03-08T05:18:01.043Z bbe1bb9c lax [info] WELCOME TO
2021-03-08T05:18:01.043Z bbe1bb9c lax [info] psql:/tmp/tmp.dkmKBg:1: WARNING:
2021-03-08T05:18:01.044Z bbe1bb9c lax [info]  _____ _                               _     ____________
2021-03-08T05:18:01.045Z bbe1bb9c lax [info]   | |  _ _ __ ___   ___  ___  ___ __ _| | ___| | | | |_/ /
2021-03-08T05:18:01.045Z bbe1bb9c lax [info] |_   _(_)                             | |    |  _  \ ___ \
2021-03-08T05:18:01.046Z bbe1bb9c lax [info]   | | | |  _ ` _ \ / _ \/ __|/ __/ _` | |/ _ \ | | | ___ \
2021-03-08T05:18:01.047Z bbe1bb9c lax [info]   | | | | | | | | |  __/\__ \ (_| (_| | |  __/ |/ /| |_/ /
2021-03-08T05:18:01.048Z bbe1bb9c lax [info]                Running version 1.7.2
2021-03-08T05:18:01.048Z bbe1bb9c lax [info]   |_| |_|_| |_| |_|\___||___/\___\__,_|_|\___|___/ \____/
2021-03-08T05:18:01.049Z bbe1bb9c lax [info] For more information on TimescaleDB, please visit the following links:
2021-03-08T05:18:01.050Z bbe1bb9c lax [info]  1. Getting started: https://docs.timescale.com/getting-started
2021-03-08T05:18:01.051Z bbe1bb9c lax [info]  2. API reference documentation: https://docs.timescale.com/api
2021-03-08T05:18:01.052Z bbe1bb9c lax [info]  3. How TimescaleDB is designed: https://docs.timescale.com/introduction/architecture
2021-03-08T05:18:01.053Z bbe1bb9c lax [info] Note: TimescaleDB collects anonymous reports to better understand and assist our users.
2021-03-08T05:18:01.054Z bbe1bb9c lax [info] For more information and how to disable, please see our docs https://docs.timescaledb.com/using-timescaledb/telemetry.
2021-03-08T05:18:01.059Z bbe1bb9c lax [info] CREATE EXTENSION
2021-03-08T05:18:01.060Z bbe1bb9c lax [info] /usr/local/bin/docker-entrypoint.sh: running /docker-entrypoint-initdb.d/001_timescaledb_tune.sh
2021-03-08T05:18:01.096Z bbe1bb9c lax [info] Using postgresql.conf at this path:
2021-03-08T05:18:01.097Z bbe1bb9c lax [info] /var/lib/postgresql/data/postgresql.conf
2021-03-08T05:18:01.105Z bbe1bb9c lax [info] /tmp/timescaledb_tune.backup202103080518
2021-03-08T05:18:01.105Z bbe1bb9c lax [info] Writing backup to:
2021-03-08T05:18:01.107Z bbe1bb9c lax [info] Recommendations based on 989.20 MB of available memory and 1 CPUs for PostgreSQL 11
2021-03-08T05:18:01.108Z bbe1bb9c lax [info] effective_cache_size = 759705kB
2021-03-08T05:18:01.108Z bbe1bb9c lax [info] shared_buffers = 253235kB
2021-03-08T05:18:01.109Z bbe1bb9c lax [info] wal_buffers = 7596kB
2021-03-08T05:18:01.109Z bbe1bb9c lax [info] work_mem = 6330kB
2021-03-08T05:18:01.109Z bbe1bb9c lax [info] maintenance_work_mem = 126617kB
2021-03-08T05:18:01.110Z bbe1bb9c lax [info] default_statistics_target = 500
2021-03-08T05:18:01.110Z bbe1bb9c lax [info] min_wal_size = 512MB
2021-03-08T05:18:01.111Z bbe1bb9c lax [info] checkpoint_completion_target = 0.9
2021-03-08T05:18:01.111Z bbe1bb9c lax [info] random_page_cost = 1.1
2021-03-08T05:18:01.112Z bbe1bb9c lax [info] max_locks_per_transaction = 64
2021-03-08T05:18:01.112Z bbe1bb9c lax [info] max_connections = 20
2021-03-08T05:18:01.113Z bbe1bb9c lax [info] autovacuum_naptime = 10
2021-03-08T05:18:01.113Z bbe1bb9c lax [info] autovacuum_max_workers = 10
2021-03-08T05:18:01.114Z bbe1bb9c lax [info] timescaledb.last_tuned = '2021-03-08T05:18:01Z'
2021-03-08T05:18:01.114Z bbe1bb9c lax [info] effective_io_concurrency = 200
2021-03-08T05:18:01.115Z bbe1bb9c lax [info] timescaledb.last_tuned_version = '0.9.0'
2021-03-08T05:18:01.117Z bbe1bb9c lax [info] Saving changes to: /var/lib/postgresql/data/postgresql.conf
2021-03-08T05:18:01.120Z bbe1bb9c lax [info]
2021-03-08T05:18:01.121Z bbe1bb9c lax [info] 2021-03-08 05:18:01.107 UTC [550] LOG:  received fast shutdown request
2021-03-08T05:18:01.126Z bbe1bb9c lax [info] waiting for server to shut down....2021-03-08 05:18:01.109 UTC [550] LOG:  aborting any active transactions
2021-03-08T05:18:01.127Z bbe1bb9c lax [info] 2021-03-08 05:18:01.109 UTC [578] LOG:  terminating TimescaleDB background job "Telemetry Reporter" due to administrator command
2021-03-08T05:18:01.129Z bbe1bb9c lax [info] 2021-03-08 05:18:01.110 UTC [578] WARNING:  telemetry could not connect to "telemetry.timescale.com"
2021-03-08T05:18:01.130Z bbe1bb9c lax [info] 2021-03-08 05:18:01.110 UTC [578] DETAIL:  Interrupted system call
2021-03-08T05:18:01.131Z bbe1bb9c lax [info] 2021-03-08 05:18:01.110 UTC [575] LOG:  terminating TimescaleDB job scheduler due to administrator command
2021-03-08T05:18:01.132Z bbe1bb9c lax [info] 2021-03-08 05:18:01.110 UTC [566] LOG:  terminating TimescaleDB job scheduler due to administrator command
2021-03-08T05:18:01.134Z bbe1bb9c lax [info] 2021-03-08 05:18:01.110 UTC [557] LOG:  terminating TimescaleDB background worker launcher due to administrator command
2021-03-08T05:18:01.139Z bbe1bb9c lax [info] 2021-03-08 05:18:01.120 UTC [550] LOG:  background worker "logical replication launcher" (PID 558) exited with exit code 1
2021-03-08T05:18:01.140Z bbe1bb9c lax [info] 2021-03-08 05:18:01.121 UTC [578] FATAL:  terminating connection due to administrator command
2021-03-08T05:18:01.141Z bbe1bb9c lax [info] 2021-03-08 05:18:01.122 UTC [566] FATAL:  terminating connection due to administrator command
2021-03-08T05:18:01.142Z bbe1bb9c lax [info] 2021-03-08 05:18:01.122 UTC [575] FATAL:  terminating connection due to administrator command
2021-03-08T05:18:01.144Z bbe1bb9c lax [info] 2021-03-08 05:18:01.123 UTC [557] FATAL:  terminating connection due to administrator command
2021-03-08T05:18:01.145Z bbe1bb9c lax [info] 2021-03-08 05:18:01.123 UTC [550] LOG:  background worker "TimescaleDB Background Worker Launcher" (PID 557) exited with exit code 1
2021-03-08T05:18:01.147Z bbe1bb9c lax [info] 2021-03-08 05:18:01.123 UTC [550] LOG:  background worker "TimescaleDB Background Worker Scheduler" (PID 566) exited with exit code 1
2021-03-08T05:18:01.149Z bbe1bb9c lax [info] 2021-03-08 05:18:01.124 UTC [550] LOG:  background worker "TimescaleDB Background Worker Scheduler" (PID 575) exited with exit code 1
2021-03-08T05:18:01.150Z bbe1bb9c lax [info] 2021-03-08 05:18:01.124 UTC [550] LOG:  background worker "Telemetry Reporter" (PID 578) exited with exit code 1
2021-03-08T05:18:01.151Z bbe1bb9c lax [info] 2021-03-08 05:18:01.124 UTC [552] LOG:  shutting down
2021-03-08T05:18:01.187Z bbe1bb9c lax [info] 2021-03-08 05:18:01.173 UTC [550] LOG:  database system is shut down
2021-03-08T05:18:01.227Z bbe1bb9c lax [info] server stopped
2021-03-08T05:18:01.227Z bbe1bb9c lax [info]  done

If I add:

[mount]
source="timescale_data"
destination="/var/lib/postgresql/data"

I get:

2021-03-08T05:20:46.894Z 7870d922 lax [info] Starting instance
2021-03-08T05:20:46.929Z 7870d922 lax [info] Configuring virtual machine
2021-03-08T05:20:46.931Z 7870d922 lax [info] Pulling container image
2021-03-08T05:20:50.841Z 7870d922 lax [info] Unpacking image
2021-03-08T05:20:50.855Z 7870d922 lax [info] Preparing kernel init
2021-03-08T05:20:51.035Z 7870d922 lax [info] Setting up volume 'timescale_data'
2021-03-08T05:20:51.040Z 7870d922 lax [info] Opening encrypted volume
2021-03-08T05:20:51.414Z 7870d922 lax [info] Configuring firecracker
2021-03-08T05:20:51.492Z 7870d922 lax [info] Starting virtual machine
2021-03-08T05:20:51.678Z 7870d922 lax [info] Starting init (commit: c1dec69)...
2021-03-08T05:20:51.698Z 7870d922 lax [info] Mounting /dev/vdc at /var/lib/postgresql/data
2021-03-08T05:20:51.700Z 7870d922 lax [info] directory /var/lib/postgresql/data already exists
2021-03-08T05:20:51.705Z 7870d922 lax [info] Running: `docker-entrypoint.sh postgres` as root
2021-03-08T05:20:51.715Z 7870d922 lax [info] 2021/03/08 05:20:51 listening on [fdaa:0:17e2:a7b:85:0:1124:2]:22 (DNS: [fdaa::3]:53)
2021-03-08T05:20:51.800Z 7870d922 lax [info] The files belonging to this database system will be owned by user "postgres".
2021-03-08T05:20:51.801Z 7870d922 lax [info] This user must also own the server process.
2021-03-08T05:20:51.805Z 7870d922 lax [info] The database cluster will be initialized with locale "en_US.utf8".
2021-03-08T05:20:51.807Z 7870d922 lax [info] The default database encoding has accordingly been set to "UTF8".
2021-03-08T05:20:51.808Z 7870d922 lax [info] The default text search configuration will be set to "english".
2021-03-08T05:20:51.809Z 7870d922 lax [info] Data page checksums are disabled.
2021-03-08T05:20:51.810Z 7870d922 lax [info] If you want to create a new database system, either remove or empty
2021-03-08T05:20:51.810Z 7870d922 lax [info] initdb: directory "/var/lib/postgresql/data" exists but is not empty
2021-03-08T05:20:51.811Z 7870d922 lax [info] the directory "/var/lib/postgresql/data" or run initdb
2021-03-08T05:20:51.812Z 7870d922 lax [info] with an argument other than "/var/lib/postgresql/data".
2021-03-08T05:20:52.713Z 7870d922 lax [info] Main child exited normally with code: 1
2021-03-08T05:20:52.732Z 7870d922 lax [info] Umounting /dev/vdc from /var/lib/postgresql/data

I think this is caused by the lost+found being in the mounted drive already, but I’m a bit confused on if there’s a good way around that. I’d have to run a command that checked if lost+found exists + delete it, is that recommended?

I’ve run this a few times with different configs and previously I had a more explicit error about the lost+found, but perhaps since I re-mounted it in a few locations to test (I tried /var/lib/postgresql just for fun to see what happened, that may have written other files too), so I’ll erase and try the conditional rm.

2

Tried this Dockerfile:

FROM timescale/timescaledb-postgis:1.7.2-pg12

ENV POSTGRES_USER=postgres
ENV POSTGRES_PASSWORD=postgres
ENV POSTGRES_DB=scrape_data

COPY docker-entrypoint-initdb.d/* /docker-entrypoint-initdb.d/

RUN ls -la /docker-entrypoint-initdb.d

the ls shows:

drwxr-xr-x    1 root     root          4096 Mar  8 06:02 .
drwxr-xr-x    1 root     root          4096 Mar  8 06:02 ..
-rwxr-xr-x    1 root     root           114 Mar  8 06:00 0000_remove_fly_data.sh
-rw-r--r--    1 root     root          1208 Oct 29 14:45 000_install_timescaledb.sh
-rwxr-xr-x    1 root     root          3554 Oct 29 14:45 001_timescaledb_tune.sh

And the script:

#!/bin/bash
set -e

ls -la /var/lib/postgresql/data
rm -r /var/lib/postgresql/data/lost+found || true

echo "hi?"

But on running I don’t see the commands run, no “hi?” logged. May be putting that in the wrong place, or something else…

3

Aha ok… So turns out once initialized already postgres won’t run the scripts again (Database initialization · Issue #102 · timescale/timescaledb-kubernetes · GitHub). So I swapped to a simple Dockerfile that just removed lost+found, deployed, then re-deployed with timescale et voila, it works.

The original solution of adding a new 0000_remove_lost_found.sh should work for a clean run.

4

Did you ever get this working with a volume? We’ve been debating using the timescale image for our default postgres cluster.

5

Yea, just had to delete the lost+found using a bare image and then re-deploy and it seems to be working fine.

1 Like
6

For anyone else experiencing this problem, an easier solution might be the following:

Set your postgres data directory to be in a subdirectory of the volume mount:

[env]
POSTGRES_PASSWORD = "topsecret"
POSTGRES_USER = "postgres"
POSTGRES_DB = "postgres"
PGDATA  = "/var/lib/postgresql/data/pgdata"

[mounts]
source = "pg_data"
destination = "/var/lib/postgresql/data"

In your Dockerfile, set the container to run as the postgres user instead of root:

FROM timescale/timescaledb-postgis
USER postgres

These simple changes worked for me, they may work for you.

2 Likes
7

Thanks! very helpful.

@kurt Any news on using timescaledb as the default for fly as you mentioned before? We are considering timescaledb as our defacto db.