Airbyte Sync Issue: Postgres Data Not Going to Destination Mounts

[LsAlmada]

Hello everyone,

I am currently facing an issue with syncing data from a Postgres database using Airbyte on the Fly.io platform. I have created a volume of 250GB for the database, but it seems that the data is not going to the destination defined in the mounts section. As a result, the disk runs out of space quickly.

I have tried using the PGDATA variable for the same directory, but it didn’t work. Does anyone have any ideas on how to solve this issue? Has anyone experienced a similar problem with syncing data using Airbyte on Fly.io?

Thank you in advance for your help.

[ben-io]

Are you using Fly Postgres · Fly Docs, or your own Postgres setup? If it’s Fly Postgres, we configure the volumes for you, and you shouldn’t be setting PGDATA. If it’s your own setup, are you able to echo $PGDATA via SSH?

In any case, if you share the Postgres logs and your fly.toml, it’ll be easier for us to help.

[LsAlmada]

Hello Ben.io

I’m currently using my own postgres setup, my PGDATA variable matches the destination in the mounts section in my fly.toml file. Here’s the code:

# fly.toml app configuration file generated for lk-postgres-alexandria on 2023-04-28T10:50:03-03:00
#
# See https://fly.io/docs/reference/configuration/ for information about how to use this file.
#

app = "lk-postgres-alexandria"
kill_signal = "SIGINT"
kill_timeout = 5
primary_region = "gru"
processes = []

[build]
  image = "postgres:14.5"

[env]
  POSTGRES_DB = "alexandria"
  POSTGRES_USER = "postgres"
  PGDATA = "/postgres-data"

[experimental]
  auto_rollback = true

[[services]]
  internal_port = 5432

  [[services.ports]]
  handlers = ["pg_tls"]
  port = 5432

[mounts]
  source="alexandria"
  destination="/postgres-data"

running this command: fly ssh console -s -C df is possible to see where’s the data is going to /dev/vdb instead of /dev/vdc

[ben-io]

What files are present in /postgres-data?

[ben-io]

I was able to get it running by changing PGDATA = /postgres-data/data, because Postgres doesn’t like to run at the root of a volume.

I also had to create a POSTGRES_PASSWORD secret to get it to initialize.