Unable to extend plugin from default Postgres setup

I am trying to extend the underlying postgres db to support pgvector for vector based similarity search

FROM flyio/postgres-flex:15.3

# Install build dependencies
RUN apt-get update && \
    apt-get install -y --no-install-recommends \
        build-essential \
        curl \
        postgresql-server-dev-all

# Set the pgvector version
ARG PGVECTOR_VERSION=0.4.4

# Download and extract the pgvector release, build the extension, and install it
RUN curl -L -o pgvector.tar.gz "https://github.com/ankane/pgvector/archive/refs/tags/v${PGVECTOR_VERSION}.tar.gz" && \
    tar -xzf pgvector.tar.gz && \
    cd "pgvector-${PGVECTOR_VERSION}" && \
    make && \
    make install

# Clean up build dependencies and temporary files
RUN apt-get remove -y build-essential curl postgresql-server-dev-all && \
    apt-get autoremove -y && \
    apt-get clean && \
    rm -rf /var/lib/apt/lists/* && \
    rm -rf /pgvector.tar.gz /pgvector-${PGVECTOR_VERSION}

My docker image doesnt work ks0m1c/fly-vector:v0.4.4 but an identical image except for the version of postgres-flex used
coji/fly-pg-pgvector:latest works

Thus I have narrowed it down to something having changed within the last 3 months for postgres-flex repo flyio/postgres-flex

Its not explicit where the failure is occuring.

2023-09-02T09:17:53Z app[6e82d04c2d3387] sin [info]2023/09/02 09:17:53 listening on [fdaa:0:4a0f:a7b:ead:68aa:124c:2]:22 (DNS: [fdaa::3]:53)
2023-09-02T09:17:54Z app[6e82d04c2d3387] sin [info][    2.300427] reboot: Restarting system
2023-09-02T09:18:29Z app[6e82d04c2d3387] sin [info][    0.057849] PCI: Fatal: No config space access function found
2023-09-02T09:18:29Z app[6e82d04c2d3387] sin [info] INFO Starting init (commit: 5293a085)...
2023-09-02T09:18:29Z app[6e82d04c2d3387] sin [info] INFO Mounting /dev/vdb at /data w/ uid: 0, gid: 0 and chmod 0755
2023-09-02T09:18:29Z app[6e82d04c2d3387] sin [info] INFO Resized /data to 1069547520 bytes
2023-09-02T09:18:29Z app[6e82d04c2d3387] sin [info] INFO Preparing to run: `docker-entrypoint.sh start` as root
2023-09-02T09:18:29Z app[6e82d04c2d3387] sin [info] INFO [fly api proxy] listening at /.fly/api
2023-09-02T09:18:29Z app[6e82d04c2d3387] sin [info]2023/09/02 09:18:29 listening on [fdaa:0:4a0f:a7b:ead:68aa:124c:2]:22 (DNS: [fdaa::3]:53)
2023-09-02T09:18:29Z app[6e82d04c2d3387] sin [info]Provisioning primary
2023-09-02T09:18:30Z app[6e82d04c2d3387] sin [info]panic: failed to initialize postgres failed to init postgres: exit status 1
2023-09-02T09:18:30Z app[6e82d04c2d3387] sin [info]goroutine 1 [running]:
2023-09-02T09:18:30Z app[6e82d04c2d3387] sin [info]main.panicHandler({0x9a7320?, 0xc0001cce70})
2023-09-02T09:18:30Z app[6e82d04c2d3387] sin [info]	/go/src/github.com/fly-apps/fly-postgres/cmd/start/main.go:188 +0x55
2023-09-02T09:18:30Z app[6e82d04c2d3387] sin [info]main.main()
2023-09-02T09:18:30Z app[6e82d04c2d3387] sin [info]	/go/src/github.com/fly-apps/fly-postgres/cmd/start/main.go:65 +0xe5e
2023-09-02T09:18:30Z app[6e82d04c2d3387] sin [info] INFO Main child exited normally with code: 2
2023-09-02T09:18:30Z app[6e82d04c2d3387] sin [info] INFO Starting clean up.
2023-09-02T09:18:30Z app[6e82d04c2d3387] sin [info] INFO Umounting /dev/vdb from /data
2023-09-02T09:18:30Z app[6e82d04c2d3387] sin [info] WARN hallpass exited, pid: 265, status: signal: 15 (SIGTERM)
2023-09-02T09:18:30Z app[6e82d04c2d3387] sin [info]2023/09/02 09:18:30 listening on [fdaa:0:4a0f:a7b:ead:68aa:124c:2]:22 (DNS: [fdaa::3]:53)
2023-09-02T09:18:31Z app[6e82d04c2d3387] sin [info][    2.294952] reboot: Restarting system
2023-09-02T09:18:32Z health[6e82d04c2d3387] sin [error]Health check for your postgres role has failed. Your cluster's membership is inconsistent.

2023-09-02T09:18:50Z health[6e82d04c2d3387] sin [error]Health check for your postgres role has failed. Your cluster's membership is inconsistent.

Adding versioning for each new docker image released postgres-flex:15.3_2023_08_31 would help with troubleshooting docker images. Ideally I find out what the rootcause of the issue is but if it isnt possible is there an archive I can access older images.

Did you check the status on the machine where your Postgres DB is running? I’m seeing similar errors but noticed that the machine is taken down by “Emergency maintenance” by the fly.io engineers.

Doesn’t appear to be the case I have tried launching in both hkg and sin regions

State: stopped

VM
  ID            = 5683d13dad3e18                      
  Instance ID   = 01H9BXC830S8HJPZQKFKMGJ1T1          
  State         = stopped                             
  Image         = ks0m1c/fly_vector:latest (v0.0.43)  
  Name          = autumn-paper-9162                   
  Private IP    = fdaa:0:4a0f:a7b:a7:c58f:f628:2      
  Region        = hkg                                 
  Process Group =                                     
  CPU Kind      = shared                              
  vCPUs         = 2                                   
  Memory        = 4096                                
  Created       = 2023-09-02T21:39:25Z                
  Updated       = 2023-09-02T21:40:52Z                
  Entrypoint    =                                     
  Command       =                                     
  Volume        = vol_24o9q6jw9w55x6q4                

Event Logs
STATE   	EVENT	SOURCE	TIMESTAMP                    	INFO                                              
stopped 	exit 	flyd  	2023-09-03T05:40:52.047+08:00	exit_code=2,oom_killed=false,requested_stop=false	
started 	start	flyd  	2023-09-03T05:40:49.7+08:00  	
starting	start	flyd  	2023-09-03T05:40:49.058+08:00	
stopped 	exit 	flyd  	2023-09-03T05:40:48.113+08:00	exit_code=2,oom_killed=false,requested_stop=false	
started 	start	flyd  	2023-09-03T05:40:45.729+08:00	

For anyone experiencing a similar issue it could be an issue with the postgres image built by your local Docker platform try to use the remote builder in Fly

The steps below are courtesy of Kyle McLaren.

Create a new app: fly create my-app, then using your prefered Dockerfile run fly deploy --build-only --push --remote-only --dockerfile Dockerfile --app my-app

Then using the built image create the new postgres app using fly create pg --image-ref registry.fly.io/myapp:deployment...

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.