Can't connect to Postgres Cluster with Hanami App

[dcrossney]

I am trying to deploy a Hanami app (a ruby framework–process should be similar to Sinatra), but I’m having trouble executing db setup tasks to create and migrate the database.

DATABASE_URL was setup with fly launch. I have not changed it, but I notice that the URL does not include a database name, just user, password, and host connection.

My dockerfile, below, was created by fly launch and I added some items. fly.toml is untouched.

ARG RUBY_VERSION=3.2.2
FROM ruby:$RUBY_VERSION-slim as base

# Rack app lives here
WORKDIR /app

# Set production environment
ENV HANAMI_ENV="production" \
    HANAMI_PORT=8080 \
    BUNDLE_WITHOUT="development:test" \
    BUNDLE_DEPLOYMENT="1"

# Update gems and bundler
RUN gem update --system --no-document && \
    gem install -N bundler


# Throw-away build stage to reduce size of final image
FROM base as build

# Install packages needed to build gems
RUN apt-get update -qq && \
    apt-get install --no-install-recommends -y build-essential libpq-dev

# Install application gems
COPY Gemfile* .
RUN bundle install


# Final stage for app image
FROM base

# Run and own the application files as a non-root user for security
RUN useradd ruby --home /app --shell /bin/bash
USER ruby:ruby

# Copy built artifacts: gems, application
COPY --from=build /usr/local/bundle /usr/local/bundle
COPY --from=build --chown=ruby:ruby /app /app

# Copy application code
COPY --chown=ruby:ruby . .

# Entrypoint prepares the database
ENTRYPOINT ["./bin/fly-entrypoint"]

# Start the server
EXPOSE 8080
CMD ["bundle", "exec", "rackup", "--host", "0.0.0.0", "--port", "8080"]

I added libpq-dev to the apt-get install command on line 22. I also added the ENV block and the Entrypoint block to run my script. If all goes well, the script should execute db commands and then execute the CMD from the dockerfile.

My entrypoint script looks like this:

#!/usr/bin/env bash
# exit on error
set -o errexit

# Uncomment for first deploy
bundle exec bin/hanami db create

# Uncomment after first deploy
# bundle exec bin/hanami db migrate

# Execute the container's main process (CMD in the dockerfile)
exec "$@"

I have been trying to execute the create command to get the database set up. Once setup, I would like to always run the migrate command on each deploy.

These are the relevant log entries. All seems well until this point.:

[info] INFO Preparing to run: `./bin/entrypoint bundle exec rackup --host 0.0.0.0 --port 8080` as ruby
[info] 2023/10/03 19:39:03 listening on [fdaa:2:96dd:a7b:d828:3375:22ab:2]:22 (DNS: [fdaa::3]:53)
[info] sh: 1: psql: not found
[info] => failed to create database <app-name>_app
[info] INFO Main child exited normally with code: 127
[info] INFO Starting clean up.
[info] WARN hallpass exited, pid: 304, status: signal: 15 (SIGTERM)
[info] 2023/10/03 19:39:05 listening on [fdaa:2:96dd:a7b:d828:3375:22ab:2]:22 (DNS: [fdaa::3]:53)
[info] [ 3.287869] reboot: Restarting system 

It seems to call the entrypoint script correctly, and then fails on the third line, sh: 1: psql: not found.

The database cluster exists and I can connect to it with fly postgres connect. Fly created a default <app-name>_app database, but Hanami expects to create a database named <app-name>_production. Therefore, line four confuses me: => failed to create database <app-name>_app. The create command should not use this database name (as far as I know).

Just for completeness, if I try to use migrate, the Hanami app itself crashes (presumably due to the missing database).

How should I proceed? I don’t know how to troubleshoot this further. Is it possible that psql (which must be used by the bin/hanami commands) is no longer available in this stage of the dockerfile build? If so, how do I fix that?

Any help is appreciated!

[dcrossney]

I created a plain-vanilla Hanami app, ran fly launch and deployed (no changes to the generated dockerfile) with no problem. This leads me to conclude that adding persistence to the app is causing the problem.

Another clue is here in one of the error messages I’m getting now:

den [info] <internal:/usr/local/lib/ruby/site_ruby/3.2.0/rubygems/core_ext/kernel_require.rb>:38:in 'require': LoadError: libpq.so.5: cannot open shared object file: No such file or directory - /usr/local/bundle/gems/pg-1.5.3/lib/pg_ext.so (Sequel::AdapterNotFound)

It’s a long line, but I want to zoom in on one portion:

LoadError: libpq.so.5: cannot open shared object file: No such file or directory

This file, when installed properly, does not live in /usr/local/bundle, and therefore is not getting copied from the build stage to the final stage. I will try to figure that issue out next . . .

[rubys]

Try adding the following before the COPY --link statements:

# Install packages needed for deployment
RUN apt-get update -qq && \
    apt-get install --no-install-recommends -y postgresql-client && \
    rm -rf /var/lib/apt/lists /var/cache/apt/archives

[dcrossney]

Thank you @rubys for your suggestion! That was definitely a piece of the puzzle!

I ended up customizing the official Fly.io Rails Dockerfile and came around to the following.

# syntax = docker/dockerfile:1

ARG RUBY_VERSION=3.2.2
FROM ruby:$RUBY_VERSION-slim as base

# Hanami app lives here
WORKDIR /hanami

# Set production environment
ENV HANAMI_ENV="production" \
    BUNDLE_WITHOUT="development:test" \
    BUNDLE_DEPLOYMENT="1"

# Update gems and bundler
RUN gem update --system --no-document && \
    gem install -N bundler


# Throw-away build stage to reduce size of final image
FROM base as build

# Install packages needed to build gems
RUN apt-get update -qq && \
    apt-get install --no-install-recommends -y build-essential libpq-dev

# Install application gems
COPY --link Gemfile Gemfile.lock ./
RUN bundle install && \
    rm -rf ~/.bundle/ $BUNDLE_PATH/ruby/*/cache $BUNDLE_PATH/ruby/*/bundler/gems/*/.git

# Copy application code
COPY --link . .


# Final stage for app image
FROM base

# Install packages needed for deployment
RUN apt-get update -qq && \
    apt-get install --no-install-recommends -y curl postgresql-client && \
    rm -rf /var/lib/apt/lists /var/cache/apt/archives

# Run and own the application files as a non-root user for security
RUN useradd hanami --home /hanami --shell /bin/bash
USER hanami:hanami

# Copy built artifacts: gems, application
COPY --from=build /usr/local/bundle /usr/local/bundle
COPY --from=build --chown=hanami:hanami /hanami /hanami

# Entrypoint prepares the database
ENTRYPOINT ["./bin/fly-entrypoint"]

# Start the server
EXPOSE 8080
CMD ["bundle", "exec", "rackup", "--host", "0.0.0.0", "--port", "8080"]

I had already created my database by hand, so my fly-entrypoint script just runs migrations before executing the dockerfile CMD.

#!/usr/bin/env bash
# exit on error
set -o errexit

# Uncomment for first deploy
# bundle exec bin/hanami db setup
# bundle exec bin/hanami db create

# Uncomment after first deploy
bundle exec ./bin/hanami db migrate

# Execute the container's main process (CMD in the dockerfile)
exec "$@"

This finally worked. I was then able to fly ssh console and run bundle exec bin/hanami db seed to load my seed data.

What an ordeal! Thanks again for taking the time to point me in the right direction.

[dcrossney]

For completeness, I did the following to create my database:

fly ssh console -a <db-cluster-app-name>
createdb -h 127.0.0.1 -U postgres -W  <db-name>

I used the password generated by fly launch when I created the db-cluster. For me, the database name I needed was <app-name>_production.

I could then use fly postgres console <db-cluster-app-name> to open a psql session and poke around my databases. \l (that’s a lower-case L, for “list”) showed me that my production database was present. The migrate command in my fly-entrypoint script then got everything ready and allowed the app to launch properly.