ActiveRecord::ConnectionNotEstablished: could not connect to server: Connection refused

,
1

First time attempt to migrate basic Rails app (with postgres) from Heroku. fly launch succeeded. fly deploy fails at point of database migration. I followed the ‘Migrate from Heroku documentation’ and during fly launch responded ‘Y’ to: “Would you like to set up a Postgresql database now?” though I’m not convinced it got created…Could I have help to resolve please. Thanks Daniel

--> This release will not be available until the release command succeeds.
         Starting instance
         Configuring virtual machine
         Pulling container image
         Unpacking image
         Preparing kernel init
         Configuring firecracker
         Starting virtual machine
         Starting init (commit: 81d5330)...
         Setting up swapspace version 1, size = 512 MiB (536866816 bytes)
         no label, UUID=40c3f966-630f-4898-9633-c70a6d574d18
         Preparing to run: `bin/rails fly:release` as root
         2022/11/23 22:51:41 listening on [fdaa:0:e7d1:a7b:7e:622:1513:2]:22 (DNS: [fdaa::3]:53)
         rails aborted!
         ActiveRecord::ConnectionNotEstablished: could not connect to server: Connection refused
                Is the server running on host "::1" and accepting
                TCP/IP connections on port 5432?
         could not connect to server: Connection refused
         /app/vendor/bundle/ruby/3.0.0/gems/activerecord-6.1.6.1/lib/active_record/connection_adapters/postgresql_adapter.rb:83:in `rescue in new_client'
         /app/vendor/bundle/ruby/3.0.0/gems/activerecord-6.1.6.1/lib/active_record/connection_adapters/postgresql_adapter.rb:77:in `new_client'
         /app/vendor/bundle/ruby/3.0.0/gems/activerecord-6.1.6.1/lib/active_record/connection_adapters/abstract/connection_pool.rb:882:in `public_send'
         /app/vendor/bundle/ruby/3.0.0/gems/activerecord-6.1.6.1/lib/active_record/connection_adapters/abstract/connection_pool.rb:905:in `try_to_checkout_new_connection'
         /app/vendor/bundle/ruby/3.0.0/gems/activerecord-6.1.6.1/lib/active_record/connection_adapters/abstract/connection_pool.rb:866:in `acquire_connection'
         /app/vendor/bundle/ruby/3.0.0/gems/activerecord-6.1.6.1/lib/active_record/connection_adapters/abstract/connection_pool.rb:588:in `checkout'
         /app/vendor/bundle/ruby/3.0.0/gems/activerecord-6.1.6.1/lib/active_record/connection_adapters/abstract/connection_pool.rb:428:in `connection'
         /app/vendor/bundle/ruby/3.0.0/gems/activerecord-6.1.6.1/lib/active_record/connection_handling.rb:327:in `retrieve_connection'
         /app/vendor/bundle/ruby/3.0.0/gems/activerecord-6.1.6.1/lib/active_record/connection_handling.rb:283:in `connection'
         /app/vendor/bundle/ruby/3.0.0/gems/activerecord-6.1.6.1/lib/active_record/tasks/database_tasks.rb:237:in `migrate'
         /app/vendor/bundle/ruby/3.0.0/gems/activerecord-6.1.6.1/lib/active_record/railties/databases.rake:92:in `block (3 levels) in <main>'
         /app/vendor/bundle/ruby/3.0.0/gems/activerecord-6.1.6.1/lib/active_record/railties/databases.rake:90:in `each'
         /app/vendor/bundle/ruby/3.0.0/gems/activerecord-6.1.6.1/lib/active_record/railties/databases.rake:90:in `block (2 levels) in <main>'
         /app/vendor/bundle/ruby/3.0.0/gems/rake-13.0.6/lib/rake/task.rb:281:in `block in execute'
         /app/vendor/bundle/ruby/3.0.0/gems/rake-13.0.6/lib/rake/task.rb:281:in `execute'
         /app/vendor/bundle/ruby/3.0.0/gems/rake-13.0.6/lib/rake/task.rb:219:in `block in invoke_with_call_chain'
         /app/vendor/bundle/ruby/3.0.0/gems/rake-13.0.6/lib/rake/task.rb:199:in `synchronize'
         /app/vendor/bundle/ruby/3.0.0/gems/rake-13.0.6/lib/rake/task.rb:199:in `invoke_with_call_chain'
         /app/vendor/bundle/ruby/3.0.0/gems/rake-13.0.6/lib/rake/task.rb:243:in `block in invoke_prerequisites'
         /app/vendor/bundle/ruby/3.0.0/gems/rake-13.0.6/lib/rake/task.rb:241:in `each'
         /app/vendor/bundle/ruby/3.0.0/gems/rake-13.0.6/lib/rake/task.rb:199:in `synchronize'
         /app/vendor/bundle/ruby/3.0.0/gems/rake-13.0.6/lib/rake/task.rb:199:in `invoke_with_call_chain'
         /app/vendor/bundle/ruby/3.0.0/gems/rake-13.0.6/lib/rake/task.rb:188:in `invoke'
         /app/vendor/bundle/ruby/3.0.0/gems/rake-13.0.6/lib/rake/application.rb:116:in `block (2 levels) in top_level'
         /app/vendor/bundle/ruby/3.0.0/gems/rake-13.0.6/lib/rake/application.rb:116:in `each'
         /app/vendor/bundle/ruby/3.0.0/gems/rake-13.0.6/lib/rake/application.rb:116:in `block in top_level'
         /app/vendor/bundle/ruby/3.0.0/gems/railties-6.1.6.1/lib/rails/commands/rake/rake_command.rb:24:in `block (2 levels) in perform'
         /app/vendor/bundle/ruby/3.0.0/gems/rake-13.0.6/lib/rake/application.rb:186:in `standard_exception_handling'
         /app/vendor/bundle/ruby/3.0.0/gems/railties-6.1.6.1/lib/rails/commands/rake/rake_command.rb:24:in `block in perform'
         /app/vendor/bundle/ruby/3.0.0/gems/rake-13.0.6/lib/rake/rake_module.rb:59:in `with_application'
         /app/vendor/bundle/ruby/3.0.0/gems/railties-6.1.6.1/lib/rails/commands/rake/rake_command.rb:18:in `perform'
         /app/vendor/bundle/ruby/3.0.0/gems/railties-6.1.6.1/lib/rails/command.rb:50:in `invoke'
         /app/vendor/bundle/ruby/3.0.0/gems/bootsnap-1.13.0/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:32:in `require'
         /app/vendor/bundle/ruby/3.0.0/gems/bootsnap-1.13.0/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:32:in `require'
         Caused by:
         PG::ConnectionBad: could not connect to server: Connection refused
                TCP/IP connections on port 5432?
         could not connect to server: Connection refused
                Is the server running on host "127.0.0.1" and accepting
                TCP/IP connections on port 5432?
         /app/vendor/bundle/ruby/3.0.0/gems/pg-1.4.3/lib/pg/connection.rb:751:in `connect_to_hosts'
         /app/vendor/bundle/ruby/3.0.0/gems/pg-1.4.3/lib/pg/connection.rb:672:in `new'
         /app/vendor/bundle/ruby/3.0.0/gems/pg-1.4.3/lib/pg.rb:69:in `connect'
         /app/vendor/bundle/ruby/3.0.0/gems/activerecord-6.1.6.1/lib/active_record/connection_adapters/postgresql_adapter.rb:78:in `new_client'
         /app/vendor/bundle/ruby/3.0.0/gems/activerecord-6.1.6.1/lib/active_record/connection_adapters/postgresql_adapter.rb:37:in `postgresql_connection'
         /app/vendor/bundle/ruby/3.0.0/gems/activerecord-6.1.6.1/lib/active_record/connection_adapters/abstract/connection_pool.rb:882:in `public_send'
         /app/vendor/bundle/ruby/3.0.0/gems/activerecord-6.1.6.1/lib/active_record/connection_adapters/abstract/connection_pool.rb:882:in `new_connection'
         /app/vendor/bundle/ruby/3.0.0/gems/activerecord-6.1.6.1/lib/active_record/connection_adapters/abstract/connection_pool.rb:926:in `checkout_new_connection'
         /app/vendor/bundle/ruby/3.0.0/gems/activerecord-6.1.6.1/lib/active_record/connection_adapters/abstract/connection_pool.rb:905:in `try_to_checkout_new_connection'
         /app/vendor/bundle/ruby/3.0.0/gems/activerecord-6.1.6.1/lib/active_record/connection_adapters/abstract/connection_pool.rb:866:in `acquire_connection'
         /app/vendor/bundle/ruby/3.0.0/gems/activerecord-6.1.6.1/lib/active_record/connection_adapters/abstract/connection_pool.rb:588:in `checkout'
         /app/vendor/bundle/ruby/3.0.0/gems/activerecord-6.1.6.1/lib/active_record/connection_adapters/abstract/connection_pool.rb:1128:in `retrieve_connection'
         /app/vendor/bundle/ruby/3.0.0/gems/activerecord-6.1.6.1/lib/active_record/connection_handling.rb:327:in `retrieve_connection'
         /app/vendor/bundle/ruby/3.0.0/gems/activerecord-6.1.6.1/lib/active_record/tasks/database_tasks.rb:237:in `migrate'
         /app/vendor/bundle/ruby/3.0.0/gems/activerecord-6.1.6.1/lib/active_record/railties/databases.rake:92:in `block (3 levels) in <main>'
         /app/vendor/bundle/ruby/3.0.0/gems/activerecord-6.1.6.1/lib/active_record/railties/databases.rake:90:in `block (2 levels) in <main>'
         /app/vendor/bundle/ruby/3.0.0/gems/rake-13.0.6/lib/rake/task.rb:281:in `block in execute'
         /app/vendor/bundle/ruby/3.0.0/gems/rake-13.0.6/lib/rake/task.rb:281:in `each'
         /app/vendor/bundle/ruby/3.0.0/gems/rake-13.0.6/lib/rake/task.rb:281:in `execute'
         /app/vendor/bundle/ruby/3.0.0/gems/rake-13.0.6/lib/rake/task.rb:219:in `block in invoke_with_call_chain'
         /app/vendor/bundle/ruby/3.0.0/gems/rake-13.0.6/lib/rake/task.rb:199:in `synchronize'
         /app/vendor/bundle/ruby/3.0.0/gems/rake-13.0.6/lib/rake/task.rb:243:in `block in invoke_prerequisites'
         /app/vendor/bundle/ruby/3.0.0/gems/rake-13.0.6/lib/rake/task.rb:241:in `each'
         /app/vendor/bundle/ruby/3.0.0/gems/rake-13.0.6/lib/rake/task.rb:241:in `invoke_prerequisites'
         /app/vendor/bundle/ruby/3.0.0/gems/rake-13.0.6/lib/rake/application.rb:160:in `invoke_task'
         /app/vendor/bundle/ruby/3.0.0/gems/rake-13.0.6/lib/rake/application.rb:116:in `block (2 levels) in top_level'
         /app/vendor/bundle/ruby/3.0.0/gems/rake-13.0.6/lib/rake/application.rb:116:in `each'
         /app/vendor/bundle/ruby/3.0.0/gems/rake-13.0.6/lib/rake/application.rb:116:in `block in top_level'
         /app/vendor/bundle/ruby/3.0.0/gems/rake-13.0.6/lib/rake/application.rb:125:in `run_with_threads'
         /app/vendor/bundle/ruby/3.0.0/gems/rake-13.0.6/lib/rake/application.rb:110:in `top_level'
         /app/vendor/bundle/ruby/3.0.0/gems/railties-6.1.6.1/lib/rails/commands/rake/rake_command.rb:24:in `block (2 levels) in perform'
         /app/vendor/bundle/ruby/3.0.0/gems/rake-13.0.6/lib/rake/application.rb:186:in `standard_exception_handling'
         /app/vendor/bundle/ruby/3.0.0/gems/railties-6.1.6.1/lib/rails/commands/rake/rake_command.rb:24:in `block in perform'
         /app/vendor/bundle/ruby/3.0.0/gems/rake-13.0.6/lib/rake/rake_module.rb:59:in `with_application'
         /app/vendor/bundle/ruby/3.0.0/gems/railties-6.1.6.1/lib/rails/command.rb:50:in `invoke'
         /app/vendor/bundle/ruby/3.0.0/gems/railties-6.1.6.1/lib/rails/commands.rb:18:in `<main>'
         /app/vendor/bundle/ruby/3.0.0/gems/bootsnap-1.13.0/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:32:in `require'
         Tasks: TOP => fly:release => db:migrate
         (See full trace by running task with --trace)
         Starting clean up.
Error release command failed, deployment aborted
2

FIrst, check fly secrets list and fly postgres list.

If you had created a database, you should see DATABASE_URL in the first list, and a generated name ending in -db in the second list. Given the error above, I’m hoping and guessing that you don’t see DATABASE_URL.

If you don’t see DATABASE_URL, delete the database (if any) using fly apps destroy. Then use fly postgres create to create a database and take a note of the connection string. Finally, enter fly postgres attach from within your application directory, and it will set the appropriate secret.

1 Like
3

Yes, as you expect, secrets list - no DATABASE_URL, postgres list - a generated name ending in -db.
I followed the steps., cluster attached, secret added. All good. I assume then deploy? But fly deploy now gives this odd warning and error?

==> Verifying app config
--> Verified app config
==> Building image
Remote builder fly-builder-twilight-night-6077 ready
==> Creating build context
--> Creating build context done
==> Building image with Docker
--> docker host: 20.10.12 linux x86_64
Sending build context to Docker daemon    293kB
[+] Building 1.7s (26/26) FINISHED                                                                                                                                     
 => CACHED [internal] load remote build context                                                                                                                   0.0s
 => CACHED copy /context /                                                                                                                                        0.0s
 => resolve image config for docker.io/docker/dockerfile:experimental                                                                                             0.8s
 => CACHED docker-image://docker.io/docker/dockerfile:experimental@sha256:600e5c62eedff338b3f7a0850beb7c05866e0ef27b2d2e8c02aa468e78496ff5                        0.0s
 => [internal] load metadata for quay.io/evl.ms/fullstaq-ruby:3.0.1-jemalloc-buster-slim                                                                          0.4s
 => [base 1/6] FROM quay.io/evl.ms/fullstaq-ruby:3.0.1-jemalloc-buster-slim@sha256:fdfd9290ebf292b1390b117bc31cb67b39ecabb266d7996b71c512a6e6802e29               0.0s
 => CACHED [base 2/6] RUN mkdir /app                                                                                                                              0.0s
 => CACHED [base 3/6] WORKDIR /app                                                                                                                                0.0s
 => CACHED [base 4/6] RUN mkdir -p tmp/pids                                                                                                                       0.0s
 => CACHED [base 5/6] RUN curl https://get.volta.sh | bash                                                                                                        0.0s
 => CACHED [base 6/6] RUN volta install node@16.15.0 yarn@1.22.19 &&     gem update --system --no-document &&     gem install -N bundler -v 2.3.21                0.0s
 => CACHED [stage-4 1/8] RUN --mount=type=cache,id=prod-apt-cache,sharing=locked,target=/var/cache/apt     --mount=type=cache,id=prod-apt-lib,sharing=locked,tar  0.0s
 => CACHED [build_deps 1/1] RUN --mount=type=cache,id=dev-apt-cache,sharing=locked,target=/var/cache/apt     --mount=type=cache,id=dev-apt-lib,sharing=locked,ta  0.0s
 => CACHED [gems 1/2] COPY Gemfile* ./                                                                                                                            0.0s
 => CACHED [gems 2/2] RUN bundle install &&  rm -rf vendor/bundle/ruby/*/cache                                                                                    0.0s
 => CACHED [stage-4 2/8] COPY --from=gems /app /app                                                                                                               0.0s
 => CACHED [stage-4 3/8] COPY --from=gems /usr/lib/fullstaq-ruby/versions /usr/lib/fullstaq-ruby/versions                                                         0.0s
 => CACHED [stage-4 4/8] COPY --from=gems /usr/local/bundle /usr/local/bundle                                                                                     0.0s
 => CACHED [node_modules 1/3] COPY package*json ./                                                                                                                0.0s
 => CACHED [node_modules 2/3] COPY yarn.* ./                                                                                                                      0.0s
 => CACHED [node_modules 3/3] RUN yarn install                                                                                                                    0.0s
 => CACHED [stage-4 5/8] COPY --from=node_modules /app/node_modules /app/node_modules                                                                             0.0s
 => CACHED [stage-4 6/8] COPY . .                                                                                                                                 0.0s
 => CACHED [stage-4 7/8] RUN chmod +x /app/bin/* &&     sed -i 's/ruby.exe\r*/ruby/' /app/bin/* &&     sed -i '/^#!/aDir.chdir File.expand_path("..", __dir__)'   0.0s
 => CACHED [stage-4 8/8] RUN bin/rails fly:build                                                                                                                  0.0s
 => exporting to image                                                                                                                                            0.0s
 => => exporting layers                                                                                                                                           0.0s
 => => writing image sha256:5ec159350355b0c963ed2454dff8c99cff2a6f91cb18d61058aafc1ab8e36d0a                                                                      0.0s
 => => naming to registry.fly.io/bridge-scores:deployment-01GJKGY7TFHX35FPCX6G71G9NK                                                                              0.0s
--> Building image done
==> Pushing image to fly
The push refers to repository [registry.fly.io/bridge-scores]
e7d1038a9f22: Pushed 
9dec533a5a7d: Pushed 
e4dddaba814a: Pushed 
f6bff13b44e5: Pushed 
a66803a4e516: Pushed 
33af412630ef: Pushed 
b11e250bc738: Pushed 
345b43e09d37: Pushed 
a604946e65ba: Pushed 
db0001e00ac7: Pushed 
b4161b6026c2: Pushed 
5f70bf18a086: Pushed 
ab5353ffda06: Pushed 
a00c7f3f4a7f: Pushed 
764055ebc9a7: Pushed 
deployment-01GJKGY7TFHX35FPCX6G71G9NK: digest: sha256:94740e00cbcfc53fb86f64c36af206cdcf22fcdd37ec9ffba33f714c4f1e430a size: 3475
--> Pushing image done
image: registry.fly.io/bridge-scores:deployment-01GJKGY7TFHX35FPCX6G71G9NK
image size: 666 MB
? This feature is highly experimental and may produce unexpected results. Proceed? Yes
Running release command: bin/rails fly:release ✓
Error release command failed - aborting deployment. failed to wait for VM 0e286d79a3d786 in started state: failed to wait for machine to be ready
4

OK, you have definitely wandered off into experimental territory. Unless you have a specific experimental feature you are looking to make use if, it probably best at this point to start clean. If you type fly apps list you will see your app as well as your db. Destroy both by running fly apps destroy followed by the name of the app, then rerun the command with the name of the db. Next run fly launch and reply yes each time it asks you to replace a file, and then yes to creating a postgre database. If you app makes use of websockets or caching, also say yes to redis, otherwise say no.

5

yes somehow…
back on track now, possibly. Deployment successful, but database transfer error:

$ fly ssh console
Connecting to fdaa:0:e7d1:a7b:7e:d68c:1067:2... complete
# pg_dump -Fc --no-acl --no-owner -d $HEROKU_DATABASE_URL | pg_restore --verbose --clean --no-acl --no-owner -d $DATABASE_URL
pg_dump: server version: 13.9 (Ubuntu 13.9-1.pgdg20.04+1); pg_dump version: 11.18 (Debian 11.18-0+deb10u1)
pg_dump: aborting because of server version mismatch
pg_restore: [archiver] input file is too short (read 0, expected 5)

the documentation pre-empts this problem “You may need to upgrade your Heroku database to match the version of the source Fly database…”
From Heroku, i can see the postgres version is 13.9 and the newly attached version on Fly seems to be 14.4. Is this the issue? What is the reference to 11.18? On Heroku, I only have the option to upgrade to 14.5. Is that going to do me any good if the versions have to exactly match?

6

There actually are three versions involved. Your source (Heroku) is 13.9. Your app is 11.18. And the DB is running 14.4.

Try reapplying this change: Error failed to fetch an image or build from source...Unknown desc = quay.io/evl.ms/fullstaq-ruby:3.0.1-jemalloc-bullseye-slim: not found - #11 by SimBed :

ARG RUBY_VERSION=3.0.1
ARG VARIANT=jemalloc-buster-slim
7

that I had already done…

8

OK, we are either going to need debian bullseye, or to add instructions to your Dockerfile to install a later version of postgres on top of buster.

If you can upgrade your version of Ruby to 3.0.2 or later, you can change buster to bullseye in the variant and be up and running.

If you want to stay with Ruby 3.0.1, you can follow these instructions to install postgresql 14 on debian buster: PG database migration from Heroku not working - #19 by rjdavies

1 Like
9

I went for the Ruby 3.0.2 upgrade and

ARG RUBY_VERSION=3.0.2
ARG VARIANT=jemalloc-bullseye-slim

in the Docker file.
Deploy successful. Data transfer successful. Thanks for sticking with me.

1 Like
10

And in return, thanks for sticking with it. I’m continuing to do more testing, but the direction I’m heading for the future is to generate Dockerfiles that put the debian release name in the variant and to install a version of postgresql that matches the deployed server that fly provides.