Transferring Postgres from Heroku

kaspermeyer · August 27, 2022, 7:50am

I’m attempting to migrate a Rails app from Heroku, and I’ve followed the instructions without any issues up to the section about database migration. I ran the command laid out from the guide:

$ pg_dump --no-owner -C -d $HEROKU_DATABASE_URL | psql -d $DATABASE_URL

However, the following error occurs:

pg_dump: option requires an argument -- d
Try "pg_dump --help" for more information.
psql: option requires an argument -- d
Try "psql --help" for more information.

I have tried replacing the two variables with the actual database URL’s, but the $DATABASE_URL is an internal address, which makes me think I’m missing something.

Am I supposed to run the command from within my VM? Hope someone can help

Update: I attempted to run the command from within the app container with fly ssh console but got the following error:

$ pg_dump --no-owner -C -d $HEROKU_DATABASE_URL | psql -d $DATABASE_URL
pg_dump: server version: 14.4 (Ubuntu 14.4-1.pgdg20.04+1); pg_dump version: 11.17 (Debian 11.17-0+deb10u1)
pg_dump: aborting because of server version mismatch

tj1 · August 27, 2022, 8:59am

I think that you have to ssh to your app before running these commands.

fly ssh console

Otherwise, you can also load a database dump from your machine by adjusting the following commands:

flyctl proxy 15432:5432 -a my-postgres-app
psql -U <your-user> -h localhost -p 15432 <your-db> < <your_dump_file.sql>

flyHigh · August 28, 2022, 1:33am

This works for me from the command line when at my Rails project
The first may not be necessary depending on how you have Heroku set up

   heroku login -I
    heroku pg:backups:capture
    heroku pg:backups:download

This downloads the database to your project folder. Uploading to fly.io is another story and in fact that is what I’m trying to figure out.

I hadn’t read those instructions when I first posted, but it makes some sense to download and upload rather than trying to do in one step. When I deploy I’m getting errors about a database table not being seen, but I’ve got the rails app running in localhost also. It seems the fly deploy uploads the database too. I just started with this yesterday, so haven’t gotten very far and will separately post if I can’t figure it out.

tj1 · August 28, 2022, 5:07am

It’s the fly proxy command with psql that I listed above.

wiznaibus · August 29, 2022, 10:05am

flyctl proxy 15432:5432 -a my-postgres-app just hangs for me. No error.

tj1 · August 29, 2022, 2:44pm

That sets up a proxy from the app port 5432 to local port 15432. Then you can use psql against localhost with port 15432.

wiznaibus · August 29, 2022, 2:49pm

Understood. When I set up the proxy on the app port, it hangs. I can’t even get to step two.

tj1 · August 29, 2022, 3:07pm

It opens up a foreground process that should remain open. You’ll need to use psql in a second terminal.

$ fly proxy 15432:5432 -a my-app
Proxying local port 15432 to remote [my-app.internal]:5432

wiznaibus · August 29, 2022, 3:20pm

Ahhh, did not know that. Will try when I’m back.

dornby · August 29, 2022, 5:19pm

I’m having the same issues as @wiznaibus here.
I have flyctl proxy 15432:5432 -a my-app in a terminal and it’s proxying

But when I run

psql -U hora -h localhost -p 15432 my-postgres-db < data.sql

I get

zsh: no such file or directory: data.sql

I have tried doing the same with latest.dump downloaded straight from heroku and that was saved at the root of my project:

psql -U hora -h localhost -p 15432 my-postgres-db < latest.dump

but this one just hangs Do you know how to get that .sql file to dump onto my db?

tj1 · August 29, 2022, 8:02pm

Trying to be clearer this time.

Obtain a sql dump file from heroku and have it locally.
Proxy the postgres from fly to localhost.

$ fly proxy 15432:5432 -a my-app
Proxying local port 15432 to remote [my-app.internal]:5432

Use psql like you would normally, but on port 15432. For instance, you should be able to connect, etc.


$ psql -U your_user -h localhost -p 15432 
psql #

$ psql -U your_user -h localhost -p 15432 < your_dump_file
...this will load the database dump and give some output...

@dornby - see step 3. If updated the user/db info, make sure you can connect directly first before trying to load the dump file.

jedi · August 30, 2022, 9:23am

@tj1 thank you for your approach
but i’m having trouble
hope you can assist
(this is google translate)

tj1 · August 30, 2022, 2:59pm

You’ve connected correctly, but you need to use the correct database name.

psql -U postgres -h localhost -p 15432 -d your_database < latest.dump

You can find out the name of your databases by connecting locally and listing them with \l

postgres=# \l
                                    List of databases
   Name    |   Owner    | Encoding |  Collate   |   Ctype    |     Access privileges   
  
-----------+------------+----------+------------+------------+-------------------------
--
 main      | flypgadmin | UTF8     | en_US.utf8 | en_US.utf8 | 
 postgres  | flypgadmin | UTF8     | en_US.utf8 | en_US.utf8 | 
 template0 | flypgadmin | UTF8     | en_US.utf8 | en_US.utf8 | =c/flypgadmin           
 +
           |            |          |            |            | flypgadmin=CTc/flypgadmi
n
 template1 | flypgadmin | UTF8     | en_US.utf8 | en_US.utf8 | =c/flypgadmin           
 +
           |            |          |            |            | flypgadmin=CTc/flypgadmi
n
(4 rows)

dornby · August 30, 2022, 3:59pm

Wuhu it finally worked for me.

Thank you so much @tj1 for leading me on the way. Here is a precise breakdown of what I did for it to work, with a lot of repetition from @tj1 's advice:

Get your latest dump by running

heroku login
heroku pg:backups:capture
heroku pg:backups:download

In one terminal, launch:

fly proxy 15432:5432 -a my-db-app-name
# make sure it's the name of your db-app, not your main app

In another terminal, list your db names to get the exact name of your DB with \l:

psql -U postgres -h localhost -p 15432
postgres=> \l

   Name    |   Owner    | Encoding |  Collate   |   Ctype    |     Access privileges
-----------+------------+----------+------------+------------+---------------------------
 foobar  | flypgadmin | UTF8     | en_US.utf8 | en_US.utf8 |
 postgres  | flypgadmin | UTF8     | en_US.utf8 | en_US.utf8 |
 template0 | flypgadmin | UTF8     | en_US.utf8 | en_US.utf8 | =c/flypgadmin            +
           |            |          |            |            | flypgadmin=CTc/flypgadmin
 template1 | flypgadmin | UTF8     | en_US.utf8 | en_US.utf8 | =c/flypgadmin            +
           |            |          |            |            | flypgadmin=CTc/flypgadmin
(4 rows)

Once you have your name, here foobar, import your dump with pg_restore:

pg_restore -U postgres -h localhost -p 15432 -d foobar < latest.dump

And voilà, or at least it worked for me.

jedi · August 31, 2022, 10:30am

@tj1 @dornby
Thanks for the guidance of the two masters
I also succeeded

But I have a new problem.
I use “pgadmin” or “tableplus” software to connect postgresql db.

An error occurs:
could not translate host name “dbname-db.internal” to address: nodename nor servname provided, or not known.

“host name” I have tried the following but failed:
top2.nearest.of.dbname-db.internal
dbname-db.internal

I have closed the fly proxy and connected again, but the same error occurs.
If I turn on fly proxy and use localhost:15432 to connect,
Can connect to db, but not production db.

I’m a beginner and would appreciate some guidance.

(this is google translate)

dornby · August 31, 2022, 12:27pm

@jedi I too use TablePlus and all worked fine:

Keep your fly proxy open
Use the following in TablePlus:

host: localhost
user: postgres (well, your db user name)
password: your db password
port: 15432
database: the same name as we used in our last step in my previous message: foobar in the example

jedi · August 31, 2022, 12:50pm

@dornby thanks for your reply

I have enabled “fly proxy”.
Follow your instructions to connect,
It can be successfully connected.

I tried to change the data in the db in TablePlus and save it,
Then “puts db data” on the Internet,
But it shows the old data (maintain the data of the original heroku db)

Obviously the data I changed in TablePlus was not updated synchronously,
Or what action is required to upload the database?

please help thanks

(this is google translate)

jedi · August 31, 2022, 3:45pm

@dornby

OMG
my DATABASE_URL=$(heroku config:get DATABASE_URL)

I’m such an idiot…
It is normal to change DATABASE_URL to fly’s db…

Thank u

dornby · September 1, 2022, 3:36pm

@jedi you’re not an idiot
Well done on solving this one, I wouldn’t have managed to help

lortza · October 15, 2022, 8:32pm

Also for anyone else following along here, you’ll need to add your heroku app name to that command DATABASE_URL=$(heroku config:get DATABASE_URL -a your-heroku-app-name).

Topic		Replies	Views
Help migrating postgres from heroku rails app Questions / Help postgres , rails	16	1495	November 16, 2022
PG database migration from Heroku not working Questions / Help postgres	21	2183	January 20, 2023
Heroku to fly postgres migration guide not working Questions / Help	12	709	August 31, 2022
PostgreSQL database import	3	359	May 21, 2023
Migrating from Heroku Postgres to Fly Postgres: A Complete Guide Show & Tell docs , grafana , postgres	1	2648	October 18, 2022

Transferring Postgres from Heroku

Related topics