fly migrate-to-v2 - postgres edition

Fresh Produce
41

It’s not 100% healthy, one of the two VMs is not healthy, more errors were shared in the other topic referenced above.

I also referenced the issues we are having migrating to V2 in reference to Thai topic.

42 
$ fly status

Instances
ID      	PROCESS	VERSION	REGION	DESIRED	STATUS           	HEALTH CHECKS     	RESTARTS	CREATED
4de836d7	app    	15     	ord   	run    	running (leader) 	3 total, 3 passing	0       	2023-05-02T21:48:38Z
19ba1e0e	app    	15     	ord   	run    	running (replica)	3 total, 3 passing	1       	2023-05-02T21:22:00Z
43

@shaun This is new then as of today, one of the two has been failing for some time now, again you can see the errors that were happening all week in the other thread I referenced above.

End of the day we want to be able to move these to V2 with fly flex and have no idea how to do so.

44

Unfortunately, you’re only option to move onto Flex right now is to spin up a new application that’s running Flex and import your data via fly pg import tool.

45

Is there docs on this? I have never seen this tool before.

Edit: Found it, we were able to migrate by creating a new PG app and running the pg import tool.

1 Like
46

I run it again today and it worked!

1 Like
47

I encountered this error when trying to migrate:

 can't get role for xxxx:xxxx:xxxx Get "http://xxxx:xxxx:xxxx /commands/admin/role": connect tcp [xxxx:xxxx:xxxx ]:5500: operation timed out

The reason this was happening, in my case, is that another dev was proxying the database at the time.

So, if you are migrating, make sure to tell everyone else with access to disconnect.

After everyone has disconnected, just do this before migrating:

fly agent restart
48

I had an issue where the migrate-to-v2 command on the postgres cluster timed out. It left our database in read-only state. I wrote a postmortem on it here.

In short, if your DB gets stuck in a read only mode as a result of a timeout on a migration you will want to set it back to writable using this command.

# connect to postgres psql
fly postgres connect -a <postgres_fly_app_name>
# This will show false for the postgres DB
SHOW default_transaction_read_only; 
# connect to the production db
\connect <production_db_name>
# This will now show the root cause of issue
SHOW default_transaction_read_only;
# the following command will only work for the current connection
SET default_transaction_read_only TO off;
# connect back to postgres
\connect postgres
# fix the real problem
alter database <production_db_name> set default_transaction_read_only=off;

I hope this helps someone with a failed postgres migration stuck in a read only state.

49

Hi,

I tried migrating two Postgres instances connected to two apps to v2. The first one worked fine, but the second one failed.

After the migration, I first got an error about “No leader found” and after trying to restart the app and scale it up/down I’m now getting “Error: no 6pn ips found for…”

Any ideas on what I should do next to get back up and running?

I also tried to create a new volume from some of the snapshots but that fails with

Error: failed to create volume: failed to create volume: EOF

1 Like
50

Hello, I have the exact same issue but restarting the agent has not helped. I have been trying to migrate for a few weeks now, it won’t work.

I get:

$ fly migrate-to-v2
Error: 404: 404 page not found

This is for a postgre app: nara-db

51

Mine was auto-migrated, I think.

It no longer works and I am getting the same errors as you.

52

For what it’s worth, I couldn’t get i to work and created a new DB instance and imported a backup.