Can't connect to db anymore after migrating between orgs

Problem looks identical to PG Connection timed out after moving app between orgs. Connecting to db from command line produces a message that seems to indicate it’s trying to connect to a stale IP:

$ flyctl postgres connect -a frontendeval-api-db
Connecting to fdaa:1:1094:a7b:a1:80d:2bbb:2...

This IP doesn’t appear in any of the IPs from fly ips (which was empty until I allocated a private and public v6 IP, neither of which seem to be used by fly pg connect)

Any way for me to recover this? Unfortunately I can’t just nuke the db as in the linked thread. I need to at least get a dump of it before I recreate it but there doesn’t seem to be any way for me to access it at all.

This looked pretty unrecoverable so I was able to find a backup and nuke my db, so we can consider this resolved for now.

I just wanted to add that I was also unable to use my old builder that migrated to the new org. It looks like org migration is pretty broken right now, despite it being really easy to do.

I’m good to go now, but happy to provide any information that might help you debug.

Sorry for the delay.

This command might’ve shown the correct IP:

fly ips private

Private, per-instance, IPs are allocated automatically for each instance and are unknown to our API. The only way to get them is via DNS (like resolving frontendeval-api-db.internal).

I think this is because we don’t renew the private IPs before a restart the moved app. Each network ID is encoded in the private IPs and that’s part of how we enforce limited access. This is possibly even worse for machines because IPs are fixed by machines, even across restarts. I’m not sure we have logic to fix that if the network ID changes between restarts.

Both remote builders and postgres clusters are using machines. Easiest fix for a remote builder is to destroy and re-create it.

This will likely have to wait a little bit until we have more people available to debug. I’m only making up theories at this point :slight_smile:

Very sorry about the inconvenience, I’m glad you were able to get out from under these issues.

Thanks for the response @jerome, I actually tried that command as well before provisioning a new v6 IP, and it showed nothing. There were neither private nor public IPs. I may take a crack at repro-ing the problem with a fresh install if it helps you debug. Happy to help however I can.

This definitely threw a wrench into things, but I was very happy with the experience otherwise, and overjoyed to get away from Heroku.

1 Like

Completely unrelated and apologies for polluting the thread: this page has an error on it

fly certs show requires a hostname, so it’d be flyctl certs show -a custom-quartz Wasn’t sure where to report this but since I may have your attention I thought it’d be worth mentioning!

1 Like