$ flyctl postgres connect -a frontendeval-api-db
Connecting to fdaa:1:1094:a7b:a1:80d:2bbb:2...
This IP doesn’t appear in any of the IPs from fly ips (which was empty until I allocated a private and public v6 IP, neither of which seem to be used by fly pg connect)
Any way for me to recover this? Unfortunately I can’t just nuke the db as in the linked thread. I need to at least get a dump of it before I recreate it but there doesn’t seem to be any way for me to access it at all.
This looked pretty unrecoverable so I was able to find a backup and nuke my db, so we can consider this resolved for now.
I just wanted to add that I was also unable to use my old builder that migrated to the new org. It looks like org migration is pretty broken right now, despite it being really easy to do.
I’m good to go now, but happy to provide any information that might help you debug.
Private, per-instance, IPs are allocated automatically for each instance and are unknown to our API. The only way to get them is via DNS (like resolving frontendeval-api-db.internal).
I think this is because we don’t renew the private IPs before a restart the moved app. Each network ID is encoded in the private IPs and that’s part of how we enforce limited access. This is possibly even worse for machines because IPs are fixed by machines, even across restarts. I’m not sure we have logic to fix that if the network ID changes between restarts.
Both remote builders and postgres clusters are using machines. Easiest fix for a remote builder is to destroy and re-create it.
This will likely have to wait a little bit until we have more people available to debug. I’m only making up theories at this point
Very sorry about the inconvenience, I’m glad you were able to get out from under these issues.
Thanks for the response @jerome, I actually tried that command as well before provisioning a new v6 IP, and it showed nothing. There were neither private nor public IPs. I may take a crack at repro-ing the problem with a fresh install if it helps you debug. Happy to help however I can.
This definitely threw a wrench into things, but I was very happy with the experience otherwise, and overjoyed to get away from Heroku.
fly certs show requires a hostname, so it’d be flyctl certs show -a custom-quartz example.com. Wasn’t sure where to report this but since I may have your attention I thought it’d be worth mentioning!
