We have a number of applications running on Nomad and are trying to transition them to Machines.
So far we’ve successfully migrated most of them but 3.
The failing ones are all
*-db - the Apps migrated successfully.
I’m taking over the project and have been around for a month so it might be that I’m missing something fairly basic… Some guidance would be really awesome.
*-db nodes didn’t have TOML config - I used
fly config save ... to get them and then be able to use theses config files when issuing
fly migrate-to-v2 -c ... commands.
Two of them (Let’s call them
Error: can't get role for REDACTED:5493:2: Get "http://REDACTED:5493:2:5500/commands/admin/role": connect tcp [REDACTED:5493:2]:5500: operation timed out
Sometimes, the timeout doesn’t occur but instead we get an immediate
Error: 404: 404 page not found error for
prod-db and this for
This migration process will do the following, in order: * Update your postgres app to the latest supported image version * Create volumes for new PG replicas to use * Lock your application, preventing changes during the migration * Create machines, copying the configuration of each existing VM * Create 2 "app" machines * Wait for at least one new in-region PG replica to become synced * Remove legacy VMs after health checks pass for the new machines * Remove 2 allocs * Set the application platform version to "machines" * Unlock your application ? Would you like to continue? Yes ==> Migrating saas-staging-db to the V2 platform > Upgrading postgres image > Setting postgres primary to readonly failed while migrating: 404: 404 page not found ? Would you like to enter interactive troubleshooting mode? If not, the migration will be rolled back. Yes failed while troubleshooting: 404: 404 page not found Error: 404: 404 page not found
LOG_LEVEL=debug trace, we see that the 404 happens around this command:
DEBUG role for REDACTED: leader DEBUG flypg will connect to: http://REDACTED:5500 DEBUG --> POST http://REDACTED:5500/commands/admin/readonly/enable DEBUG <-- 404 http://REDACTED:5500/commands/admin/readonly/enable (51.11ms) DEBUG 404 page not found
How can I troubleshoot this?