Good morning,
We have a number of applications running on Nomad and are trying to transition them to Machines.
So far we’ve successfully migrated most of them but 3.
The failing ones are all *-db - the Apps migrated successfully.
I’m taking over the project and have been around for a month so it might be that I’m missing something fairly basic… Some guidance would be really awesome.
The *-db nodes didn’t have TOML config - I used fly config save ... to get them and then be able to use theses config files when issuing fly migrate-to-v2 -c ... commands.
Two of them (Let’s call them prod-db and saas-staging-db) logs:
Error: can't get role for REDACTED:5493:2: Get "http://REDACTED:5493:2:5500/commands/admin/role": connect tcp [REDACTED:5493:2]:5500: operation timed out
Sometimes, the timeout doesn’t occur but instead we get an immediate Error: 404: 404 page not found error for prod-db and this for saas-staging-db:
This migration process will do the following, in order:
* Update your postgres app to the latest supported image version
* Create volumes for new PG replicas to use
* Lock your application, preventing changes during the migration
* Create machines, copying the configuration of each existing VM
* Create 2 "app" machines
* Wait for at least one new in-region PG replica to become synced
* Remove legacy VMs after health checks pass for the new machines
* Remove 2 allocs
* Set the application platform version to "machines"
* Unlock your application
? Would you like to continue? Yes
==> Migrating saas-staging-db to the V2 platform
> Upgrading postgres image
> Setting postgres primary to readonly
failed while migrating: 404: 404 page not found
? Would you like to enter interactive troubleshooting mode? If not, the migration will be rolled back. Yes
failed while troubleshooting: 404: 404 page not found
Error: 404: 404 page not found
With a LOG_LEVEL=debug trace, we see that the 404 happens around this command:
DEBUG role for REDACTED: leader
DEBUG flypg will connect to: http://REDACTED:5500
DEBUG --> POST http://REDACTED:5500/commands/admin/readonly/enable
DEBUG <-- 404 http://REDACTED:5500/commands/admin/readonly/enable (51.11ms)
DEBUG 404 page not found
How can I troubleshoot this?