Fly v2 upgrade failing.

Good morning,

We have a number of applications running on Nomad and are trying to transition them to Machines.
So far we’ve successfully migrated most of them but 3.

The failing ones are all *-db - the Apps migrated successfully.
I’m taking over the project and have been around for a month so it might be that I’m missing something fairly basic… Some guidance would be really awesome.

The *-db nodes didn’t have TOML config - I used fly config save ... to get them and then be able to use theses config files when issuing fly migrate-to-v2 -c ... commands.

Two of them (Let’s call them prod-db and saas-staging-db) logs:

Error: can't get role for REDACTED:5493:2: Get "http://REDACTED:5493:2:5500/commands/admin/role": connect tcp [REDACTED:5493:2]:5500: operation timed out

Sometimes, the timeout doesn’t occur but instead we get an immediate Error: 404: 404 page not found error for prod-db and this for saas-staging-db:

This migration process will do the following, in order:
 * Update your postgres app to the latest supported image version
 * Create volumes for new PG replicas to use
 * Lock your application, preventing changes during the migration
 * Create machines, copying the configuration of each existing VM
   * Create 2 "app" machines
 * Wait for at least one new in-region PG replica to become synced
 * Remove legacy VMs after health checks pass for the new machines
   * Remove 2 allocs
 * Set the application platform version to "machines"
 * Unlock your application
? Would you like to continue? Yes
==> Migrating saas-staging-db to the V2 platform
>  Upgrading postgres image
>  Setting postgres primary to readonly
failed while migrating: 404: 404 page not found

? Would you like to enter interactive troubleshooting mode? If not, the migration will be rolled back. Yes
failed while troubleshooting: 404: 404 page not found

Error: 404: 404 page not found

With a LOG_LEVEL=debug trace, we see that the 404 happens around this command:

DEBUG role for REDACTED: leader
DEBUG flypg will connect to: http://REDACTED:5500
DEBUG --> POST http://REDACTED:5500/commands/admin/readonly/enable
DEBUG <-- 404 http://REDACTED:5500/commands/admin/readonly/enable (51.11ms)
DEBUG 404 page not found

How can I troubleshoot this?

Bump… Anyone please?

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.