Could you please rerun that with LOG_LEVEL=debug before the command? I’d like to figure out what’s going wrong here - this is definitely a bug on our side. Thanks!
This is very verbose, sorry.
This migration process will do the following, in order:
* Update your postgres app to the latest supported image version
* Create volumes for new PG replicas to use
* Lock your application, preventing changes during the migration
* Create machines, copying the configuration of each existing VM
* Create 2 "app" machines
* Wait for at least one new in-region PG replica to become synced
* Remove legacy VMs after health checks pass for the new machines
* Remove 2 allocs
* Set the application platform version to "machines"
* Unlock your application
? Would you like to continue? Yes
==> Migrating diciprd-db to the V2 platform
> Upgrading postgres image
DEBUG --> POST https://api.fly.io/graphql
DEBUG {
"query": "query($appName: String!) { app(name: $appName) { imageVersionTrackingEnabled imageUpgradeAvailable imageDetails { registry repository tag digest version } latestImageDetails { registry repository tag digest version } } }",
"variables": {
"appName": "diciprd-db"
}
}
DEBUG {}
DEBUG <-- 200 https://api.fly.io/graphql (178.78ms)
DEBUG {
"data": {
"app": {
"imageVersionTrackingEnabled": true,
"imageUpgradeAvailable": false,
"imageDetails": {
"registry": "registry-1.docker.io",
"repository": "flyio/postgres",
"tag": "14.6",
"digest": "sha256:3c25db96357a78e827ca7dbbf4963089bdfcd48fb05a40cd82cfff5c61fb7710",
"version": "v0.0.41"
},
"latestImageDetails": {
"registry": "registry-1.docker.io",
"repository": "flyio/postgres",
"tag": "14.6",
"digest": "sha256:3c25db96357a78e827ca7dbbf4963089bdfcd48fb05a40cd82cfff5c61fb7710",
"version": "v0.0.41"
}
}
}
}
> Setting postgres primary to readonly
DEBUG --> POST https://api.fly.io/graphql
DEBUG {
"query": "mutation($input: ValidateWireGuardPeersInput!) { validateWireGuardPeers(input: $input) { invalidPeerIps } }",
"variables": {
"input": {
"peerIps": [
"fdaa:0:4d65:a7b:1596:0:a:202"
]
}
}
}
DEBUG {}
DEBUG <-- 200 https://api.fly.io/graphql (195.87ms)
DEBUG {
"data": {
"validateWireGuardPeers": {
"invalidPeerIps": []
}
}
}
DEBUG --> POST https://api.fly.io/graphql
DEBUG {
"query": "\n# @genqlient\nquery AgentGetInstances ($appName: String!) {\n\tapp(name: $appName) {\n\t\torganization {\n\t\t\tslug\n\t\t}\n\t\tid\n\t\tname\n\t\tallocations(showCompleted: false) {\n\t\t\tid\n\t\t\tregion\n\t\t\tprivateIP\n\t\t}\n\t\tmachines {\n\t\t\tnodes {\n\t\t\t\tstate\n\t\t\t\tid\n\t\t\t\tregion\n\t\t\t\tips {\n\t\t\t\t\tnodes {\n\t\t\t\t\t\tkind\n\t\t\t\t\t\tfamily\n\t\t\t\t\t\tip\n\t\t\t\t\t}\n\t\t\t\t}\n\t\t\t}\n\t\t}\n\t}\n}\n",
"variables": {
"appName": "diciprd-db"
},
"operationName": "AgentGetInstances"
}
DEBUG {0xc0010feff0}
DEBUG <-- 200 https://api.fly.io/graphql (151.64ms)
DEBUG {
"data": {
"app": {
"organization": {
"slug": "personal"
},
"id": "diciprd-db",
"name": "diciprd-db",
"allocations": [
{
"id": "be3238c8-1e7a-6f91-21de-4bfa6c75c8e3",
"region": "mad",
"privateIP": "fdaa:0:4d65:a7b:2658:1:af82:2"
},
{
"id": "cb3965d1-89f6-6b95-091e-7065ea63da74",
"region": "mad",
"privateIP": "fdaa:0:4d65:a7b:2656:1:af83:2"
}
],
"machines": {
"nodes": []
}
}
}
}
DEBUG gqlGetInstances() result: &{[mad.diciprd-db.internal (fdaa:0:4d65:a7b:2658:1:af82:2) mad.diciprd-db.internal (fdaa:0:4d65:a7b:2656:1:af83:2)] [fdaa:0:4d65:a7b:2658:1:af82:2 fdaa:0:4d65:a7b:2656:1:af83:2]}
DEBUG gqlErr: <nil> agentErr: <nil>
DEBUG flypg will connect to: http://fdaa:0:4d65:a7b:2658:1:af82:2:5500
DEBUG --> GET http://fdaa:0:4d65:a7b:2658:1:af82:2:5500/commands/admin/role
DEBUG <-- 200 http://fdaa:0:4d65:a7b:2658:1:af82:2:5500/commands/admin/role (44.11ms)
DEBUG {
"result": "replica"
}
DEBUG flypg will connect to: http://fdaa:0:4d65:a7b:2656:1:af83:2:5500
DEBUG --> GET http://fdaa:0:4d65:a7b:2656:1:af83:2:5500/commands/admin/role
DEBUG <-- 200 http://fdaa:0:4d65:a7b:2656:1:af83:2:5500/commands/admin/role (47.93ms)
DEBUG {
"result": "leader"
}
DEBUG flypg will connect to: http://fdaa:0:4d65:a7b:2656:1:af83:2:5500
DEBUG --> POST http://fdaa:0:4d65:a7b:2656:1:af83:2:5500/commands/admin/readonly/enable
DEBUG <-- 200 http://fdaa:0:4d65:a7b:2656:1:af83:2:5500/commands/admin/readonly/enable (54.71ms)
DEBUG {
"result": true
}
DEBUG flypg will connect to: http://fdaa:0:4d65:a7b:2658:1:af82:2:5500
DEBUG --> POST http://fdaa:0:4d65:a7b:2658:1:af82:2:5500/commands/admin/haproxy/restart
DEBUG <-- 200 http://fdaa:0:4d65:a7b:2658:1:af82:2:5500/commands/admin/haproxy/restart (50.42ms)
DEBUG {
"result": true
}
> Creating new postgres volumes
Creatings 2 new volume(s) in 'mad'
DEBUG --> POST https://api.machines.dev/v1/apps/diciprd-db/volumes
DEBUG {
"name": "pg_data_machines",
"region": "mad",
"size_gb": 0,
"encrypted": false,
"require_unique_zone": null,
"machines_only": true,
"snapshot_id": null,
"source_volume_id": null,
"compute": null
}
DEBUG {0xc000c17b60}
DEBUG <-- 400 https://api.machines.dev/v1/apps/diciprd-db/volumes (7.63ms)
DEBUG {
"error": "size_gb must be between 1GB and 500GB"
}
failed while migrating: failed to create volume: size_gb must be between 1GB and 500GB
I’m curious if it has something to do with the fact that, on screen, it’s showing used 0 MB used… (it’s definitely not zero, although it’s also a small db).
Is there anything before the big prompt? For some reason, it’s trying to create 0gb volumes, and the existing volumes are queried at the very start of the process before it’s prompted you for anything.
In the meantime, I’m going to look at updating migrate-to-v2 to use our new volumes API endpoints, as it’s possibly the last holdout in flyctl. That shouldn’t change behavior, but it’s possible that could unblock you.
There is, I tried to remove thinking it was not important. Here it goes, from the begining to the “prompt” of “are you sure you want to…?”
ubuntu@ip-172-31-35-191:~$ LOG_LEVEL=debug fly migrate-to-v2
DEBUG Loaded flyctl config from/home/ubuntu/.fly/config.yml
DEBUG determined hostname: "ip-172-31-35-191"
DEBUG determined working directory: "/home/ubuntu"
DEBUG determined user home directory: "/home/ubuntu"
DEBUG determined config directory: "/home/ubuntu/.fly"
DEBUG ensured config directory exists.
DEBUG ensured config directory perms.
DEBUG cache loaded.
DEBUG config initialized.
DEBUG client initialized.
DEBUG app config loaded from /home/ubuntu/fly.toml
DEBUG --> POST https://api.fly.io/graphql
DEBUG {
"query": "query ($appName: String!) { appbasic:app(name: $appName) { id name platformVersion organization { id slug paidPlan } } }",
"variables": {
"appName": "diciprd-db"
}
}
DEBUG {}
DEBUG <-- 200 https://api.fly.io/graphql (137.71ms)
DEBUG {
"data": {
"appbasic": {
"id": "diciprd-db",
"name": "diciprd-db",
"platformVersion": "nomad",
"organization": {
"id": "kRXe1L9oZlXV2uZ6ZyMOpeYMOxfnaM",
"slug": "personal",
"paidPlan": false
}
}
}
}
DEBUG --> POST https://api.fly.io/graphql
DEBUG {
"query": "query ($appName: String!) { appcompact:app(name: $appName) { id name hostname deployed status appUrl platformVersion organization { id slug paidPlan } postgresAppRole: role { name } imageDetails { repository version } } }",
"variables": {
"appName": "diciprd-db"
}
}
DEBUG Config has metrics token
DEBUG {}
DEBUG Starting task manager
DEBUG --> POST https://api.fly.io/graphql
DEBUG {
"query": "query { platform { requestRegion regions { name code latitude longitude gatewayAvailable requiresPaidPlan } } }",
"variables": null
}
DEBUG {}
DEBUG started querying for new release
DEBUG <-- 200 https://api.fly.io/graphql (193.77ms)
DEBUG {
"data": {
"platform": {
"requestRegion": "cdg",
"regions": [
{
"name": "Amsterdam, Netherlands",
"code": "ams",
"latitude": 52.374342,
"longitude": 4.895439,
"gatewayAvailable": true,
"requiresPaidPlan": false
},
{
"name": "Stockholm, Sweden",
"code": "arn",
"latitude": 59.6512,
"longitude": 17.9178,
"gatewayAvailable": false,
"requiresPaidPlan": false
},
{
"name": "Atlanta, Georgia (US)",
"code": "atl",
"latitude": 33.6407,
"longitude": -84.4277,
"gatewayAvailable": false,
"requiresPaidPlan": false
},
{
"name": "Bogotá, Colombia",
"code": "bog",
"latitude": 4.70159,
"longitude": -74.1469,
"gatewayAvailable": false,
"requiresPaidPlan": false
},
{
"name": "Mumbai, India",
"code": "bom",
"latitude": 19.097403,
"longitude": 72.874245,
"gatewayAvailable": false,
"requiresPaidPlan": true
},
{
"name": "Boston, Massachusetts (US)",
"code": "bos",
"latitude": 42.366978,
"longitude": -71.022362,
"gatewayAvailable": false,
"requiresPaidPlan": false
},
{
"name": "Paris, France",
"code": "cdg",
"latitude": 48.860875,
"longitude": 2.353477,
"gatewayAvailable": true,
"requiresPaidPlan": false
},
{
"name": "Denver, Colorado (US)",
"code": "den",
"latitude": 39.7392,
"longitude": -104.9847,
"gatewayAvailable": false,
"requiresPaidPlan": false
},
{
"name": "Dallas, Texas (US)",
"code": "dfw",
"latitude": 32.778287,
"longitude": -96.7984,
"gatewayAvailable": true,
"requiresPaidPlan": false
},
{
"name": "Secaucus, NJ (US)",
"code": "ewr",
"latitude": 40.789543,
"longitude": -74.056534,
"gatewayAvailable": false,
"requiresPaidPlan": false
},
{
"name": "Ezeiza, Argentina",
"code": "eze",
"latitude": -34.8222,
"longitude": -58.5358,
"gatewayAvailable": false,
"requiresPaidPlan": false
},
{
"name": "Frankfurt, Germany",
"code": "fra",
"latitude": 50.1167,
"longitude": 8.6833,
"gatewayAvailable": true,
"requiresPaidPlan": true
},
{
"name": "Guadalajara, Mexico",
"code": "gdl",
"latitude": 20.5217,
"longitude": -103.3109,
"gatewayAvailable": false,
"requiresPaidPlan": false
},
{
"name": "Rio de Janeiro, Brazil",
"code": "gig",
"latitude": -22.8099,
"longitude": -43.2505,
"gatewayAvailable": false,
"requiresPaidPlan": false
},
{
"name": "Sao Paulo, Brazil",
"code": "gru",
"latitude": -23.549664,
"longitude": -46.654351,
"gatewayAvailable": false,
"requiresPaidPlan": false
},
{
"name": "Hong Kong, Hong Kong",
"code": "hkg",
"latitude": 22.250971,
"longitude": 114.203224,
"gatewayAvailable": true,
"requiresPaidPlan": false
},
{
"name": "Ashburn, Virginia (US)",
"code": "iad",
"latitude": 39.02214,
"longitude": -77.462556,
"gatewayAvailable": true,
"requiresPaidPlan": false
},
{
"name": "Johannesburg, South Africa",
"code": "jnb",
"latitude": -26.13629,
"longitude": 28.20298,
"gatewayAvailable": false,
"requiresPaidPlan": false
},
{
"name": "Los Angeles, California (US)",
"code": "lax",
"latitude": 33.9416,
"longitude": -118.4085,
"gatewayAvailable": true,
"requiresPaidPlan": false
},
{
"name": "London, United Kingdom",
"code": "lhr",
"latitude": 51.516434,
"longitude": -0.125656,
"gatewayAvailable": true,
"requiresPaidPlan": false
},
{
"name": "Chennai (Madras), India",
"code": "maa",
"latitude": 13.064429,
"longitude": 80.253067,
"gatewayAvailable": true,
"requiresPaidPlan": true
},
{
"name": "Madrid, Spain",
"code": "mad",
"latitude": 40.4381,
"longitude": -3.82,
"gatewayAvailable": false,
"requiresPaidPlan": false
},
{
"name": "Miami, Florida (US)",
"code": "mia",
"latitude": 25.7877,
"longitude": -80.2241,
"gatewayAvailable": false,
"requiresPaidPlan": false
},
{
"name": "Tokyo, Japan",
"code": "nrt",
"latitude": 35.621608,
"longitude": 139.741851,
"gatewayAvailable": true,
"requiresPaidPlan": false
},
{
"name": "Chicago, Illinois (US)",
"code": "ord",
"latitude": 41.891544,
"longitude": -87.630386,
"gatewayAvailable": true,
"requiresPaidPlan": false
},
{
"name": "Bucharest, Romania",
"code": "otp",
"latitude": 44.4325,
"longitude": 26.1039,
"gatewayAvailable": false,
"requiresPaidPlan": false
},
{
"name": "Phoenix, Arizona (US)",
"code": "phx",
"latitude": 33.416084,
"longitude": -112.009482,
"gatewayAvailable": false,
"requiresPaidPlan": false
},
{
"name": "Querétaro, Mexico",
"code": "qro",
"latitude": 20.62,
"longitude": -100.1863,
"gatewayAvailable": false,
"requiresPaidPlan": false
},
{
"name": "Santiago, Chile",
"code": "scl",
"latitude": -33.36572,
"longitude": -70.64292,
"gatewayAvailable": true,
"requiresPaidPlan": false
},
{
"name": "Seattle, Washington (US)",
"code": "sea",
"latitude": 47.6097,
"longitude": -122.3331,
"gatewayAvailable": true,
"requiresPaidPlan": false
},
{
"name": "Singapore, Singapore",
"code": "sin",
"latitude": 1.3,
"longitude": 103.8,
"gatewayAvailable": true,
"requiresPaidPlan": false
},
{
"name": "San Jose, California (US)",
"code": "sjc",
"latitude": 37.351601,
"longitude": -121.896744,
"gatewayAvailable": true,
"requiresPaidPlan": false
},
{
"name": "Sydney, Australia",
"code": "syd",
"latitude": -33.866033,
"longitude": 151.20693,
"gatewayAvailable": true,
"requiresPaidPlan": false
},
{
"name": "Warsaw, Poland",
"code": "waw",
"latitude": 52.1657,
"longitude": 20.9671,
"gatewayAvailable": false,
"requiresPaidPlan": false
},
{
"name": "Montreal, Canada",
"code": "yul",
"latitude": 45.48647,
"longitude": -73.75549,
"gatewayAvailable": false,
"requiresPaidPlan": false
},
{
"name": "Toronto, Canada",
"code": "yyz",
"latitude": 43.644632,
"longitude": -79.384228,
"gatewayAvailable": true,
"requiresPaidPlan": false
}
]
}
}
}
DEBUG <-- 200 https://api.fly.io/graphql (269.06ms)
DEBUG {
"data": {
"appcompact": {
"id": "diciprd-db",
"name": "diciprd-db",
"hostname": "diciprd-db.fly.dev",
"deployed": true,
"appUrl": null,
"platformVersion": "nomad",
"organization": {
"id": "kRXe1L9oZlXV2uZ6ZyMOpeYMOxfnaM",
"slug": "personal",
"paidPlan": false
},
"postgresAppRole": {
"name": "postgres_cluster"
},
"imageDetails": {
"repository": "flyio/postgres",
"version": "v0.0.41"
},
"status": "running"
}
}
}
DEBUG --> POST https://api.fly.io/graphql
DEBUG {
"query": "mutation($input: ValidateWireGuardPeersInput!) { validateWireGuardPeers(input: $input) { invalidPeerIps } }",
"variables": {
"input": {
"peerIps": [
"fdaa:0:4d65:a7b:1596:0:a:202"
]
}
}
}
DEBUG {}
DEBUG querying for release resulted to v0.1.81
DEBUG <-- 200 https://api.fly.io/graphql (129ms)
DEBUG {
"data": {
"validateWireGuardPeers": {
"invalidPeerIps": []
}
}
}
DEBUG --> POST https://api.fly.io/graphql
DEBUG {
"query": "query ($appName: String!) { appcompact:app(name: $appName) { id name hostname deployed status appUrl platformVersion organization { id slug paidPlan } postgresAppRole: role { name } imageDetails { repository version } } }",
"variables": {
"appName": "diciprd-db"
}
}
DEBUG {}
DEBUG <-- 200 https://api.fly.io/graphql (177.09ms)
DEBUG {
"data": {
"appcompact": {
"id": "diciprd-db",
"name": "diciprd-db",
"hostname": "diciprd-db.fly.dev",
"deployed": true,
"appUrl": null,
"platformVersion": "nomad",
"organization": {
"id": "kRXe1L9oZlXV2uZ6ZyMOpeYMOxfnaM",
"slug": "personal",
"paidPlan": false
},
"postgresAppRole": {
"name": "postgres_cluster"
},
"imageDetails": {
"repository": "flyio/postgres",
"version": "v0.0.41"
},
"status": "running"
}
}
}
DEBUG --> POST https://api.fly.io/graphql
DEBUG {
"query": "query ($appName: String!) { app(name: $appName) { id name hostname deployed status version appUrl platformVersion currentRelease { evaluationId status inProgress version } config { definition } organization { id slug paidPlan } services { description protocol internalPort ports { port handlers } } ipAddresses { nodes { id address type createdAt } } imageDetails { registry repository tag digest version } volumes { nodes { id sizeGb name region attached_alloc_id: attachedAllocationId } } machines{ nodes { id name config state region createdAt app { name } ips { nodes { family kind ip maskSize } } host { id } } } postgresAppRole: role { name } limitedAccessTokens { nodes { id name expiresAt } } } }",
"variables": {
"appName": "diciprd-db"
}
}
DEBUG {}
DEBUG <-- 200 https://api.fly.io/graphql (227.01ms)
DEBUG {
"data": {
"app": {
"id": "diciprd-db",
"name": "diciprd-db",
"hostname": "diciprd-db.fly.dev",
"deployed": true,
"version": 1,
"appUrl": null,
"platformVersion": "nomad",
"currentRelease": {
"evaluationId": null,
"status": "succeeded",
"inProgress": false,
"version": 1
},
"config": {
"definition": {
"kill_timeout": 300,
"kill_signal": "SIGTERM",
"processes": [],
"mounts": [
{
"processes": [],
"source": "pg_data",
"destination": "/data",
"encrypted": false
}
],
"experimental": {
"entrypoint": [],
"cmd": [],
"exec": [],
"auto_rollback": false,
"enable_consul": true,
"private_network": true
},
"services": [],
"checks": {
"role": {
"interval": "15s",
"timeout": "10s",
"grace_period": "30s",
"restart_limit": 0,
"port": 5500,
"method": "get",
"protocol": "http",
"path": "/flycheck/role",
"tls_skip_verify": false,
"headers": [],
"type": "http"
},
"pg": {
"interval": "15s",
"timeout": "10s",
"grace_period": "30s",
"restart_limit": 0,
"port": 5500,
"method": "get",
"protocol": "http",
"path": "/flycheck/pg",
"tls_skip_verify": false,
"headers": [],
"type": "http"
},
"vm": {
"interval": "1m",
"timeout": "10s",
"grace_period": "1s",
"restart_limit": 0,
"port": 5500,
"method": "get",
"protocol": "http",
"path": "/flycheck/vm",
"tls_skip_verify": false,
"headers": [],
"type": "http"
}
},
"env": {
"PRIMARY_REGION": "mad"
},
"metrics": {
"port": 9187,
"path": "/metrics"
}
}
},
"organization": {
"id": "kRXe1L9oZlXV2uZ6ZyMOpeYMOxfnaM",
"slug": "personal",
"paidPlan": false
},
"services": [],
"imageDetails": {
"registry": "registry-1.docker.io",
"repository": "flyio/postgres",
"tag": "14.6",
"digest": "sha256:3c25db96357a78e827ca7dbbf4963089bdfcd48fb05a40cd82cfff5c61fb7710",
"version": "v0.0.41"
},
"postgresAppRole": {
"name": "postgres_cluster"
},
"ipAddresses": {
"nodes": []
},
"machines": {
"nodes": []
},
"limitedAccessTokens": {
"nodes": []
},
"status": "running",
"volumes": {
"nodes": [
{
"id": "vol_gez1nvxlozzvmxl7",
"sizeGb": 50,
"name": "pg_data",
"region": "mad",
"attached_alloc_id": "be3238c8-1e7a-6f91-21de-4bfa6c75c8e3"
},
{
"id": "vol_8zmjnv8150yrywgx",
"sizeGb": 50,
"name": "pg_data",
"region": "mad",
"attached_alloc_id": "cb3965d1-89f6-6b95-091e-7065ea63da74"
}
]
}
}
}
}
DEBUG --> POST https://api.fly.io/graphql
DEBUG {
"query": "query ($appName: String!) { appcompact:app(name: $appName) { id name hostname deployed status appUrl platformVersion organization { id slug paidPlan } postgresAppRole: role { name } imageDetails { repository version } } }",
"variables": {
"appName": "diciprd-db"
}
}
DEBUG {}
DEBUG <-- 200 https://api.fly.io/graphql (181.42ms)
DEBUG {
"data": {
"appcompact": {
"id": "diciprd-db",
"name": "diciprd-db",
"hostname": "diciprd-db.fly.dev",
"deployed": true,
"appUrl": null,
"platformVersion": "nomad",
"organization": {
"id": "kRXe1L9oZlXV2uZ6ZyMOpeYMOxfnaM",
"slug": "personal",
"paidPlan": false
},
"postgresAppRole": {
"name": "postgres_cluster"
},
"imageDetails": {
"repository": "flyio/postgres",
"version": "v0.0.41"
},
"status": "running"
}
}
}
DEBUG --> POST https://api.fly.io/graphql
DEBUG {
"query": "query($appName: String!) { app(name: $appName) { config { definition } } }",
"variables": {
"appName": "diciprd-db"
}
}
DEBUG {}
DEBUG <-- 200 https://api.fly.io/graphql (116.91ms)
DEBUG {
"data": {
"app": {
"config": {
"definition": {
"kill_timeout": 300,
"kill_signal": "SIGTERM",
"processes": [],
"mounts": [
{
"processes": [],
"source": "pg_data",
"destination": "/data",
"encrypted": false
}
],
"experimental": {
"entrypoint": [],
"cmd": [],
"exec": [],
"auto_rollback": false,
"enable_consul": true,
"private_network": true
},
"services": [],
"checks": {
"role": {
"interval": "15s",
"timeout": "10s",
"grace_period": "30s",
"restart_limit": 0,
"port": 5500,
"method": "get",
"protocol": "http",
"path": "/flycheck/role",
"tls_skip_verify": false,
"headers": [],
"type": "http"
},
"pg": {
"interval": "15s",
"timeout": "10s",
"grace_period": "30s",
"restart_limit": 0,
"port": 5500,
"method": "get",
"protocol": "http",
"path": "/flycheck/pg",
"tls_skip_verify": false,
"headers": [],
"type": "http"
},
"vm": {
"interval": "1m",
"timeout": "10s",
"grace_period": "1s",
"restart_limit": 0,
"port": 5500,
"method": "get",
"protocol": "http",
"path": "/flycheck/vm",
"tls_skip_verify": false,
"headers": [],
"type": "http"
}
},
"env": {
"PRIMARY_REGION": "mad"
},
"metrics": {
"port": 9187,
"path": "/metrics"
}
}
}
}
}
}
DEBUG --> POST https://api.fly.io/graphql
DEBUG {
"query": "query($appName: String!) { app(name: $appName) { autoscaling { enabled minCount maxCount balanceRegions regions { code minCount weight } } } }",
"variables": {
"appName": "diciprd-db"
}
}
DEBUG {}
DEBUG <-- 200 https://api.fly.io/graphql (115.12ms)
DEBUG {
"data": {
"app": {
"autoscaling": {
"enabled": false,
"minCount": 0,
"maxCount": 10,
"balanceRegions": false,
"regions": [
{
"code": "mad",
"minCount": 0,
"weight": 100
}
]
}
}
}
}
DEBUG --> POST https://api.fly.io/graphql
DEBUG {
"query": "query($appName: String!, $showCompleted: Boolean!) { appstatus:app(name: $appName) { allocations(showCompleted: $showCompleted) { id idShort version latestVersion status desiredStatus totalCheckCount passingCheckCount warningCheckCount criticalCheckCount createdAt updatedAt canary region restarts healthy privateIP taskName checks { status output name } } } }",
"variables": {
"appName": "diciprd-db",
"showCompleted": false
}
}
DEBUG {}
DEBUG <-- 200 https://api.fly.io/graphql (171.25ms)
DEBUG {
"data": {
"appstatus": {
"allocations": [
{
"id": "be3238c8-1e7a-6f91-21de-4bfa6c75c8e3",
"idShort": "be3238c8",
"version": 1,
"latestVersion": true,
"status": "running",
"desiredStatus": "run",
"totalCheckCount": 3,
"passingCheckCount": 3,
"warningCheckCount": 0,
"criticalCheckCount": 0,
"createdAt": "2023-08-22T09:59:24Z",
"updatedAt": "2023-08-22T09:59:24Z",
"canary": false,
"region": "mad",
"restarts": 0,
"healthy": true,
"privateIP": "fdaa:0:4d65:a7b:2658:1:af82:2",
"taskName": "app",
"checks": [
{
"status": "passing",
"name": "vm",
"output": "HTTP GET http://172.19.65.90:5500/flycheck/vm: 200 OK Output: [✓] checkDisk: 46.33 GB (94.7%) free space on /data/ (30.42µs)\n[✓] checkLoad: load averages: 0.01 0.03 0.00 (48.78µs)\n[✓] memory: system spent 0s of the last 60s waiting on memory (22.36µs)\n[✓] cpu: system spent 906ms of the last 60s waiting on cpu (24.71µs)\n[✓] io: system spent 0s of the last 60s waiting on io (16.22µs)"
},
{
"status": "passing",
"name": "pg",
"output": "HTTP GET http://172.19.65.90:5500/flycheck/pg: 200 OK Output: [✓] transactions: readonly (236.11µs)\n[✓] replication: syncing from fdaa:0:4d65:a7b:2656:1:af83:2 (119.96µs)\n[✓] connections: 7 used, 3 reserved, 300 max (3.64ms)"
},
{
"status": "passing",
"name": "role",
"output": "replica"
}
]
},
{
"id": "cb3965d1-89f6-6b95-091e-7065ea63da74",
"idShort": "cb3965d1",
"version": 1,
"latestVersion": true,
"status": "running",
"desiredStatus": "run",
"totalCheckCount": 3,
"passingCheckCount": 3,
"warningCheckCount": 0,
"criticalCheckCount": 0,
"createdAt": "2023-08-22T09:57:50Z",
"updatedAt": "2023-08-22T09:57:50Z",
"canary": false,
"region": "mad",
"restarts": 0,
"healthy": true,
"privateIP": "fdaa:0:4d65:a7b:2656:1:af83:2",
"taskName": "app",
"checks": [
{
"status": "passing",
"name": "vm",
"output": "HTTP GET http://172.19.64.242:5500/flycheck/vm: 200 OK Output: [✓] checkDisk: 46.32 GB (94.7%) free space on /data/ (31.42µs)\n[✓] checkLoad: load averages: 0.00 0.02 0.02 (73.08µs)\n[✓] memory: system spent 0s of the last 60s waiting on memory (52.36µs)\n[✓] cpu: system spent 954ms of the last 60s waiting on cpu (17.02µs)\n[✓] io: system spent 0s of the last 60s waiting on io (13.34µs)"
},
{
"status": "passing",
"name": "pg",
"output": "HTTP GET http://172.19.64.242:5500/flycheck/pg: 200 OK Output: [✓] transactions: read/write (171.89µs)\n[✓] replicationLag: fdaa:0:4d65:a7b:2658:1:af82:2 is lagging 0s (111ns)\n[✓] connections: 10 used, 3 reserved, 300 max (3.16ms)"
},
{
"status": "passing",
"name": "role",
"output": "leader"
}
]
}
]
}
}
}
DEBUG --> POST https://api.fly.io/graphql
DEBUG {
"query": "query($appName: String!) { app(name: $appName) { vmSize { name cpuCores memoryGb memoryMb priceMonth priceSecond } taskGroupCounts { name count } processGroups { name vmSize { name cpuCores memoryGb memoryMb priceMonth priceSecond } maxPerRegion } } }",
"variables": {
"appName": "diciprd-db"
}
}
DEBUG {}
DEBUG <-- 200 https://api.fly.io/graphql (122.12ms)
DEBUG {
"data": {
"app": {
"vmSize": {
"name": "dedicated-cpu-1x",
"cpuCores": 1.0,
"memoryGb": 2.0,
"memoryMb": 2048,
"priceMonth": 31.0,
"priceSecond": 1.196e-05
},
"taskGroupCounts": [
{
"name": "app",
"count": 2
}
],
"processGroups": [
{
"name": "app",
"vmSize": {
"name": "dedicated-cpu-1x",
"cpuCores": 1.0,
"memoryGb": 2.0,
"memoryMb": 2048,
"priceMonth": 31.0,
"priceSecond": 1.196e-05
},
"maxPerRegion": 0
}
]
}
}
}
DEBUG --> POST https://api.fly.io/graphql
DEBUG {
"query": "mutation($appName: ID!) { enablePostgresConsul(input: {appId: $appName}) { consulUrl } }",
"variables": {
"appName": "diciprd-db"
}
}
DEBUG {}
DEBUG <-- 200 https://api.fly.io/graphql (219.11ms)
DEBUG {
"data": {
"enablePostgresConsul": {
"consulUrl": "https://:baafa451-78b9-4c18-7b30-c3ebe054d7e2@consul-fra-2.fly-shared.net/diciprd-db-6r85yql574r92pvl/"
}
}
}
DEBUG --> POST https://api.fly.io/graphql
DEBUG {
"query": "mutation($input: ValidateWireGuardPeersInput!) { validateWireGuardPeers(input: $input) { invalidPeerIps } }",
"variables": {
"input": {
"peerIps": [
"fdaa:0:4d65:a7b:1596:0:a:202"
]
}
}
}
DEBUG {}
DEBUG <-- 200 https://api.fly.io/graphql (108.02ms)
DEBUG {
"data": {
"validateWireGuardPeers": {
"invalidPeerIps": []
}
}
}
DEBUG --> POST https://api.fly.io/graphql
DEBUG {
"query": "\n# @genqlient\nquery AgentGetInstances ($appName: String!) {\n\tapp(name: $appName) {\n\t\torganization {\n\t\t\tslug\n\t\t}\n\t\tid\n\t\tname\n\t\tallocations(showCompleted: false) {\n\t\t\tid\n\t\t\tregion\n\t\t\tprivateIP\n\t\t}\n\t\tmachines {\n\t\t\tnodes {\n\t\t\t\tstate\n\t\t\t\tid\n\t\t\t\tregion\n\t\t\t\tips {\n\t\t\t\t\tnodes {\n\t\t\t\t\t\tkind\n\t\t\t\t\t\tfamily\n\t\t\t\t\t\tip\n\t\t\t\t\t}\n\t\t\t\t}\n\t\t\t}\n\t\t}\n\t}\n}\n",
"variables": {
"appName": "diciprd-db"
},
"operationName": "AgentGetInstances"
}
DEBUG {0xc000e65740}
DEBUG <-- 200 https://api.fly.io/graphql (149.1ms)
DEBUG {
"data": {
"app": {
"organization": {
"slug": "personal"
},
"id": "diciprd-db",
"name": "diciprd-db",
"allocations": [
{
"id": "be3238c8-1e7a-6f91-21de-4bfa6c75c8e3",
"region": "mad",
"privateIP": "fdaa:0:4d65:a7b:2658:1:af82:2"
},
{
"id": "cb3965d1-89f6-6b95-091e-7065ea63da74",
"region": "mad",
"privateIP": "fdaa:0:4d65:a7b:2656:1:af83:2"
}
],
"machines": {
"nodes": []
}
}
}
}
DEBUG gqlGetInstances() result: &{[mad.diciprd-db.internal (fdaa:0:4d65:a7b:2658:1:af82:2) mad.diciprd-db.internal (fdaa:0:4d65:a7b:2656:1:af83:2)] [fdaa:0:4d65:a7b:2658:1:af82:2 fdaa:0:4d65:a7b:2656:1:af83:2]}
DEBUG gqlErr: <nil> agentErr: <nil>
DEBUG flypg will connect to: http://fdaa:0:4d65:a7b:2658:1:af82:2:5500
DEBUG --> GET http://fdaa:0:4d65:a7b:2658:1:af82:2:5500/commands/admin/role
DEBUG <-- 200 http://fdaa:0:4d65:a7b:2658:1:af82:2:5500/commands/admin/role (43.5ms)
DEBUG {
"result": "replica"
}
DEBUG flypg will connect to: http://fdaa:0:4d65:a7b:2656:1:af83:2:5500
DEBUG --> GET http://fdaa:0:4d65:a7b:2656:1:af83:2:5500/commands/admin/role
DEBUG <-- 200 http://fdaa:0:4d65:a7b:2656:1:af83:2:5500/commands/admin/role (43.71ms)
DEBUG {
"result": "leader"
}
DEBUG flypg will connect to: http://fdaa:0:4d65:a7b:2656:1:af83:2:5500
DEBUG --> GET http://fdaa:0:4d65:a7b:2656:1:af83:2:5500/commands/admin/settings/view
DEBUG [
"max_wal_senders",
"max_replication_slots"
]
DEBUG {0xc000ee70b0}
DEBUG <-- 200 http://fdaa:0:4d65:a7b:2656:1:af83:2:5500/commands/admin/settings/view (46.81ms)
DEBUG {
"result": {
"settings": [
{
"name": "max_replication_slots",
"setting": "20",
"context": "postmaster",
"vartype": "integer",
"min_val": "0",
"max_val": "262143",
"short_desc": "Sets the maximum number of simultaneously defined replication slots."
},
{
"name": "max_wal_senders",
"setting": "47",
"context": "postmaster",
"vartype": "integer",
"min_val": "0",
"max_val": "262143",
"short_desc": "Sets the maximum number of simultaneously running WAL sender processes."
}
]
}
}
DEBUG --> GET https://api.machines.dev/v1/apps/diciprd-db/machines
DEBUG <-- 200 https://api.machines.dev/v1/apps/diciprd-db/machines (274.7ms)
DEBUG []
This migration process will do the following, in order:
* Update your postgres app to the latest supported image version
* Create volumes for new PG replicas to use
* Lock your application, preventing changes during the migration
* Create machines, copying the configuration of each existing VM
* Create 2 "app" machines
* Wait for at least one new in-region PG replica to become synced
* Remove legacy VMs after health checks pass for the new machines
* Remove 2 allocs
* Set the application platform version to "machines"
* Unlock your application
? Would you like to continue? Yes
==> Migrating diciprd-db to the V2 platform
> Upgrading postgres image
DEBUG --> POST https://api.fly.io/graphql
Thanks for all the debug info! Very helpful, and very strange bug here.
I pushed out an update that slightly changes how migrate-to-v2 gets information about current volumes, which could end up fixing this problem (or it could do nothing). That should be available within an hour or so, as flyctl v0.1.83.
I hope this works for you! If it doesn’t, I’d be happy either continuing to try to track this issue down, or helping you export and reimport your database. Sorry for the trouble!
1 Like
Well, that did fix the previous error, but it didn’t finished successfully and I guess volumes weren’t attached to the new app. Trying to do that manually to see if I can recover my database.
ubuntu@ip-172-31-35-191:~$ fly migrate-to-v2
This migration process will do the following, in order:
* Update your postgres app to the latest supported image version
* Create volumes for new PG replicas to use
* Lock your application, preventing changes during the migration
* Create machines, copying the configuration of each existing VM
* Create 2 "app" machines
* Wait for at least one new in-region PG replica to become synced
* Remove legacy VMs after health checks pass for the new machines
* Remove 2 allocs
* Set the application platform version to "machines"
* Unlock your application
? Would you like to continue? Yes
==> Migrating diciprd-db to the V2 platform
> Upgrading postgres image
> Setting postgres primary to readonly
> Creating new postgres volumes
> Locking app to prevent changes during the migration
> Enabling machine creation on app
> Creating an app release to register this migration
> Starting machines
INFO Using wait timeout: 5m0s lease timeout: 13s delay between lease refreshes: 4s
failed while migrating: Process group 'app' needs volumes with name 'pg_data_machines' to fullfill mounts defined in fly.toml; Run `fly volume create pg_data_machines -r REGION` for the following regions and counts: mad=2
? Would you like to enter interactive troubleshooting mode? If not, the migration will be rolled back. No
==> (!) An error has occurred. Attempting to rollback changes...
> Removing machines
> Disabling readonly
failed while rolling back application: Post "http://fdaa:0:4d65:a7b:1bf:abec:d924:2:5500/commands/admin/haproxy/restart": connect tcp [fdaa:0:4d65:a7b:1bf:abec:d924:2]:5500: operation timed out
Error: Process group 'app' needs volumes with name 'pg_data_machines' to fullfill mounts defined in fly.toml; Run `fly volume create pg_data_machines -r REGION` for the following regions and counts: mad=2
ubuntu@ip-172-31-35-191:~$
ubuntu@ip-172-31-35-191:~$ fly volume create pg_data_machines -r mad
Warning! Individual volumes are pinned to individual hosts. You should create two or more volumes per application. You will have downtime if you only create one. Learn more at https://fly.io/docs/reference/volumes/
? Do you still want to use the volumes feature? No
ubuntu@ip-172-31-35-191:~$ fly volume create pg_data_machines -r mad=2
Warning! Individual volumes are pinned to individual hosts. You should create two or more volumes per application. You will have downtime if you only create one. Learn more at https://fly.io/docs/reference/volumes/
? Do you still want to use the volumes feature? Yes
Error: region mad=2 not found
ubuntu@ip-172-31-35-191:~$ fly volume create pg_data_machines -r mad
Warning! Individual volumes are pinned to individual hosts. You should create two or more volumes per application. You will have downtime if you only create one. Learn more at https://fly.io/docs/reference/volumes/
? Do you still want to use the volumes feature? Yes
ID: vol_5vgl30xw80kqqe8r
Name: pg_data_machines
App: diciprd-db
Region: mad
Zone: bc6b
Size GB: 3
Encrypted: true
Created at: 23 Aug 23 09:52 UTC
Update: the application is now working, with a single volume attached (I didn’t do anything, aside from creating the new volume pg_data_machines - as you can see in the previews command sequence - but that is not being used, so I’m guessing I can delete it 
) - it recovered by itself (maybe I was too quick trying to refresh the app).
I can’t find in the documentation instructions on how to attach an existing volume to an existing app, so I’m not sure what to do next (I do have the ddl and data on hand, if needed - this is a very small database, with about 250kb of data, so creating a new cluster is always an option for me at the moment).