Thanks for your detailed response Jerome!
Ah thank you. This seems promising:
➜ postgres-ha git:(main) export LOG_LEVEL=debug && fly deploy
DEBUG Loaded flyctl config from/Users/cduruk/.fly/config.yml
DEBUG Loading app config from/Users/cduruk/Code/felt/postgres-ha/fly.toml
DEBUG Working Directory: /Users/cduruk/Code/felt/postgres-ha
DEBUG App Config File: /Users/cduruk/Code/felt/postgres-ha/fly.toml
Deploying felt-pg
==> Validating app configuration
DEBUG --> POST https://api.fly.io/graphql {{"query":"query($appName: String!, $definition: JSON!) { app(name: $appName) { parseConfig(definition: $definition) { definition valid errors services { description } } } }","variables":{"appName":"felt-pg","definition":{"checks":{"pg":{"args":["pg"],"command":"flycheck","grace_period":"10s","interval":30000,"restart_limit":0,"timeout":10000,"type":"script"},"role":{"args":["role"],"command":"flycheck","interval":60000,"restart_limit":0,"timeout":10000,"type":"script"},"vm":{"args":["vm"],"command":"flycheck","interval":30000,"restart_limit":0,"timeout":10000,"type":"script"}},"env":{"BACKEND_STORE":"etcdv3","PRIMARY_REGION":"lax"},"experimental":{"auto_rollback":false,"enable_etcd":true,"private_network":true},"kill_signal":"SIGTERM","kill_timeout":300,"metrics":{"path":"/metrics","port":9187},"mount":{"destination":"/data","source":"pg_data"}}}}
}
DEBUG <-- 200 https://api.fly.io/graphql (211.92ms) {"data":{"app":{"parseConfig":{"definition":{"kill_timeout":300,"kill_signal":"SIGTERM","mounts":[{"source":"pg_data","destination":"/data","encrypted":false}],"experimental":{"allowed_public_ports":[],"auto_rollback":false,"enable_etcd":true,"private_network":true},"services":[],"checks":{"pg":{"interval":"30s","timeout":"10s","grace_period":"10s","restart_limit":0,"command":"flycheck","args":["pg"],"type":"script"},"role":{"interval":"1m","timeout":"10s","grace_period":"1s","restart_limit":0,"command":"flycheck","args":["role"],"type":"script"},"vm":{"interval":"30s","timeout":"10s","grace_period":"1s","restart_limit":0,"command":"flycheck","args":["vm"],"type":"script"}},"env":{"BACKEND_STORE":"etcdv3","PRIMARY_REGION":"lax"},"metrics":{"port":9187,"path":"/metrics"}},"valid":true,"errors":[],"services":[]}}}}
--> Validating app configuration done
DEBUG trying local docker daemon
DEBUG Local docker daemon unavailable
DEBUG trying remote docker daemon
DEBUG Trying 'Buildpacks' strategy
DEBUG no buildpack builder configured, skipping
DEBUG result image:<nil> error:<nil>
DEBUG Trying 'Dockerfile' strategy
DEBUG --> POST https://api.fly.io/graphql {{"query":"mutation($input: EnsureRemoteBuilderInput!) { ensureRemoteBuilder(input: $input) { url, app { name } } }","variables":{"input":{"appName":"felt-pg","organizationId":null}}}
}
DEBUG <-- 200 https://api.fly.io/graphql (609.43ms) {"data":{"ensureRemoteBuilder":{"url":"https://fly-builder-delicate-field-7831.fly.dev","app":{"name":"fly-builder-delicate-field-7831"}}}}
DEBUG Remote Docker builder host: tcp://fly-builder-delicate-field-7831.internal:2375
Waiting for remote builder fly-builder-delicate-field-7831... starting ⣾ DEBUG --> POST https://api.fly.io/graphql {{"query":"query ($appName: String!) { app(name: $appName) { id name hostname deployed status version appUrl organization { id slug } services { description protocol internalPort ports { port handlers } } ipAddresses { nodes { id address type createdAt } } } }","variables":{"appName":"felt-pg"}}
}
DEBUG --> POST https://api.fly.io/graphql {{"query":"query($appName: String!, $showCompleted: Boolean!) { appstatus:app(name: $appName) { id name deployed status hostname version appUrl organization { slug } deploymentStatus { id status version description placedCount promoted desiredCount healthyCount unhealthyCount } allocations(showCompleted: $showCompleted) { id idShort version latestVersion status desiredStatus totalCheckCount passingCheckCount warningCheckCount criticalCheckCount createdAt updatedAt canary region restarts healthy privateIP checks { status output name } } } }","variables":{"appName":"fly-builder-delicate-field-7831","showCompleted":false}}
}
DEBUG <-- 200 https://api.fly.io/graphql (200.55ms) {"data":{"appstatus":{"id":"fly-builder-delicate-field-7831","name":"fly-builder-delicate-field-7831","deployed":true,"status":"dead","hostname":"fly-builder-delicate-field-7831.fly.dev","version":1,"appUrl":"https://213.188.196.127","organization":{"slug":"felt"},"deploymentStatus":null,"allocations":[]}}}
Waiting for remote builder fly-builder-delicate-field-7831... starting ⣽ DEBUG <-- 200 https://api.fly.io/graphql (326.19ms) {"data":{"app":{"id":"felt-pg","name":"felt-pg","hostname":"felt-pg.fly.dev","deployed":true,"status":"running","version":1,"appUrl":null,"organization":{"id":"60L7BO3NPGLMXHvjvN6nmoY6nAFgGN","slug":"felt"},"services":[],"ipAddresses":{"nodes":[]}}}}
DEBUG creating wireguard config for org felt
DEBUG Found WireGuard state in local configuration
DEBUG Establishing WireGuard connection (interactive-DurukAir-M1-can-felt-com-979)
Waiting for remote builder fly-builder-delicate-field-7831... starting ⡿ DEBUG --> POST https://api.fly.io/graphql {{"query":"query($appName: String!, $showCompleted: Boolean!) { appstatus:app(name: $appName) { id name deployed status hostname version appUrl organization { slug } deploymentStatus { id status version description placedCount promoted desiredCount healthyCount unhealthyCount } allocations(showCompleted: $showCompleted) { id idShort version latestVersion status desiredStatus totalCheckCount passingCheckCount warningCheckCount criticalCheckCount createdAt updatedAt canary region restarts healthy privateIP checks { status output name } } } }","variables":{"appName":"fly-builder-delicate-field-7831","showCompleted":false}}
}
Waiting for remote builder fly-builder-delicate-field-7831... starting ⣯ DEBUG <-- 200 https://api.fly.io/graphql (340.21ms) {"data":{"appstatus":{"id":"fly-builder-delicate-field-7831","name":"fly-builder-delicate-field-7831","deployed":true,"status":"running","hostname":"fly-builder-delicate-field-7831.fly.dev","version":1,"appUrl":"https://213.188.196.127","organization":{"slug":"felt"},"deploymentStatus":null,"allocations":[]}}}
Waiting for remote builder fly-builder-delicate-field-7831... starting ⣻ DEBUG --> POST https://api.fly.io/graphql {{"query":"query($appName: String!, $showCompleted: Boolean!) { appstatus:app(name: $appName) { id name deployed status hostname version appUrl organization { slug } deploymentStatus { id status version description placedCount promoted desiredCount healthyCount unhealthyCount } allocations(showCompleted: $showCompleted) { id idShort version latestVersion status desiredStatus totalCheckCount passingCheckCount warningCheckCount criticalCheckCount createdAt updatedAt canary region restarts healthy privateIP checks { status output name } } } }","variables":{"appName":"fly-builder-delicate-field-7831","showCompleted":false}}
}
DEBUG <-- 200 https://api.fly.io/graphql (211.71ms) {"data":{"appstatus":{"id":"fly-builder-delicate-field-7831","name":"fly-builder-delicate-field-7831","deployed":true,"status":"running","hostname":"fly-builder-delicate-field-7831.fly.dev","version":1,"appUrl":"https://213.188.196.127","organization":{"slug":"felt"},"deploymentStatus":null,"allocations":[{"id":"81fd30a3-b637-c9dc-e523-405068b20142","idShort":"81fd30a3","version":0,"latestVersion":false,"status":"pending","desiredStatus":"run","totalCheckCount":0,"passingCheckCount":0,"warningCheckCount":0,"criticalCheckCount":0,"createdAt":"2021-08-04T15:41:45Z","updatedAt":"2021-08-04T15:41:45Z","canary":false,"region":"iad","restarts":0,"healthy":true,"privateIP":null,"checks":[]}]}}}
Waiting for remote builder fly-builder-delicate-field-7831... connecting ⣽ DEBUG Remote builder unavailable, retrying in 200ms (err: Cannot connect to the Docker daemon at tcp://fly-builder-delicate-field-7831.internal:2375. Is the docker daemon running?)
Waiting for remote builder fly-builder-delicate-field-7831... connecting ⣽ DEBUG Remote builder unavailable, retrying in 220.661499ms (err: Cannot connect to the Docker daemon at tcp://fly-builder-delicate-field-7831.internal:2375. Is the docker daemon running?)
Waiting for remote builder fly-builder-delicate-field-7831... connecting ⣽ DEBUG Remote builder unavailable, retrying in 232.435039ms (err: Cannot connect to the Docker daemon at tcp://fly-builder-delicate-field-7831.internal:2375. Is the docker daemon running?)
Waiting for remote builder fly-builder-delicate-field-7831... connecting ⣻ DEBUG Remote builder unavailable, retrying in 303.368908ms (err: Cannot connect to the Docker daemon at tcp://fly-builder-delicate-field-7831.internal:2375. Is the docker daemon running?)
Waiting for remote builder fly-builder-delicate-field-7831... connecting ⣻ DEBUG Remote builder unavailable, retrying in 332.341387ms (err: Cannot connect to the Docker daemon at tcp://fly-builder-delicate-field-7831.internal:2375. Is the docker daemon running?)
Waiting for remote builder fly-builder-delicate-field-7831... connecting ⢿ DEBUG Remote builder unavailable, retrying in 225.12494ms (err: Cannot connect to the Docker daemon at tcp://fly-builder-delicate-field-7831.internal:2375. Is the docker daemon running?)
Waiting for remote builder fly-builder-delicate-field-7831... connecting ⣯
If this is helpful, I was able to unblock myself by spinning up an EC2 instance to deploy stuff, but obviously not as fun/easy as doing stuff locally.
I noticed the Wireguard state stuff, so I removed those peers using fly wireguard remove just in case they were getting in the way, but that didnt seem to help.