Cannot connect to remote builder

Questions / Help Build debugging
1

Trying to deploy a new app based on a Docker image, having trouble using the remote builders. I basically cloned a repo with a Dockerfile, went thru the fly launch and fly deploy workflow — it created the fly.toml file and i basically didn’t change anything in it.

I can see the remote builder app starting up but it’s hanging at the connecting state and then times out.

WARN Remote builder did not start on time. Check remote builder logs with `flyctl logs -a fly-builder-winter-fog-5754`
Error error connecting to docker: remote builder app unavailable

When I look at the logs of the builder, I just see this

2021-08-03T19:10:32.214967167Z app[d771b9d5] iad [info] time="2021-08-03T19:10:32.213942018Z" level=debug msg="checking docker activity"
2021-08-03T19:10:32.216426771Z app[d771b9d5] iad [info] time="2021-08-03T19:10:32.214995934Z" level=debug msg="Calling GET /v1.41/containers/json?filters=%7B%22status%22%3A%7B%22running%22%3Atrue%7D%7D&limit=0"
2021-08-03T19:10:33.218526602Z app[d771b9d5] iad [info] time="2021-08-03T19:10:33.217446115Z" level=debug msg="checking docker activity"
2021-08-03T19:10:33.219918058Z app[d771b9d5] iad [info] time="2021-08-03T19:10:33.217752690Z" level=debug msg="Calling GET /v1.41/containers/json?filters=%7B%22status%22%3A%7B%22running%22%3Atrue%7D%7D&limit=0"

The interesting thing is if I create a similar app by just fly launch --image and point it at the urbica:martin image, it seems to work fine. It’s essentially the same Dockerfile though, so maybe I’m not wrapping my head around something right.

I’m on Big Sur with

➜  ~ fly version
fly v0.0.229 darwin/arm64 Commit: 7800371 BuildDate: 2021-07-13T20:31:06Z
2

Ugh, that’s annoying. flyctl can’t reach the remote builder over the on-demand WireGuard tunnel. This could be from a network config, DNS, firewalls, or the builder app not starting fast enough. Whatever the reason, we’re working on fixing it or explaining why the connection fails.

Do you get the same error if you try again?

flyctl launch --image creates an app using a public image from Docker Hub while cloning and deploying will build the app from source (the Dockerfile). You’re right, the dockerfile is the same, but the prebuilt image bypasses the remote builder.

3

I do, unfortunately. I had the macOS Wireguard app on with the tunnel activated when I first tried but deactivating etc didnt seem to help. Otherwise, I have a pretty generic home network I’m working off of — the regular DNS from my ISP, a home network with no specific firewall etc.

Any debugging steps that would be helpful to you?

4

Oh not sure if this is helpful — I did notice from the web dashboard that some of the builder apps occasionally were created under my personal account versus on the Felt organization profile (basing off of the superscript). I am quite confident that I never picked my personal profile, and it seems to be mostly right, but it was a tad worrying.

5

I don’t have access to a Linux machine right now but this is intriguing. Sorry if me trying to debug this is annoying on the docs.

6

I’m not sure if WireGuard connections from the macos App would interefere, but it might yes.

Can you try LOG_LEVEL=debug with your flyctl deploy command? This should print a lot more information and perhaps the “real” reason for the connection issues.

The remote builder app should only be created once per organization, when needed. If you build an app assigned to your personal org, it will create a remote builder on it. You’re not being billed for it. However, it should be provisioned on the correct organization, if not that’s an issue (first time I’ve seen this though).

7

Thanks for your detailed response Jerome!

Ah thank you. This seems promising:

➜  postgres-ha git:(main) export LOG_LEVEL=debug && fly deploy
DEBUG Loaded flyctl config from/Users/cduruk/.fly/config.yml
DEBUG Loading app config from/Users/cduruk/Code/felt/postgres-ha/fly.toml
DEBUG Working Directory: /Users/cduruk/Code/felt/postgres-ha
DEBUG App Config File: /Users/cduruk/Code/felt/postgres-ha/fly.toml
Deploying felt-pg
==> Validating app configuration
DEBUG --> POST https://api.fly.io/graphql {{"query":"query($appName: String!, $definition: JSON!) { app(name: $appName) { parseConfig(definition: $definition) { definition valid errors services { description } } } }","variables":{"appName":"felt-pg","definition":{"checks":{"pg":{"args":["pg"],"command":"flycheck","grace_period":"10s","interval":30000,"restart_limit":0,"timeout":10000,"type":"script"},"role":{"args":["role"],"command":"flycheck","interval":60000,"restart_limit":0,"timeout":10000,"type":"script"},"vm":{"args":["vm"],"command":"flycheck","interval":30000,"restart_limit":0,"timeout":10000,"type":"script"}},"env":{"BACKEND_STORE":"etcdv3","PRIMARY_REGION":"lax"},"experimental":{"auto_rollback":false,"enable_etcd":true,"private_network":true},"kill_signal":"SIGTERM","kill_timeout":300,"metrics":{"path":"/metrics","port":9187},"mount":{"destination":"/data","source":"pg_data"}}}}
}
DEBUG <-- 200 https://api.fly.io/graphql (211.92ms) {"data":{"app":{"parseConfig":{"definition":{"kill_timeout":300,"kill_signal":"SIGTERM","mounts":[{"source":"pg_data","destination":"/data","encrypted":false}],"experimental":{"allowed_public_ports":[],"auto_rollback":false,"enable_etcd":true,"private_network":true},"services":[],"checks":{"pg":{"interval":"30s","timeout":"10s","grace_period":"10s","restart_limit":0,"command":"flycheck","args":["pg"],"type":"script"},"role":{"interval":"1m","timeout":"10s","grace_period":"1s","restart_limit":0,"command":"flycheck","args":["role"],"type":"script"},"vm":{"interval":"30s","timeout":"10s","grace_period":"1s","restart_limit":0,"command":"flycheck","args":["vm"],"type":"script"}},"env":{"BACKEND_STORE":"etcdv3","PRIMARY_REGION":"lax"},"metrics":{"port":9187,"path":"/metrics"}},"valid":true,"errors":[],"services":[]}}}}
--> Validating app configuration done
DEBUG trying local docker daemon
DEBUG Local docker daemon unavailable
DEBUG trying remote docker daemon
DEBUG Trying 'Buildpacks' strategy
DEBUG no buildpack builder configured, skipping
DEBUG result image:<nil> error:<nil>
DEBUG Trying 'Dockerfile' strategy
DEBUG --> POST https://api.fly.io/graphql {{"query":"mutation($input: EnsureRemoteBuilderInput!) { ensureRemoteBuilder(input: $input) { url, app { name } } }","variables":{"input":{"appName":"felt-pg","organizationId":null}}}
}
DEBUG <-- 200 https://api.fly.io/graphql (609.43ms) {"data":{"ensureRemoteBuilder":{"url":"https://fly-builder-delicate-field-7831.fly.dev","app":{"name":"fly-builder-delicate-field-7831"}}}}
DEBUG Remote Docker builder host: tcp://fly-builder-delicate-field-7831.internal:2375
Waiting for remote builder fly-builder-delicate-field-7831... starting ⣾ DEBUG --> POST https://api.fly.io/graphql {{"query":"query ($appName: String!) { app(name: $appName) { id name hostname deployed status version appUrl organization { id slug } services { description protocol internalPort ports { port handlers } } ipAddresses { nodes { id address type createdAt } } } }","variables":{"appName":"felt-pg"}}
}
DEBUG --> POST https://api.fly.io/graphql {{"query":"query($appName: String!, $showCompleted: Boolean!) { appstatus:app(name: $appName) { id name deployed status hostname version appUrl organization { slug } deploymentStatus { id status version description placedCount promoted desiredCount healthyCount unhealthyCount } allocations(showCompleted: $showCompleted) { id idShort version latestVersion status desiredStatus totalCheckCount passingCheckCount warningCheckCount criticalCheckCount createdAt updatedAt canary region restarts healthy privateIP checks { status output name } } } }","variables":{"appName":"fly-builder-delicate-field-7831","showCompleted":false}}
}
DEBUG <-- 200 https://api.fly.io/graphql (200.55ms) {"data":{"appstatus":{"id":"fly-builder-delicate-field-7831","name":"fly-builder-delicate-field-7831","deployed":true,"status":"dead","hostname":"fly-builder-delicate-field-7831.fly.dev","version":1,"appUrl":"https://213.188.196.127","organization":{"slug":"felt"},"deploymentStatus":null,"allocations":[]}}}
Waiting for remote builder fly-builder-delicate-field-7831... starting ⣽ DEBUG <-- 200 https://api.fly.io/graphql (326.19ms) {"data":{"app":{"id":"felt-pg","name":"felt-pg","hostname":"felt-pg.fly.dev","deployed":true,"status":"running","version":1,"appUrl":null,"organization":{"id":"60L7BO3NPGLMXHvjvN6nmoY6nAFgGN","slug":"felt"},"services":[],"ipAddresses":{"nodes":[]}}}}
DEBUG creating wireguard config for org felt
DEBUG Found WireGuard state in local configuration
DEBUG Establishing WireGuard connection (interactive-DurukAir-M1-can-felt-com-979)
Waiting for remote builder fly-builder-delicate-field-7831... starting ⡿ DEBUG --> POST https://api.fly.io/graphql {{"query":"query($appName: String!, $showCompleted: Boolean!) { appstatus:app(name: $appName) { id name deployed status hostname version appUrl organization { slug } deploymentStatus { id status version description placedCount promoted desiredCount healthyCount unhealthyCount } allocations(showCompleted: $showCompleted) { id idShort version latestVersion status desiredStatus totalCheckCount passingCheckCount warningCheckCount criticalCheckCount createdAt updatedAt canary region restarts healthy privateIP checks { status output name } } } }","variables":{"appName":"fly-builder-delicate-field-7831","showCompleted":false}}
}
Waiting for remote builder fly-builder-delicate-field-7831... starting ⣯ DEBUG <-- 200 https://api.fly.io/graphql (340.21ms) {"data":{"appstatus":{"id":"fly-builder-delicate-field-7831","name":"fly-builder-delicate-field-7831","deployed":true,"status":"running","hostname":"fly-builder-delicate-field-7831.fly.dev","version":1,"appUrl":"https://213.188.196.127","organization":{"slug":"felt"},"deploymentStatus":null,"allocations":[]}}}
Waiting for remote builder fly-builder-delicate-field-7831... starting ⣻ DEBUG --> POST https://api.fly.io/graphql {{"query":"query($appName: String!, $showCompleted: Boolean!) { appstatus:app(name: $appName) { id name deployed status hostname version appUrl organization { slug } deploymentStatus { id status version description placedCount promoted desiredCount healthyCount unhealthyCount } allocations(showCompleted: $showCompleted) { id idShort version latestVersion status desiredStatus totalCheckCount passingCheckCount warningCheckCount criticalCheckCount createdAt updatedAt canary region restarts healthy privateIP checks { status output name } } } }","variables":{"appName":"fly-builder-delicate-field-7831","showCompleted":false}}
}
DEBUG <-- 200 https://api.fly.io/graphql (211.71ms) {"data":{"appstatus":{"id":"fly-builder-delicate-field-7831","name":"fly-builder-delicate-field-7831","deployed":true,"status":"running","hostname":"fly-builder-delicate-field-7831.fly.dev","version":1,"appUrl":"https://213.188.196.127","organization":{"slug":"felt"},"deploymentStatus":null,"allocations":[{"id":"81fd30a3-b637-c9dc-e523-405068b20142","idShort":"81fd30a3","version":0,"latestVersion":false,"status":"pending","desiredStatus":"run","totalCheckCount":0,"passingCheckCount":0,"warningCheckCount":0,"criticalCheckCount":0,"createdAt":"2021-08-04T15:41:45Z","updatedAt":"2021-08-04T15:41:45Z","canary":false,"region":"iad","restarts":0,"healthy":true,"privateIP":null,"checks":[]}]}}}
Waiting for remote builder fly-builder-delicate-field-7831... connecting ⣽ DEBUG Remote builder unavailable, retrying in 200ms (err: Cannot connect to the Docker daemon at tcp://fly-builder-delicate-field-7831.internal:2375. Is the docker daemon running?)
Waiting for remote builder fly-builder-delicate-field-7831... connecting ⣽ DEBUG Remote builder unavailable, retrying in 220.661499ms (err: Cannot connect to the Docker daemon at tcp://fly-builder-delicate-field-7831.internal:2375. Is the docker daemon running?)
Waiting for remote builder fly-builder-delicate-field-7831... connecting ⣽ DEBUG Remote builder unavailable, retrying in 232.435039ms (err: Cannot connect to the Docker daemon at tcp://fly-builder-delicate-field-7831.internal:2375. Is the docker daemon running?)
Waiting for remote builder fly-builder-delicate-field-7831... connecting ⣻ DEBUG Remote builder unavailable, retrying in 303.368908ms (err: Cannot connect to the Docker daemon at tcp://fly-builder-delicate-field-7831.internal:2375. Is the docker daemon running?)
Waiting for remote builder fly-builder-delicate-field-7831... connecting ⣻ DEBUG Remote builder unavailable, retrying in 332.341387ms (err: Cannot connect to the Docker daemon at tcp://fly-builder-delicate-field-7831.internal:2375. Is the docker daemon running?)
Waiting for remote builder fly-builder-delicate-field-7831... connecting ⢿ DEBUG Remote builder unavailable, retrying in 225.12494ms (err: Cannot connect to the Docker daemon at tcp://fly-builder-delicate-field-7831.internal:2375. Is the docker daemon running?)
Waiting for remote builder fly-builder-delicate-field-7831... connecting ⣯

If this is helpful, I was able to unblock myself by spinning up an EC2 instance to deploy stuff, but obviously not as fun/easy as doing stuff locally.

I noticed the Wireguard state stuff, so I removed those peers using fly wireguard remove just in case they were getting in the way, but that didnt seem to help.

8

Sorry for the noise here.

I am not sure if I got to the bottom of the problem but I was able to “unblock” myself by removing the ~/.fly/config.yml which seemed to have some Wireguard information.

My uninformed guess I removed those peers using the CLI at some point so they were stale. So when I ran a fly deploy the next time, it created a new connection that was valid and the deploy went fine.

Hope this helps further debugging and other people running into this issue.

9

Not noise, this is very helpful! Thank you for following up while we work through this.

10

FWIW, this is a bug that we need to fix: Deleting some WireGuard peers permanently confuses the agent · Issue #467 · superfly/flyctl · GitHub

2 Likes