Remote Builder Apps going unresponsive

Remote builders seemingly go into an unresponsive mode after some idle time, and doesn’t seem to come back to life. I’ve been destroying the remote builder apps to be able to do, well, remote builds.

e.g. remote build deploy that hangs:

% LOG_LEVEL=debug flyctl deploy --remote-only --dockerfile Containerfile
DEBUG Loaded flyctl config from${HOME}/.fly/config.yml
DEBUG Checking for updates...
DEBUG Loading app config from${HOME}/${app_path}/fly.toml
DEBUG Working Directory: ${HOME}/${app_path}
DEBUG App Config File: ${HOME}/${app_path}/fly.toml
Deploying bdd
==> Validating app configuration
DEBUG --> POST https://api.fly.io/graphql { [...cut...] }
DEBUG <-- 200 https://api.fly.io/graphql (330.11ms) { [...cut...] }
--> Validating app configuration done
Services
TCP 80/443 ⇢ 8080
DEBUG trying remote docker daemon
DEBUG Trying 'Buildpacks' strategy
DEBUG no buildpack builder configured, skipping
DEBUG result image:<nil> error:<nil>
DEBUG Trying 'Dockerfile' strategy
DEBUG --> POST https://api.fly.io/graphql {{"query":"mutation($input: EnsureMachineRemoteBuilderInput!) { ensureMachineRemoteBuilder(input: $input) { machine { id state ips { nodes { family kind ip } } }, app { name organization { slug } } } }","variables":{"input":{"appName":"bdd","organizationId":null}}}
}
DEBUG <-- 200 https://api.fly.io/graphql (320.85ms) {"data":{"ensureMachineRemoteBuilder":{"machine":{"id":"82dd3b5e","state":"starting","ips":{"nodes":[{"family":"v6","kind":"privatenet","ip":"fdaa:0:847:a7b:ad0:0:4baa:2"},{"family":"v6","kind":"public","ip":"2604:1380:45e1:3002::4baa:3"},{"family":"v4","kind":"private","ip":"172.19.0.58"}]}},"app":{"name":"fly-builder-holy-rain-159","organization":{"slug":"personal"}}}}}
DEBUG checking ip &{Family:v6 Kind:privatenet IP:fdaa:0:847:a7b:ad0:0:4baa:2 MaskSize:0}
Waiting for remote builder fly-builder-holy-rain-159... connecting ⣾ DEBUG --> POST https://api.fly.io/graphql {{"query":"query ($appName: String!) { app(name: $appName) { id name hostname deployed status version appUrl organization { id slug } services { description protocol internalPort ports { port handlers } } ipAddresses { nodes { id address type createdAt } } } }","variables":{"appName":"bdd"}}
}
Waiting for remote builder fly-builder-holy-rain-159... connecting ⣽ DEBUG <-- 200 https://api.fly.io/graphql (359.31ms) {"data":{"app":{"id":"bdd","name":"bdd","hostname":"bdd.fly.dev","deployed":true,"status":"running","version":35,"appUrl":"https://213.188.195.170","organization":{"id":"ZRJPoxp8kwJ0ZuK1KxyBZAnyBoCYG0","slug":"personal"},"services":[{"description":"TCP 80/443 ⇢ 8080","protocol":"TCP","internalPort":8080,"ports":[{"port":80,"handlers":["HTTP"]},{"port":443,"handlers":["TLS","HTTP"]}]}],"ipAddresses":{"nodes":[{"id":"89D45zK5yOwYBTBNA1O09Aebn8sBvA","address":"213.188.195.170","type":"v4","createdAt":"2021-01-10T23:55:39Z"},{"id":"D361ODBOn4xgjT48Kb3VBKkqPzHgV9","address":"2a09:8280:1:715e:4041:f149:6ac0:dc3e","type":"v6","createdAt":"2021-01-10T23:55:41Z"}]}}}}
DEBUG --> POST https://api.fly.io/graphql {{"query":"mutation($input: ValidateWireGuardPeersInput!) { validateWireGuardPeers(input: $input) { invalidPeerIps } }","variables":{"input":{"peerIps":["fdaa:0:847:a7b:e23:0:a:2"]}}}
}
Waiting for remote builder fly-builder-holy-rain-159... connecting ⣻ DEBUG <-- 200 https://api.fly.io/graphql (304.89ms) {"data":{"validateWireGuardPeers":{"invalidPeerIps":[]}}}
DEBUG result image:<nil> error:error connecting to docker: context canceled

I decide to SIGINT above as it wasn’t making any progress.
Then proceeded destroying the builder app.

% flyctl apps destroy fly-builder-holy-rain-159

Try again, and now it succeeds.

% LOG_LEVEL=debug flyctl deploy --remote-only --dockerfile Containerfile
DEBUG Loaded flyctl config from${HOME}/.fly/config.yml
DEBUG Checking for updates...
DEBUG Loading app config from${HOME}/${app_path}/fly.toml
DEBUG Working Directory: ${HOME}/${app_path}
DEBUG App Config File: ${HOME}/${app_path}/fly.toml
Deploying bdd
==> Validating app configuration
DEBUG --> POST https://api.fly.io/graphql { [...cut...] }
DEBUG <-- 200 https://api.fly.io/graphql (339.96ms) { [...cut...]}
--> Validating app configuration done
Services
TCP 80/443 ⇢ 8080
DEBUG trying remote docker daemon
DEBUG Trying 'Buildpacks' strategy
DEBUG no buildpack builder configured, skipping
DEBUG result image:<nil> error:<nil>
DEBUG Trying 'Dockerfile' strategy
DEBUG --> POST https://api.fly.io/graphql {{"query":"mutation($input: EnsureMachineRemoteBuilderInput!) { ensureMachineRemoteBuilder(input: $input) { machine { id state ips { nodes { family kind ip } } }, app { name organization { slug } } } }","variables":{"input":{"appName":"bdd","organizationId":null}}}
}
DEBUG <-- 200 https://api.fly.io/graphql (1.63s) {"data":{"ensureMachineRemoteBuilder":{"machine":{"id":"84ee6a91","state":"starting","ips":{"nodes":[{"family":"v6","kind":"privatenet","ip":"fdaa:0:847:a7b:ad0:0:4c62:2"},{"family":"v6","kind":"public","ip":"2604:1380:45e1:3002::4c62:3"},{"family":"v4","kind":"private","ip":"172.19.1.170"}]}},"app":{"name":"fly-builder-wispy-mountain-4171","organization":{"slug":"personal"}}}}}
DEBUG checking ip &{Family:v6 Kind:privatenet IP:fdaa:0:847:a7b:ad0:0:4c62:2 MaskSize:0}
Waiting for remote builder fly-builder-wispy-mountain-4171... connecting ⣾ DEBUG --> POST https://api.fly.io/graphql {{"query":"query ($appName: String!) { app(name: $appName) { id name hostname deployed status version appUrl organization { id slug } services { description protocol internalPort ports { port handlers } } ipAddresses { nodes { id address type createdAt } } } }","variables":{"appName":"bdd"}}
}
Waiting for remote builder fly-builder-wispy-mountain-4171... connecting ⣽ DEBUG <-- 200 https://api.fly.io/graphql (375.24ms) {"data":{"app":{"id":"bdd","name":"bdd","hostname":"bdd.fly.dev","deployed":true,"status":"running","version":35,"appUrl":"https://213.188.195.170","organization":{"id":"ZRJPoxp8kwJ0ZuK1KxyBZAnyBoCYG0","slug":"personal"},"services":[{"description":"TCP 80/443 ⇢ 8080","protocol":"TCP","internalPort":8080,"ports":[{"port":80,"handlers":["HTTP"]},{"port":443,"handlers":["TLS","HTTP"]}]}],"ipAddresses":{"nodes":[{"id":"89D45zK5yOwYBTBNA1O09Aebn8sBvA","address":"213.188.195.170","type":"v4","createdAt":"2021-01-10T23:55:39Z"},{"id":"D361ODBOn4xgjT48Kb3VBKkqPzHgV9","address":"2a09:8280:1:715e:4041:f149:6ac0:dc3e","type":"v6","createdAt":"2021-01-10T23:55:41Z"}]}}}}
DEBUG --> POST https://api.fly.io/graphql {{"query":"mutation($input: ValidateWireGuardPeersInput!) { validateWireGuardPeers(input: $input) { invalidPeerIps } }","variables":{"input":{"peerIps":["fdaa:0:847:a7b:e23:0:a:2"]}}}
}
Waiting for remote builder fly-builder-wispy-mountain-4171... connecting ⣻ DEBUG <-- 200 https://api.fly.io/graphql (137.95ms) {"data":{"validateWireGuardPeers":{"invalidPeerIps":[]}}}
Waiting for remote builder fly-builder-wispy-mountain-4171... connecting ⣾ DEBUG Remote builder available, but pinging again in 200ms to be sure
Waiting for remote builder fly-builder-wispy-mountain-4171... connecting ⣻ DEBUG Remote builder available, but pinging again in 218.381322ms to be sure
Waiting for remote builder fly-builder-wispy-mountain-4171... connecting ⡿ DEBUG Remote builder is ready to build!
Remote builder fly-builder-wispy-mountain-4171 ready
[...cut...]

Ah this is super helpful. We just uncovered the logs from your failed restart and it seems like there was a problem booting getting the network interface back for the existing builder. We’re debugging.

1 Like

I am also experiencing some issues while deploying

Remote builder fly-builder-bitter-cloud-8683 ready
==> Building image with Buildpacks
--> docker host: 20.10.8 linux x86_64
20: Pulling from heroku/buildpacks
Digest: sha256:c42ef608e159258b9b1570cd011686fd4d3f63a0053662bfd5e45f180a64861b
Status: Image is up to date for heroku/buildpacks:20
20: Pulling from heroku/pack
Digest: sha256:dd3a7e568f9f3251b915707f063592310acda36b98341c60f2e58dcd03a9d4e6
Status: Image is up to date for heroku/pack:20

Error fetching base layers: saving base image with ID "sha256:3361b2975280d5ab215aa3b9d7f336817a7e6b9f79ec8ec7968c66fe15a9e432" from the docker daemon: error during connect: Get "http://[fdaa:0:3758:a7b:14bf:0:4ce9:2]:2375/v1.41/images/get?names=sha256%3A3361b2975280d5ab215aa3b9d7f336817a7e6b9f79ec8ec7968c66fe15a9e432": err err handling connect: connection failed: connect tcp [fdaa:0:3758:a7b:14bf:0:4ce9:2]:2375: connection was refused

I’m having the same “waiting for remote builder” problem. My builder is fly-builder-broken-smoke-5499. Even fly apps destroy fly-builder-broken-smoke-5499 fails, so I’m not able to deploy at all.

Based on advice in another thread, I tried fly wireguard remove, but that didn’t help. Deploying still blocks indefinitely, as does destroying the builder app.

@arthurgleckler @vinay_puppal We’re looking into both problems, will get back with an update soon.

I discovered a workaround. While destroying the builder app through flyctl didn’t work, there is a way to do it through the web UI. Once that was done, I was able to deploy again.

1 Like

Thanks for the update, glad to know you’re not stuck. We’re analysing the logs and we’ll try to make sure the problem doesn’t come up again.

1 Like