Remote builders seemingly go into an unresponsive mode after some idle time, and doesn’t seem to come back to life. I’ve been destroying the remote builder apps to be able to do, well, remote builds.
e.g. remote build deploy that hangs:
% LOG_LEVEL=debug flyctl deploy --remote-only --dockerfile Containerfile
DEBUG Loaded flyctl config from${HOME}/.fly/config.yml
DEBUG Checking for updates...
DEBUG Loading app config from${HOME}/${app_path}/fly.toml
DEBUG Working Directory: ${HOME}/${app_path}
DEBUG App Config File: ${HOME}/${app_path}/fly.toml
Deploying bdd
==> Validating app configuration
DEBUG --> POST https://api.fly.io/graphql { [...cut...] }
DEBUG <-- 200 https://api.fly.io/graphql (330.11ms) { [...cut...] }
--> Validating app configuration done
Services
TCP 80/443 ⇢ 8080
DEBUG trying remote docker daemon
DEBUG Trying 'Buildpacks' strategy
DEBUG no buildpack builder configured, skipping
DEBUG result image:<nil> error:<nil>
DEBUG Trying 'Dockerfile' strategy
DEBUG --> POST https://api.fly.io/graphql {{"query":"mutation($input: EnsureMachineRemoteBuilderInput!) { ensureMachineRemoteBuilder(input: $input) { machine { id state ips { nodes { family kind ip } } }, app { name organization { slug } } } }","variables":{"input":{"appName":"bdd","organizationId":null}}}
}
DEBUG <-- 200 https://api.fly.io/graphql (320.85ms) {"data":{"ensureMachineRemoteBuilder":{"machine":{"id":"82dd3b5e","state":"starting","ips":{"nodes":[{"family":"v6","kind":"privatenet","ip":"fdaa:0:847:a7b:ad0:0:4baa:2"},{"family":"v6","kind":"public","ip":"2604:1380:45e1:3002::4baa:3"},{"family":"v4","kind":"private","ip":"172.19.0.58"}]}},"app":{"name":"fly-builder-holy-rain-159","organization":{"slug":"personal"}}}}}
DEBUG checking ip &{Family:v6 Kind:privatenet IP:fdaa:0:847:a7b:ad0:0:4baa:2 MaskSize:0}
Waiting for remote builder fly-builder-holy-rain-159... connecting ⣾ DEBUG --> POST https://api.fly.io/graphql {{"query":"query ($appName: String!) { app(name: $appName) { id name hostname deployed status version appUrl organization { id slug } services { description protocol internalPort ports { port handlers } } ipAddresses { nodes { id address type createdAt } } } }","variables":{"appName":"bdd"}}
}
Waiting for remote builder fly-builder-holy-rain-159... connecting ⣽ DEBUG <-- 200 https://api.fly.io/graphql (359.31ms) {"data":{"app":{"id":"bdd","name":"bdd","hostname":"bdd.fly.dev","deployed":true,"status":"running","version":35,"appUrl":"https://213.188.195.170","organization":{"id":"ZRJPoxp8kwJ0ZuK1KxyBZAnyBoCYG0","slug":"personal"},"services":[{"description":"TCP 80/443 ⇢ 8080","protocol":"TCP","internalPort":8080,"ports":[{"port":80,"handlers":["HTTP"]},{"port":443,"handlers":["TLS","HTTP"]}]}],"ipAddresses":{"nodes":[{"id":"89D45zK5yOwYBTBNA1O09Aebn8sBvA","address":"213.188.195.170","type":"v4","createdAt":"2021-01-10T23:55:39Z"},{"id":"D361ODBOn4xgjT48Kb3VBKkqPzHgV9","address":"2a09:8280:1:715e:4041:f149:6ac0:dc3e","type":"v6","createdAt":"2021-01-10T23:55:41Z"}]}}}}
DEBUG --> POST https://api.fly.io/graphql {{"query":"mutation($input: ValidateWireGuardPeersInput!) { validateWireGuardPeers(input: $input) { invalidPeerIps } }","variables":{"input":{"peerIps":["fdaa:0:847:a7b:e23:0:a:2"]}}}
}
Waiting for remote builder fly-builder-holy-rain-159... connecting ⣻ DEBUG <-- 200 https://api.fly.io/graphql (304.89ms) {"data":{"validateWireGuardPeers":{"invalidPeerIps":[]}}}
DEBUG result image:<nil> error:error connecting to docker: context canceled
I decide to SIGINT above as it wasn’t making any progress.
Then proceeded destroying the builder app.
% flyctl apps destroy fly-builder-holy-rain-159
Try again, and now it succeeds.
% LOG_LEVEL=debug flyctl deploy --remote-only --dockerfile Containerfile
DEBUG Loaded flyctl config from${HOME}/.fly/config.yml
DEBUG Checking for updates...
DEBUG Loading app config from${HOME}/${app_path}/fly.toml
DEBUG Working Directory: ${HOME}/${app_path}
DEBUG App Config File: ${HOME}/${app_path}/fly.toml
Deploying bdd
==> Validating app configuration
DEBUG --> POST https://api.fly.io/graphql { [...cut...] }
DEBUG <-- 200 https://api.fly.io/graphql (339.96ms) { [...cut...]}
--> Validating app configuration done
Services
TCP 80/443 ⇢ 8080
DEBUG trying remote docker daemon
DEBUG Trying 'Buildpacks' strategy
DEBUG no buildpack builder configured, skipping
DEBUG result image:<nil> error:<nil>
DEBUG Trying 'Dockerfile' strategy
DEBUG --> POST https://api.fly.io/graphql {{"query":"mutation($input: EnsureMachineRemoteBuilderInput!) { ensureMachineRemoteBuilder(input: $input) { machine { id state ips { nodes { family kind ip } } }, app { name organization { slug } } } }","variables":{"input":{"appName":"bdd","organizationId":null}}}
}
DEBUG <-- 200 https://api.fly.io/graphql (1.63s) {"data":{"ensureMachineRemoteBuilder":{"machine":{"id":"84ee6a91","state":"starting","ips":{"nodes":[{"family":"v6","kind":"privatenet","ip":"fdaa:0:847:a7b:ad0:0:4c62:2"},{"family":"v6","kind":"public","ip":"2604:1380:45e1:3002::4c62:3"},{"family":"v4","kind":"private","ip":"172.19.1.170"}]}},"app":{"name":"fly-builder-wispy-mountain-4171","organization":{"slug":"personal"}}}}}
DEBUG checking ip &{Family:v6 Kind:privatenet IP:fdaa:0:847:a7b:ad0:0:4c62:2 MaskSize:0}
Waiting for remote builder fly-builder-wispy-mountain-4171... connecting ⣾ DEBUG --> POST https://api.fly.io/graphql {{"query":"query ($appName: String!) { app(name: $appName) { id name hostname deployed status version appUrl organization { id slug } services { description protocol internalPort ports { port handlers } } ipAddresses { nodes { id address type createdAt } } } }","variables":{"appName":"bdd"}}
}
Waiting for remote builder fly-builder-wispy-mountain-4171... connecting ⣽ DEBUG <-- 200 https://api.fly.io/graphql (375.24ms) {"data":{"app":{"id":"bdd","name":"bdd","hostname":"bdd.fly.dev","deployed":true,"status":"running","version":35,"appUrl":"https://213.188.195.170","organization":{"id":"ZRJPoxp8kwJ0ZuK1KxyBZAnyBoCYG0","slug":"personal"},"services":[{"description":"TCP 80/443 ⇢ 8080","protocol":"TCP","internalPort":8080,"ports":[{"port":80,"handlers":["HTTP"]},{"port":443,"handlers":["TLS","HTTP"]}]}],"ipAddresses":{"nodes":[{"id":"89D45zK5yOwYBTBNA1O09Aebn8sBvA","address":"213.188.195.170","type":"v4","createdAt":"2021-01-10T23:55:39Z"},{"id":"D361ODBOn4xgjT48Kb3VBKkqPzHgV9","address":"2a09:8280:1:715e:4041:f149:6ac0:dc3e","type":"v6","createdAt":"2021-01-10T23:55:41Z"}]}}}}
DEBUG --> POST https://api.fly.io/graphql {{"query":"mutation($input: ValidateWireGuardPeersInput!) { validateWireGuardPeers(input: $input) { invalidPeerIps } }","variables":{"input":{"peerIps":["fdaa:0:847:a7b:e23:0:a:2"]}}}
}
Waiting for remote builder fly-builder-wispy-mountain-4171... connecting ⣻ DEBUG <-- 200 https://api.fly.io/graphql (137.95ms) {"data":{"validateWireGuardPeers":{"invalidPeerIps":[]}}}
Waiting for remote builder fly-builder-wispy-mountain-4171... connecting ⣾ DEBUG Remote builder available, but pinging again in 200ms to be sure
Waiting for remote builder fly-builder-wispy-mountain-4171... connecting ⣻ DEBUG Remote builder available, but pinging again in 218.381322ms to be sure
Waiting for remote builder fly-builder-wispy-mountain-4171... connecting ⡿ DEBUG Remote builder is ready to build!
Remote builder fly-builder-wispy-mountain-4171 ready
[...cut...]
