Saw some other posts about remote builders having issues and it sounds like some bugs have been swatted there, but I haven’t yet been able to get a remote build to proceed past the connecting phase. Some logs:
LOG_LEVEL=debug fly agent daemon-start
DEBUG Loaded flyctl config from/home/user/.fly/config.yml
DEBUG determined hostname: "mgmt"
DEBUG determined working directory: "/home/user"
DEBUG determined user home directory: "/home/user"
DEBUG determined config directory: "/home/user/.fly"
DEBUG ensured config directory exists.
DEBUG ensured config directory perms.
DEBUG cache loaded.
DEBUG config initialized.
DEBUG initialized task manager.
DEBUG skipped querying for new release
DEBUG client initialized.
2022/01/29 09:43:36.715478 srv OK 15783
DEBUG --> POST https://api.fly.io/graphql {{"query":"mutation($input: ValidateWireGuardPeersInput!) { validat$
WireGuardPeers(input: $input) { invalidPeerIps } }","variables":{"input":{"peerIps":["fdaa:0:48a8:a7b:1537:0:$
:102"]}}}
}
DEBUG <-- 200 https://api.fly.io/graphql (243.81ms) {"data":{"validateWireGuardPeers":{"invalidPeerIps":[]}}}
2022/01/29 09:45:37.030996 srv validated wireguard peers
2022/01/29 09:46:39.599999 #1 connected ...
2022/01/29 09:46:39.600127 srv config change at: 2022-01-29 09:46:39.598207552 -0800 PST
2022/01/29 09:46:39.600149 #1 <- ( 4) "ping"
2022/01/29 09:46:39.600215 #1 -> ( 58) "8\x00ok {\"PID\":15783,\"Version\":\"0.0.292\",\"Background\":false$
\n"
2022/01/29 09:46:39.600231 #1 dropped.
2022/01/29 09:46:39.600425 #2 connected ...
2022/01/29 09:46:39.600460 #2 <- ( 18) "establish personal"
DEBUG --> POST https://api.fly.io/graphql {{"query":"query($orgType: OrganizationType) { organizations(type: $
orgType) { nodes { id slug name type } } }"
...
2022/01/29 09:54:57.583848 #14 dropped.
2022/01/29 09:54:57.584492 #15 connected ...
2022/01/29 09:54:57.584747 #15 <- ( 53) "connect personal [fdaa:0:48a8:a7b:87:0:847e:2]:2375 0"
DEBUG --> POST https://api.fly.io/graphql {{"query":"mutation($input: ValidateWireGuardPeersInput!) { validateWireGuardPeers(input: $input) { invalidPeerIps } }","variables":{"input":{"peerIps":["fdaa:0:48a8:a7b:1537:0:a:102"]}}}
}
DEBUG <-- 200 https://api.fly.io/graphql (187.55ms) {"data":{"validateWireGuardPeers":{"invalidPeerIps":[]}}}
2022/01/29 09:55:38.343591 srv validated wireguard peers
$ LOG_LEVEL=debug fly deploy --remote-only
DEBUG client initialized.
DEBUG app config loaded from /home/user/flyio/fly.toml
==> Verifying app config
--> Verified app config
==> Building image
DEBUG trying remote docker daemon
DEBUG Reporting buildDEBUG --> POST https://api.fly.io/graphql {{"query":"mutation($input: StartSourceBuildInput!) { startSourceBuild(input: $input) { sourceBuild { id } } }","variables":{"input":{"appId":"REDACTED"}}}
}
DEBUG <-- 200 https://api.fly.io/graphql (526.89ms) {"errors":[{"message":"StartSourceBuildInput isn't a defined input type (on $input)","locations":[{"line":1,"column":10}],"path":["mutation"],"extensions":{"code":"variableRequiresValidType","typeName":"StartSourceBuildInput","variableName":"input"}},{"message":"Field 'startSourceBuild' doesn't exist on type 'Mutations'","locations":[{"line":1,"column":44}],"path":["mutation","startSourceBuild"],"extensions":{"code":"undefinedField","typeName":"Mutations","fieldName":"startSourceBuild"}},{"message":"Variable $input is declared by anonymous mutation but not used","locations":[{"line":1,"column":1}],"path":["mutation"],"extensions":{"code":"variableNotUsed","variableName":"input"}}]}
DEBUG Failed storing buildDEBUG Trying 'Buildpacks' strategy
DEBUG no buildpack builder configured, skipping
DEBUG result image:<nil> error:<nil>
DEBUG Trying 'Dockerfile' strategy
DEBUG --> POST https://api.fly.io/graphql {{"query":"mutation($input: EnsureMachineRemoteBuilderInput!) { ensureMachineRemoteBuilder(input: $input) { machine { id state ips { nodes { family kind ip } } }, app { name organization { slug } } } }","variables":{"input":{"appName":"REDACTED","organizationId":null}}}
}
DEBUG <-- 200 https://api.fly.io/graphql (194.2ms) {"data":{"ensureMachineRemoteBuilder":{"machine":{"id":"REDACTED","state":"starting","ips":{"nodes":[{"family":"v6","kind":"privatenet","ip":"fdaa:0:48a8:a7b:87:0:847e:2"},{"family":"v6","kind":"public","ip":"2604:1380:4510:401::847e:3"},{"family":"v4","kind":"private","ip":"172.19.7.26"}]}},"app":{"name":"fly-builder-REDACTED-9008","organization":{"slug":"personal"}}}}}
DEBUG checking ip &{Family:v6 Kind:privatenet IP:fdaa:0:48a8:a7b:87:0:847e:2 MaskSize:0}
Waiting for remote builder fly-builder-REDACTED-9008... connecting ⣾ DEBUG --> POST https://api.fly.io/graphql {{"query":"query ($appName: String!) { app(name: $appName) { id name hostname deployed status version appUrl config { definition } organization { id slug } services { description protocol internalPort ports { port handlers } } ipAddresses { nodes { id address type createdAt } } imageDetails { repository version } } }","variables":{"appName":"REDACTED"}}
}
Waiting for remote builder fly-builder-REDACTED-9008... connecting
...
DEBUG --> POST https://api.fly.io/graphql {{"query":"mutation($input: ValidateWireGuardPeersInput!) { validateWireGuardPeers(input: $input) { invalidPeerIps } }","variables":{"input":{"peerIps":["fdaa:0:48a8:a7b:1537:0:a:102"]}}}
}
Waiting for remote builder fly-builder-REDACTED-9008... connecting ⡿ DEBUG <-- 200 https://api.fly.io/graphql (241.12ms) {"data":{"validateWireGuardPeers":{"invalidPeerIps":[]}}}
...
Waiting for remote builder fly-builder-REDACTED-9008... connecting ⣻ DEBUG Remote builder unavailable, retrying in 200ms (err: error during connect: Get "http://[fdaa:0:48a8:a7b:87:0:847e:2]:2375/_ping": connect tcp [fdaa:0:48a8:a7b:87:0:847e:2]:2375: operation timed out)
WARN Remote builder did not start on time. Check remote builder logs with `flyctl logs -a fly-builder-REDACTED-9008`
DEBUG result image:<nil> error:error connecting to docker: remote builder app unavailable
Error failed to fetch an image or build from source: error connecting to docker: remote builder app unavailable
Status of wireguard tunnels:
$ flyctl wireguard list
Automatically selected personal organization: REDACTED
+-----------------------------------+--------+------------------------------+
| NAME | REGION | PEER IP |
+-----------------------------------+--------+------------------------------+
| interactive-linux--REDACTED-428 | lax | fdaa:0:48a8:xxx:xxxx:0:a:102 |
| interactive-linux--REDACTED-572 | lax | fdaa:0:48a8:xxx:xxxx:0:a:2 |
+-----------------------------------+--------+------------------------------+
Have tried deleting fly/config.yml, re-auth’ing and starting from scratch, but no luck. Happy to troubleshoot further.
