Failure to build Dockerfile remotely

Questions / Help Build debugging
1

Hi,

I am having problems with building a Docker based app remotely.

❯ LOG_LEVEL=debug fly deploy --app <app-name> --remote-only
...
DEBUG cache loaded.
DEBUG config initialized.
DEBUG initialized task manager.
DEBUG skipped querying for new release
DEBUG client initialized.
DEBUG app config loaded from /path/to/fly.toml
==> Verifying app config
--> Verified app config
==> Building image
DEBUG trying remote docker daemon
DEBUG Reporting buildDEBUG --> POST https://api.fly.io/graphql {{"query":"mutation($input: StartSourceBuildInput!) { startSourceBuild(input: $input) { sourceBuild { id } } }","variables":{"input":{"appId":"<app-name>"}}}
}
DEBUG <-- 200 https://api.fly.io/graphql (391.62ms) {"errors":[{"message":"StartSourceBuildInput isn't a defined input type (on $input)","locations":[{"line":1,"column":10}],"path":["mutation"],"extensions":{"code":"variableRequiresValidType","typeName":"StartSourceBuildInput","variableName":"input"}},{"message":"Field 'startSourceBuild' doesn't exist on type 'Mutations'","locations":[{"line":1,"column":44}],"path":["mutation","startSourceBuild"],"extensions":{"code":"undefinedField","typeName":"Mutations","fieldName":"startSourceBuild"}},{"message":"Variable $input is declared by anonymous mutation but not used","locations":[{"line":1,"column":1}],"path":["mutation"],"extensions":{"code":"variableNotUsed","variableName":"input"}}]}
DEBUG Failed storing buildDEBUG Trying 'Buildpacks' strategy
DEBUG no buildpack builder configured, skipping
DEBUG result image:<nil> error:<nil>
DEBUG Trying 'Dockerfile' strategy
DEBUG --> POST https://api.fly.io/graphql {{"query":"mutation($input: EnsureMachineRemoteBuilderInput!) { ensureMachineRemoteBuilder(input: $input) { machine { id state ips { nodes { family kind ip } } }, app { name organization { slug } } } }","variables":{"input":{"appName":"<app-name>","organizationId":null}}}
}
DEBUG <-- 500 https://api.fly.io/graphql (208.24ms) {"errors":[{"message":"An unknown error occured.","extensions":{"code":"SERVER_ERROR"}}],"data":{}}
DEBUG result image:<nil> error:error connecting to docker: An unknown error occured.
Error failed to fetch an image or build from source: error connecting to docker: An unknown error occured.

Is there something wrong with the request? Local build does work.

2

I see this error in our exception tracking. Did this happen the first time you attempted a deploy or on a second or later?

3

Also, will you give it another try?

4

@kurt it didn’t happen the first few times. I will retry and report

5

So trying out fly for the first time, and now stuck on this issue. Any resolution other than to install docker?

> LOG_LEVEL=debug fly deploy --remote-only --verbose
…
DEBUG cache loaded.
DEBUG config initialized.
DEBUG initialized task manager.
DEBUG skipped querying for new release
Update available 0.0.318 -> v0.0.320.
Run "fly version update" to upgrade.
DEBUG client initialized.
DEBUG app config loaded from /Users/gil/Documents/pihole/fly.toml
==> Verifying app config
--> Verified app config
==> Building image
DEBUG trying remote docker daemon
DEBUG Reporting buildDEBUG --> POST https://api.fly.io/graphql {{"query":"mutation($input: StartSourceBuildInput!) { startSourceBuild(input: $input) { sourceBuild { id } } }","variables":{"input":{"appId":"sky-pi"}}}
}
DEBUG <-- 200 https://api.fly.io/graphql (325.91ms) {"errors":[{"message":"StartSourceBuildInput isn't a defined input type (on $input)","locations":[{"line":1,"column":10}],"path":["mutation"],"extensions":{"code":"variableRequiresValidType","typeName":"StartSourceBuildInput","variableName":"input"}},{"message":"Field 'startSourceBuild' doesn't exist on type 'Mutations'","locations":[{"line":1,"column":44}],"path":["mutation","startSourceBuild"],"extensions":{"code":"undefinedField","typeName":"Mutations","fieldName":"startSourceBuild"}},{"message":"Variable $input is declared by anonymous mutation but not used","locations":[{"line":1,"column":1}],"path":["mutation"],"extensions":{"code":"variableNotUsed","variableName":"input"}}]}
DEBUG Failed storing buildDEBUG Trying 'Buildpacks' strategy
DEBUG no buildpack builder configured, skipping
DEBUG result image:<nil> error:<nil>
DEBUG Trying 'Dockerfile' strategy
DEBUG --> POST https://api.fly.io/graphql {{"query":"mutation($input: EnsureMachineRemoteBuilderInput!) { ensureMachineRemoteBuilder(input: $input) { machine { id state ips { nodes { family kind ip } } }, app { name organization { id slug } } } }","variables":{"input":{"appName":"sky-pi","organizationId":null}}}
}
DEBUG <-- 500 https://api.fly.io/graphql (19.34s) {"errors":[{"message":"An unknown error occured.","extensions":{"code":"SERVER_ERROR"}}],"data":{}}
DEBUG result image:<nil> error:error connecting to docker: An unknown error occured.
Error failed to fetch an image or build from source: error connecting to docker: An unknown error occured.
6

So it seems like the issue is not 100% the same, since my last graphql call takes >19s to fail, while original issue failed quickly.

7

Assuming it’s the same issue as this person had (their debug log output certainly looks similar to yours, at the end, with the 500 error), it may be related to the remote builder:

Installing docker locally would certainly avoid needing to use a remote builder. However if you’d prefer to, you could try (if you haven’t already) destroying your current remote builder. A new one will be made automatically on your next deploy. It’s not worth doing that more than once but it may fix random, temporary issues, if that particular one happens to have something wrong. You can see its name using fly apps list and then fly destroy its-name to delete it.

Else it’s not that, clearly, and something Fly needs to investigate.

8

Thanks for the suggestion. Unfortunately, destroying the builder and re-deploying did not change anything.

9

Destroying builders only really helps when their disk is full.

Are you running a firewall or VPN on your system? Some of those interfere with the networking we do. We have a possible workaround for UDP/wireguard issues, try this:

10

No outgoing firewalls or VPNs here. The IPs are 8.8.8.8 and 8.8.4.4 are blocked on my router though.

Anyway, it does not appear to be a connection issue, as I read the logs. Your graphql endpoint is the one reporting an error with a code 500.

A bit more for the diagnostics, it seems that the builder is failing provisioning, and just says Status = pending. Maybe there is an issue with the Frankfurt region?

11

Oh yes that 500 is a good point. We’re investigating, I think your builder is in a weird state. There are some internal errors about it not getting a good IP address.

12

We ended up disabling one of our builder hosts. If you give it another try, you might get a different result. We haven’t identified the bug yet, though, so it’s possible you’ll hit it on different hardware too.

13

Thanks, it is working for me now!