Deployments not working: error connecting to docker

Hello! Please help me solve the problem, I can’t complete the deploy.

% fly deploy
==> Verifying app config
→ Verified app config
==> Building image
Error failed to fetch an image or build from source: error connecting to docker: failed building options: failed probing “personal”: context deadline exceeded

% fly deploy --remote-only
==> Verifying app config
→ Verified app config
==> Building image
Error failed to fetch an image or build from source: error connecting to docker: failed building options: failed probing “personal”: context deadline exceeded

A couple of thoughts:

Maybe try:
fly auth login
… to double-check you are authenticated, to rule that out as an issue.

Then I think you can add a log level so show some more details. I suspect Fly may need those extra debug lines to try and see what’s happening:

LOG_LEVEL=debug fly deploy

If you can paste in that output (removing anything you don’t want revealed, like your app name) here, that will help diagnose it.

3 Likes

Same. Receiving a 502 Bad Gateway code when deploying. fly auth login also returns “An unknown server error occured.”

Meanwhile fly status hangs and the finally returns a 502.

fly auth login also returns “An unknown server error occured.”

LOG_LEVEL=debug fly deploy
DEBUG Loaded flyctl config from/Users/Kto_tut/.fly/config.yml
DEBUG determined hostname: “MacBook-Pro-Victoriya.local”
DEBUG determined working directory: “/Users/Kto_tut/Desktop/thirdweb-lootbox-tutorial-main”
DEBUG determined user home directory: “/Users/Kto_tut”
DEBUG determined config directory: “/Users/Kto_tut/.fly”
DEBUG ensured config directory exists.
DEBUG ensured config directory perms.
DEBUG cache loaded.
DEBUG config initialized.
DEBUG initialized task manager.
DEBUG started querying for new release
DEBUG client initialized.
DEBUG app config loaded from /Users/Kto_tut/Desktop/thirdweb-lootbox-tutorial-main/fly.toml
==> Verifying app config
→ Verified app config
==> Building image
DEBUG trying local docker daemon
DEBUG Local docker daemon unavailable
DEBUG trying remote docker daemon
DEBUG Reporting buildDEBUG → POST GraphQL Playground {{“query”:“mutation($input: StartSourceBuildInput!) { startSourceBuild(input: $input) { sourceBuild { id } } }”,“variables”:{“input”:{“appId”:“lootbox-quiz-alex”}}}
}
DEBUG querying for release resulted to v0.0.295
DEBUG ← 200 GraphQL Playground (481.98ms) {“errors”:[{“message”:“StartSourceBuildInput isn’t a defined input type (on $input)”,“locations”:[{“line”:1,“column”:10}],“path”:[“mutation”],“extensions”:{“code”:“variableRequiresValidType”,“typeName”:“StartSourceBuildInput”,“variableName”:“input”}},{“message”:“Field ‘startSourceBuild’ doesn’t exist on type ‘Mutations’”,“locations”:[{“line”:1,“column”:44}],“path”:[“mutation”,“startSourceBuild”],“extensions”:{“code”:“undefinedField”,“typeName”:“Mutations”,“fieldName”:“startSourceBuild”}},{“message”:“Variable $input is declared by anonymous mutation but not used”,“locations”:[{“line”:1,“column”:1}],“path”:[“mutation”],“extensions”:{“code”:“variableNotUsed”,“variableName”:“input”}}]}
DEBUG Failed storing buildDEBUG Trying ‘Buildpacks’ strategy
DEBUG no buildpack builder configured, skipping
DEBUG result image: error:
DEBUG Trying ‘Dockerfile’ strategy
DEBUG → POST GraphQL Playground {{“query”:“mutation($input: EnsureMachineRemoteBuilderInput!) { ensureMachineRemoteBuilder(input: $input) { machine { id state ips { nodes { family kind ip } } }, app { name organization { slug } } } }”,“variables”:{“input”:{“appName”:“lootbox-quiz-alex”,“organizationId”:null}}}
}
DEBUG ← 200 GraphQL Playground (259.83ms) {“data”:{“ensureMachineRemoteBuilder”:{“machine”:{“id”:“1579a9d1”,“state”:“starting”,“ips”:{“nodes”:[{“family”:“v6”,“kind”:“privatenet”,“ip”:“fdaa:0:4b73:a7b:23c3:0:8ca3:2”},{“family”:“v6”,“kind”:“public”,“ip”:“2604:1380:4091:3602::8ca3:3”},{“family”:“v4”,“kind”:“private”,“ip”:“172.19.11.82”}]}},“app”:{“name”:“fly-builder-small-sky-7039”,“organization”:{“slug”:“personal”}}}}}
DEBUG checking ip &{Family:v6 Kind:privatenet IP:fdaa:0:4b73:a7b:23c3:0:8ca3:2 MaskSize:0}
Waiting for remote builder fly-builder-small-sky-7039… ⣾ DEBUG → POST GraphQL Playground {{“query”:“query ($appName: String!) { app(name: $appName) { id name hostname deployed status version appUrl config { definition } organization { id slug } services { description protocol internalPort ports { port handlers } } ipAddresses { nodes { id address type createdAt } } imageDetails { repository version } } }”,“variables”:{“appName”:“lootbox-quiz-alex”}}
}
Waiting for remote builder fly-builder-small-sky-7039… ⣽ DEBUG ← 200 GraphQL Playground (408.68ms) {“data”:{“app”:{“id”:“lootbox-quiz-alex”,“name”:“lootbox-quiz-alex”,“hostname”:“lootbox-quiz-alex.fly.dev”,“deployed”:false,“status”:“pending”,“version”:0,“appUrl”:null,“config”:{“definition”:{“kill_timeout”:5,“kill_signal”:“SIGINT”,“processes”:,“experimental”:{“allowed_public_ports”:,“auto_rollback”:true},“services”:[{“processes”:[“app”],“protocol”:“tcp”,“internal_port”:8080,“concurrency”:{“soft_limit”:20,“hard_limit”:25,“type”:“connections”},“ports”:[{“port”:80,“handlers”:[“http”]},{“port”:443,“handlers”:[“tls”,“http”]}],“tcp_checks”:[{“interval”:“15s”,“timeout”:“2s”,“grace_period”:“1s”,“restart_limit”:0}],“http_checks”:,“script_checks”:}],“env”:{}}},“organization”:{“id”:“AP276N9M5j2qJTaRa50QLKp0Q8Fx1l”,“slug”:“personal”},“services”:[{“description”:“TCP 80/443 ⇢ 8080”,“protocol”:“TCP”,“internalPort”:8080,“ports”:[{“port”:80,“handlers”:[“HTTP”]},{“port”:443,“handlers”:[“TLS”,“HTTP”]}]}],“ipAddresses”:{“nodes”:},“imageDetails”:{“repository”:“unknown”,“version”:“unknown”}}}}
DEBUG → POST GraphQL Playground {{“query”:“mutation($input: ValidateWireGuardPeersInput!) { validateWireGuardPeers(input: $input) { invalidPeerIps } }”,“variables”:{“input”:{“peerIps”:[“fdaa:0:4b73:a7b:16a9:0:a:2”]}}}
}
Waiting for remote builder fly-builder-small-sky-7039… ⣻ DEBUG ← 200 GraphQL Playground (160.18ms) {“data”:{“validateWireGuardPeers”:{“invalidPeerIps”:}}}
DEBUG result image: error:error connecting to docker: failed building options: failed probing “personal”: context deadline exceeded
Error failed to fetch an image or build from source: error connecting to docker: failed building options: failed probing “personal”: context deadline exceeded

Hmm that invalidPeerIps does not sound good.

I suspect it may be related to this. Sounds like Fly is on the case:

@Alexander_Izmailov @greg we are indeed looking into it.

That’s just a call to invalidate bad wireguard peers.

The issue appears to be related to wireguard here.

@Alexander_Izmailov can you try these steps?

fly agent stop
LOG_LEVEL=debug fly agent daemon-start

and in a different terminal

LOG_LEVEL=debug fly deploy

This should give us a lot more information about what’s going on. You should get plenty of logs from both the agent and the deploy process.

I think something is happening, but very slowly, does not allow to post large messages here.

% fly agent stop
% LOG_LEVEL=debug fly agent daemon-start
DEBUG Loaded flyctl config from/Users/Kto_tut/.fly/config.yml
DEBUG determined hostname: “MacBook-Pro-Victoriya.local”
DEBUG determined working directory: “/Users/Kto_tut/Desktop/thirdweb-lootbox-tutorial-main”
DEBUG determined user home directory: “/Users/Kto_tut”
DEBUG determined config directory: “/Users/Kto_tut/.fly”
DEBUG ensured config directory exists.
DEBUG ensured config directory perms.
DEBUG cache loaded.
DEBUG config initialized.
DEBUG initialized task manager.
DEBUG started querying for new release
DEBUG client initialized.
2022/02/08 19:51:57.898574 srv OK 25607
DEBUG querying for release resulted to v0.0.295
2022/02/08 19:53:02.226651 #1 connected …
2022/02/08 19:53:02.226924 srv config change at: 2022-02-08 19:53:02.226349296 +0300 MSK
2022/02/08 19:53:02.226948 #1 ← ( 4) “ping”
2022/02/08 19:53:02.227040 #1 → ( 58) “8\x00ok {“PID”:25607,“Version”:“0.0.295”,“Background”:false}\n”
2022/02/08 19:53:02.227063 #1 dropped.
2022/02/08 19:53:02.227303 #2 connected …
2022/02/08 19:53:02.227356 #2 ← ( 18) “establish personal”
DEBUG → POST GraphQL Playground {{“query”:“query($orgType: OrganizationType) { organizations(type: $orgType) { nodes { id slug name type } } }”,“variables”:null}
}
DEBUG ← 200 GraphQL Playground (650.4ms) {“data”:{“organizations”:{“nodes”:[{“id”:“AP276N9M5j2qJTaRa50QLKp0Q8Fx1l”,“slug”:“personal”,“name”:“Alexander Izmailov”,“type”:“PERSONAL”}]}}}
wg connect fdaa:0:4b73::3 ams1.gateway.6pn.dev:51820 fdaa:0:4b73:a7b:16a9:0:a:0 fdaa:0:4b73::
2022/02/08 19:53:03.095099 #2 → ( 729) “\xd7\x02ok {“WireGuardState”:{“org”:“personal”,“name”:“interactive-MacBook-Pro-Victoriya-izmailov-shura-gmail-com-805”,“region”:“ams”,“localprivate”:“CHwtc7Ax/FYMioOnSwPjINtMGPQ4lZTjp2MXT6mJ+UY=”,“localpublic”:”+sFSGa8LxY1REZZyRZRViJN27nK4DkGLZRhrg3Jr5tA=",“dns”:"",“peer”:{“peerip”:“fdaa:0:4b73:a7b:16a9:0:a:2”,“endpointip”:“ams1.gateway.6pn.dev”,“pubkey”:“yUyg63j5+17YeJ7gRhxoQuF6rvdX0JF59M6skytJFTQ=”}},“TunnelConfig”:{“LocalPrivateKey”:"+sFSGa8LxY1REZZyRZRViJN27nK4DkGLZRhrg3Jr5tA=",“LocalNetwork”:“fdaa:0:4b73:a7b:16a9:0:a:0/120”,“RemotePublicKey”:“yUyg63j5+17YeJ7gRhxoQuF6rvdX0JF59M6skytJFTQ=”,“RemoteNetwork”:“fdaa:0:4b73::/48”,“Endpoint”:“ams1.gateway.6pn.dev:51820”,“DNS”:“fdaa:0:4b73::3”,“KeepAlive”:0,“MTU”:0,“LogLevel”:0}}\n"
2022/02/08 19:53:03.095148 #2 dropped.
2022/02/08 19:53:03.095547 #3 connected …
2022/02/08 19:53:03.095619 #3 ← ( 14) “probe personal”
2022/02/08 19:53:03.095627 srv probing “personal” …
2022/02/08 19:53:08.096783 #3 → ( 58) “8\x00err failed probing “personal”: context deadline exceeded”
2022/02/08 19:53:08.096885 #3 dropped.
DEBUG → POST GraphQL Playground {{“query”:“mutation($input: ValidateWireGuardPeersInput!) { validateWireGuardPeers(input: $input) { invalidPeerIps } }”,“variables”:{“input”:{“peerIps”:[“fdaa:0:4b73:a7b:16a9:0:a:2”]}}}
}
DEBUG ← 200 GraphQL Playground (176.99ms) {“data”:{“validateWireGuardPeers”:{“invalidPeerIps”:}}}
2022/02/08 19:53:58.139716 srv validated wireguard peers
DEBUG → POST GraphQL Playground {{“query”:“mutation($input: ValidateWireGuardPeersInput!) { validateWireGuardPeers(input: $input) { invalidPeerIps } }”,“variables”:{“input”:{“peerIps”:[“fdaa:0:4b73:a7b:16a9:0:a:2”]}}}
}
DEBUG ← 200 GraphQL Playground (394.8ms) {“data”:{“validateWireGuardPeers”:{“invalidPeerIps”:}}}
2022/02/08 19:55:58.597100 srv validated wireguard peers
DEBUG → POST GraphQL Playground {{“query”:“mutation($input: ValidateWireGuardPeersInput!) { validateWireGuardPeers(input: $input) { invalidPeerIps } }”,“variables”:{“input”:{“peerIps”:[“fdaa:0:4b73:a7b:16a9:0:a:2”]}}}
}
DEBUG ← 200 GraphQL Playground (411.8ms) {“data”:{“validateWireGuardPeers”:{“invalidPeerIps”:}}}
2022/02/08 19:57:59.072247 srv validated wireguard peers
DEBUG → POST GraphQL Playground {{“query”:“mutation($input: ValidateWireGuardPeersInput!) { validateWireGuardPeers(input: $input) { invalidPeerIps } }”,“variables”:{“input”:{“peerIps”:[“fdaa:0:4b73:a7b:16a9:0:a:2”]}}}
}
DEBUG ← 200 GraphQL Playground (425.38ms) {“data”:{“validateWireGuardPeers”:{“invalidPeerIps”:}}}
2022/02/08 19:59:59.568066 srv validated wireguard peers
DEBUG → POST GraphQL Playground {{“query”:“mutation($input: ValidateWireGuardPeersInput!) { validateWireGuardPeers(input: $input) { invalidPeerIps } }”,“variables”:{“input”:{“peerIps”:[“fdaa:0:4b73:a7b:16a9:0:a:2”]}}}
}
DEBUG ← 200 GraphQL Playground (314.4ms) {“data”:{“validateWireGuardPeers”:{“invalidPeerIps”:}}}
2022/02/08 20:01:59.947169 srv validated wireguard peers

LOG_LEVEL=debug fly deploy

% LOG_LEVEL=debug fly deploy
DEBUG Loaded flyctl config from/Users/Kto_tut/.fly/config.yml
DEBUG determined hostname: “MacBook-Pro-Victoriya.local”
DEBUG determined working directory: “/Users/Kto_tut/Desktop/thirdweb-lootbox-tutorial-main”
DEBUG determined user home directory: “/Users/Kto_tut”
DEBUG determined config directory: “/Users/Kto_tut/.fly”
DEBUG ensured config directory exists.
DEBUG ensured config directory perms.
DEBUG cache loaded.
DEBUG config initialized.
DEBUG initialized task manager.
DEBUG started querying for new release
DEBUG client initialized.
DEBUG app config loaded from /Users/Kto_tut/Desktop/thirdweb-lootbox-tutorial-main/fly.toml
==> Verifying app config
→ Verified app config
==> Building image
DEBUG trying local docker daemon
DEBUG Local docker daemon unavailable
DEBUG trying remote docker daemon
DEBUG Reporting buildDEBUG → POST GraphQL Playground {{“query”:“mutation($input: StartSourceBuildInput!) { startSourceBuild(input: $input) { sourceBuild { id } } }”,“variables”:{“input”:{“appId”:“lootbox-quiz-alex”}}}
}
DEBUG ← 200 GraphQL Playground (995.29ms) {“errors”:[{“message”:“StartSourceBuildInput isn’t a defined input type (on $input)”,“locations”:[{“line”:1,“column”:10}],“path”:[“mutation”],“extensions”:{“code”:“variableRequiresValidType”,“typeName”:“StartSourceBuildInput”,“variableName”:“input”}},{“message”:“Field ‘startSourceBuild’ doesn’t exist on type ‘Mutations’”,“locations”:[{“line”:1,“column”:44}],“path”:[“mutation”,“startSourceBuild”],“extensions”:{“code”:“undefinedField”,“typeName”:“Mutations”,“fieldName”:“startSourceBuild”}},{“message”:“Variable $input is declared by anonymous mutation but not used”,“locations”:[{“line”:1,“column”:1}],“path”:[“mutation”],“extensions”:{“code”:“variableNotUsed”,“variableName”:“input”}}]}
DEBUG Failed storing buildDEBUG Trying ‘Buildpacks’ strategy
DEBUG no buildpack builder configured, skipping
DEBUG result image: error:
DEBUG Trying ‘Dockerfile’ strategy
DEBUG → POST GraphQL Playground {{“query”:“mutation($input: EnsureMachineRemoteBuilderInput!) { ensureMachineRemoteBuilder(input: $input) { machine { id state ips { nodes { family kind ip } } }, app { name organization { slug } } } }”,“variables”:{“input”:{“appName”:“lootbox-quiz-alex”,“organizationId”:null}}}
}
DEBUG ← 200 GraphQL Playground (262.15ms) {“data”:{“ensureMachineRemoteBuilder”:{“machine”:{“id”:“1579a9d1”,“state”:“starting”,“ips”:{“nodes”:[{“family”:“v6”,“kind”:“privatenet”,“ip”:“fdaa:0:4b73:a7b:23c3:0:8ca3:2”},{“family”:“v6”,“kind”:“public”,“ip”:“2604:1380:4091:3602::8ca3:3”},{“family”:“v4”,“kind”:“private”,“ip”:“172.19.11.82”}]}},“app”:{“name”:“fly-builder-small-sky-7039”,“organization”:{“slug”:“personal”}}}}}
DEBUG checking ip &{Family:v6 Kind:privatenet IP:fdaa:0:4b73:a7b:23c3:0:8ca3:2 MaskSize:0}
Waiting for remote builder fly-builder-small-sky-7039… ⣾ DEBUG → POST GraphQL Playground {{“query”:“query ($appName: String!) { app(name: $appName) { id name hostname deployed status version appUrl config { definition } organization { id slug } services { description protocol internalPort ports { port handlers } } ipAddresses { nodes { id address type createdAt } } imageDetails { repository version } } }”,“variables”:{“appName”:“lootbox-quiz-alex”}}
}
Waiting for remote builder fly-builder-small-sky-7039… ⢿ DEBUG ← 200 GraphQL Playground (918.95ms) {“data”:{“app”:{“id”:“lootbox-quiz-alex”,“name”:“lootbox-quiz-alex”,“hostname”:“lootbox-quiz-alex.fly.dev”,“deployed”:false,“status”:“pending”,“version”:0,“appUrl”:null,“config”:{“definition”:{“kill_timeout”:5,“kill_signal”:“SIGINT”,“processes”:,“experimental”:{“allowed_public_ports”:,“auto_rollback”:true},“services”:[{“processes”:[“app”],“protocol”:“tcp”,“internal_port”:8080,“concurrency”:{“soft_limit”:20,“hard_limit”:25,“type”:“connections”},“ports”:[{“port”:80,“handlers”:[“http”]},{“port”:443,“handlers”:[“tls”,“http”]}],“tcp_checks”:[{“interval”:“15s”,“timeout”:“2s”,“grace_period”:“1s”,“restart_limit”:0}],“http_checks”:,“script_checks”:}],“env”:{}}},“organization”:{“id”:“AP276N9M5j2qJTaRa50QLKp0Q8Fx1l”,“slug”:“personal”},“services”:[{“description”:“TCP 80/443 ⇢ 8080”,“protocol”:“TCP”,“internalPort”:8080,“ports”:[{“port”:80,“handlers”:[“HTTP”]},{“port”:443,“handlers”:[“TLS”,“HTTP”]}]}],“ipAddresses”:{“nodes”:},“imageDetails”:{“repository”:“unknown”,“version”:“unknown”}}}}
DEBUG → POST GraphQL Playground {{“query”:“mutation($input: ValidateWireGuardPeersInput!) { validateWireGuardPeers(input: $input) { invalidPeerIps } }”,“variables”:{“input”:{“peerIps”:[“fdaa:0:4b73:a7b:16a9:0:a:2”]}}}
}
Waiting for remote builder fly-builder-small-sky-7039… ⡿ DEBUG ← 200 GraphQL Playground (283.58ms) {“data”:{“validateWireGuardPeers”:{“invalidPeerIps”:}}}
DEBUG result image: error:error connecting to docker: failed building options: failed probing “personal”: context deadline exceeded
Error failed to fetch an image or build from source: error connecting to docker: failed building options: failed probing “personal”: context deadline exceeded

Hey there,

Could you try some of these commands? They might shed more light on the issue.

fly doctor
DEV=1 fly agent probe personal
fly dig TXT _apps.internal
2 Likes

thirdweb-lootbox-tutorial-main % fly doctor
TEST STATUS
Agent couldn’t ping agent: dial unix /Users/Kto_tut/.fly/fly-agent.sock: connect: connection refused
Docker (local) failed pinging docker instance: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
Probe (app) failed probing personal: failed probing “personal”: context deadline exceeded
Token PASS
Unix socket PASS

thirdweb-lootbox-tutorial-main % DEV=1 fly agent probe personal
Error failed probing “personal”: context deadline exceeded

thirdweb-lootbox-tutorial-main % fly dig TXT _apps.internal
Error connect tcp [fdaa:0:4b73::3]:53: operation timed out

Nice, thanks! How about fly agent stop Then fly agent start?

2 Likes

thirdweb-lootbox-tutorial-main % fly agent stop
thirdweb-lootbox-tutorial-main % fly agent start
thirdweb-lootbox-tutorial-main % fly deploy
==> Verifying app config
→ Verified app config
==> Building image
Error failed to fetch an image or build from source: error connecting to docker: failed building options: failed probing “personal”: context deadline exceeded

Which operating system and version of flyctl?

uname -a
fly version

thirdweb-lootbox-tutorial-main % uname -a
Darwin MacBook-Pro-Victoriya.local 21.2.0 Darwin Kernel Version 21.2.0: Sun Nov 28 20:28:54 PST 2021; root:xnu-8019.61.5~1/RELEASE_X86_64 x86_64
thirdweb-lootbox-tutorial-main % fly version
fly v0.0.295 darwin/amd64 Commit: c47821c BuildDate: 2022-02-03T16:00:09Z

OK, how about this?

ps auwx | grep "fly agent"

Here we’re trying to tell if you have Fly’s background agent running.

thirdweb-lootbox-tutorial-main % ps auwx | grep “fly agent”
Kto_tut 25986 0,0 0,0 34140252 892 s000 S+ 8:59 0:00.00 grep fly agent
Kto_tut 25924 0,0 0,2 34897096 31368 s000 S 8:52 0:00.54 fly agent run /Users/Kto_tut/.fly/agent-logs/4002767549.log

OK, then:

tail -100 /Users/Kto_tut/.fly/agent-logs/4002767549.log

. . . ok

Thanks! I’d go ahead and delete this last reply, as it contains your wireguard peer info. That can be invalidated, but just in case!