Deployments not working: error connecting to docker

Hypermind · February 11, 2022, 1:48pm

I’m having the same problem.

Just tried the prerelease version. Here is the output from fly wireguard status:

Alive: true
Installed On Gateway At: 2022-02-11T09:27:30Z
Traffic: rx:0 tx:0

The deploy command still fails:

$ flyctl deploy
==> Verifying app config
--> Verified app config
==> Building image
Error failed to fetch an image or build from source: error connecting to docker: failed building options: failed probing "personal": context deadline exceeded

kim · February 11, 2022, 3:31pm

THX here you go!

(didn’t work at first, had to remove my brew install)

assuming you only have one ‘peer’ on the list

I have 3 peers in the list. Likley those got added when I tried to connect via livebook and wireguard tunnel to a remote app (this is for a future “halp” ticket )

here is the output for all three:

[7:26:09]phoenix/phx166(main⚡)» fly wireguard status
Automatically selected personal organization: kim.asdf@gmail.com
? Select peer: interactive-kim-kim-asdf-gmail-com-214
Alive: true
Installed On Gateway At: 2022-02-09T03:14:52Z
Traffic: rx:0 tx:0
[7:26:37]phoenix/phx166(main⚡)» fly wireguard status
Automatically selected personal organization: kim.asdf@gmail.com
? Select peer: interactive-kim-kim-asdf-gmail-com-714
Alive: true
Installed On Gateway At: 2022-02-09T03:13:12Z
Traffic: rx:0 tx:0
[7:26:44]phoenix/phx166(main⚡)» fly wireguard status
Automatically selected personal organization: kim.asdf@gmail.com
? Select peer: interactive-kim-kim-asdf-gmail-com-274
Alive: true
Last Source Address: 206.128.71.128:25504
Installed On Gateway At: 2022-02-03T23:14:54Z
Traffic: rx:39060 tx:55692

kim · February 11, 2022, 3:34pm

@Hypermind

Are you on an Apple M1 machine? In that case, try flyctl deploy --remote-only
see this comment here → Error starting Hex application - #7 by kurt

jsierles · February 11, 2022, 4:53pm

OK - can you try fly ping -o personal once again?

kim · February 11, 2022, 5:10pm

its still timing out

[9:09:16]phoenix/phx166(main⚡)» LOG_LEVEL=debug fly ping -o personal
DEBUG Loaded flyctl config from/Users/kim/.fly/config.yml
DEBUG determined hostname: "kim-"
DEBUG determined working directory: "/Users/kim/Projects/phoenix/phx166"
DEBUG determined user home directory: "/Users/kim"
DEBUG determined config directory: "/Users/kim/.fly"
DEBUG ensured config directory exists.
DEBUG ensured config directory perms.
DEBUG cache loaded.
DEBUG config initialized.
DEBUG initialized task manager.
DEBUG started querying for new release
DEBUG client initialized.
DEBUG app config loaded from /Users/kim/Projects/phoenix/phx166/fly.toml
DEBUG --> POST https://api.fly.io/graphql {{"query":"query($slug: String!) { organization(slug: $slug) { id internalNumericId slug name type } }","variables":{"slug":"personal"}}
}
DEBUG <-- 200 https://api.fly.io/graphql (1.71s) {"data":{"organization":{"id":"60L7BO3NPGLMXHvjvN6nmoY6nAFq08","internalNumericId":"29800","slug":"personal","name":"kim.sdfd@gmail.com","type":"PERSONAL"}}}
DEBUG --> POST https://api.fly.io/graphql {{"query":"mutation($input: ValidateWireGuardPeersInput!) { validateWireGuardPeers(input: $input) { invalidPeerIps } }","variables":{"input":{"peerIps":["fdaa:0:4152:a7b:177d:0:a:2"]}}}
}
DEBUG <-- 200 https://api.fly.io/graphql (563.31ms) {"data":{"validateWireGuardPeers":{"invalidPeerIps":[]}}}

jsierles · February 11, 2022, 5:59pm

OK - are you using a VPN or anything like that?

Last try here - please download the release from this page that corresponds to your machine’s architecture: Release v0.0.269-dev-tqbf-tcp-proxy-48b8696 · superfly/flyctl-dev · GitHub

This is a special release that uses a different approach (websocket TCP instead of UDP) to connect to Wireguard. Try unpacking it, setting it executable with chmod 750 flyctl*, then:

fly agent stop
./flyctl-whatever deploy --remote-only

kim · February 11, 2022, 7:33pm

no VPN or any network restrictions

Unfortunately still no . It tries to connect to the remote builder …
output with debug flag

[11:29:23]phoenix/phx166(main⚡)» fly agent stop
Update available 0.0.269-dev-tqbf-tcp-proxy-48b8696 -> v0.0.297-pre-2.
Run "fly version update" to upgrade.
[11:29:42]phoenix/phx166(main⚡)» LOG_LEVEL=debug flyctl deploy --remote-only
DEBUG Loaded flyctl config from/Users/kim/.fly/config.yml
DEBUG determined hostname: "kimasfdsdf"
DEBUG determined working directory: "/Users/kim/Projects/phoenix/phx166"
DEBUG determined user home directory: "/Users/kim"
DEBUG determined config directory: "/Users/kim/.fly"
DEBUG cache loaded.
DEBUG config initialized.
DEBUG initialized task manager.
DEBUG skipped querying for new release
Update available 0.0.269-dev-tqbf-tcp-proxy-48b8696 -> v0.0.297-pre-2.
Run "flyctl version update" to upgrade.
DEBUG client initialized.
DEBUG Loading app config from/Users/kim/Projects/phoenix/phx166/fly.toml
DEBUG Working Directory: /Users/kim/Projects/phoenix/phx166
DEBUG App Config File: /Users/kim/Projects/phoenix/phx166/fly.toml
Deploying vanilla166
==> Validating app configuration
DEBUG --> POST https://api.fly.io/graphql {{"query":"query($appName: String!, $definition: JSON!) { app(name: $appName) { parseConfig(definition: $definition) { definition valid errors services { description } } } }","variables":{"appName":"vanilla166","definition":{"deploy":{"release_command":"/app/bin/migrate"},"env":{"PHX_HOST":"vanilla166.fly.dev","PORT":"8080"},"experimental":{"allowed_public_ports":[],"auto_rollback":true},"kill_signal":"SIGTERM","kill_timeout":5,"processes":[],"services":[{"concurrency":{"hard_limit":25,"soft_limit":20,"type":"connections"},"http_checks":[],"internal_port":8080,"ports":[{"handlers":["http"],"port":80},{"handlers":["tls","http"],"port":443}],"processes":["app"],"protocol":"tcp","script_checks":[],"tcp_checks":[{"grace_period":"1s","interval":"15s","restart_limit":0,"timeout":"2s"}]}]}}}
}
DEBUG <-- 200 https://api.fly.io/graphql (704.51ms) {"data":{"app":{"parseConfig":{"definition":{"kill_timeout":5,"kill_signal":"SIGTERM","processes":[],"deploy":{"release_command":"/app/bin/migrate"},"experimental":{"allowed_public_ports":[],"auto_rollback":true},"services":[{"processes":["app"],"protocol":"tcp","internal_port":8080,"concurrency":{"soft_limit":20,"hard_limit":25,"type":"connections"},"ports":[{"port":80,"handlers":["http"]},{"port":443,"handlers":["tls","http"]}],"tcp_checks":[{"interval":"15s","timeout":"2s","grace_period":"1s","restart_limit":0}],"http_checks":[],"script_checks":[]}],"env":{"PHX_HOST":"vanilla166.fly.dev","PORT":"8080"}},"valid":true,"errors":[],"services":[{"description":"TCP 80/443 ⇢ 8080"}]}}}}
--> Validating app configuration done
Services
TCP 80/443 ⇢ 8080
DEBUG trying remote docker daemon
DEBUG Trying 'Buildpacks' strategy
DEBUG no buildpack builder configured, skipping
DEBUG result image:<nil> error:<nil>
DEBUG Trying 'Dockerfile' strategy
DEBUG --> POST https://api.fly.io/graphql {{"query":"mutation($input: EnsureMachineRemoteBuilderInput!) { ensureMachineRemoteBuilder(input: $input) { machine { id state ips { nodes { family kind ip } } }, app { name organization { slug } } } }","variables":{"input":{"appName":"vanilla166","organizationId":null}}}
}
DEBUG <-- 200 https://api.fly.io/graphql (2.96s) {"data":{"ensureMachineRemoteBuilder":{"machine":{"id":"7175a0e0","state":"starting","ips":{"nodes":[{"family":"v6","kind":"privatenet","ip":"fdaa:0:4152:a7b:1a:0:901d:2"},{"family":"v6","kind":"public","ip":"2604:1380:4111:1e04::901d:3"},{"family":"v4","kind":"private","ip":"172.19.0.98"}]}},"app":{"name":"fly-builder-winter-wind-513","organization":{"slug":"personal"}}}}}
DEBUG checking ip &{Family:v6 Kind:privatenet IP:fdaa:0:4152:a7b:1a:0:901d:2 MaskSize:0}
DEBUG --> POST https://api.fly.io/graphql {{"query":"query ($appName: String!) { app(name: $appName) { id name hostname deployed status version appUrl config { definition } organization { id slug } services { description protocol internalPort ports { port handlers } } ipAddresses { nodes { id address type createdAt } } imageDetails { repository version } } }","variables":{"appName":"vanilla166"}}
}
Waiting for remote builder fly-builder-winter-wind-513... connecting ⣻ DEBUG <-- 200 https://api.fly.io/graphql (718.44ms) {"data":{"app":{"id":"vanilla166","name":"vanilla166","hostname":"vanilla166.fly.dev","deployed":false,"status":"pending","version":0,"appUrl":null,"config":{"definition":{"kill_timeout":5,"kill_signal":"SIGINT","processes":[],"experimental":{"allowed_public_ports":[],"auto_rollback":true},"services":[{"processes":["app"],"protocol":"tcp","internal_port":8080,"concurrency":{"soft_limit":20,"hard_limit":25,"type":"connections"},"ports":[{"port":80,"handlers":["http"]},{"port":443,"handlers":["tls","http"]}],"tcp_checks":[{"interval":"15s","timeout":"2s","grace_period":"1s","restart_limit":0}],"http_checks":[],"script_checks":[]}],"env":{}}},"organization":{"id":"60L7BO3NPGLMXHvjvN6nmoY6nAFq08","slug":"personal"},"services":[{"description":"TCP 80/443 ⇢ 8080","protocol":"TCP","internalPort":8080,"ports":[{"port":80,"handlers":["HTTP"]},{"port":443,"handlers":["TLS","HTTP"]}]}],"ipAddresses":{"nodes":[]},"imageDetails":{"repository":"unknown","version":"unknown"}}}}
DEBUG --> POST https://api.fly.io/graphql {{"query":"mutation($input: ValidateWireGuardPeersInput!) { validateWireGuardPeers(input: $input) { invalidPeerIps } }","variables":{"input":{"peerIps":["fdaa:0:4152:a7b:177d:0:a:2"]}}}
}
Waiting for remote builder fly-builder-winter-wind-513... connecting ⢿ DEBUG <-- 200 https://api.fly.io/graphql (149.08ms) {"data":{"validateWireGuardPeers":{"invalidPeerIps":[]}}}
DEBUG started agent process 46338
Waiting for remote builder fly-builder-winter-wind-513... connecting ⢿

jsierles · February 12, 2022, 12:26am

Thanks to debugging help from @kim, we tracked this down to a regression in how flyctl handles wireguard peers that have been reaped in our backend, but are still active locally.

While we’re working on a fix, here’s a way you can try to fix it for now:

Stop the fly agent with fly agent stop
Remove the wireguard peer entry from ~/.fly/config.yml for the organization you’re having trouble with, including the org name key.
Start the agent and try fly ping -a orgname

If this works, deploys should work as well.
2. Stop the

Hypermind · February 12, 2022, 12:14pm

Silly me, I was using fly tool on Windows Subsystem for Linux. Your post about Apple M1 pointed me to the right direction. Once I’ve installed the native fly toolkit for Windows, everything started working.

user119 · February 15, 2022, 4:03pm

I’m having exactly the same issue and the same errors as Kim, except when I do fly wireguard status I get Alive: false.

Also trying to fix it doesn’t seem to help. Or I’m doing it wrong:

access_token: x
wire_guard_state:
  personal:
    org: personal
    name: x
    region: ams
    localpublic: x
    localprivate: x
    dns: ""
    peer:
      peerip: x
      endpointip: ams1.gateway.6pn.dev
      pubkey: x

So I have to remove this part from config.yaml, right?

  personal:
    org: personal
    name: x
    region: ams
    localpublic: x
    localprivate: x
    dns: ""
    peer:
      peerip: x
      endpointip: ams1.gateway.6pn.dev
      pubkey: x

I’m not in a hurry btw. I’m currently just trying out fly.io

jsierles · February 15, 2022, 5:10pm

Yeah, you can try: stop agent, remove that, deploy (agent will start on its own).

jsierles · February 15, 2022, 8:50pm

Also if you could provide your peerIP, we could take a look at something on our end. That’s safe to share.

Mike_James · February 21, 2022, 6:36am

getting similar issues:

❯ fly deploy
==> Verifying app config
→ Verified app config
==> Building image
Error failed to fetch an image or build from source: failed building options: failed probing “personal”: context deadline exceeded

❯ fly destroy fly-builder-holy-sunset-5566

❯ fly ping -o personal

ping seems to hang

CMD+C then I try deploying again, same as first output…

jsierles · February 21, 2022, 10:02am

Last week, we deployed a fix that should have solved this problem for most people. Which OS are you running?

In the cases where it still happens, there is likely an issue with networking. Can you confirm if you are running over a VPN of some kind? If so, turn it off and try again. Also try these commands:

fly doctor
fly agent stop; fly agent start

Then trying fly ping -o personal again.

Mike_James · February 22, 2022, 2:42am

Probe (app) failed probing personal: failed probing "personal": context deadline exceeded

Mike_James · February 22, 2022, 2:44am

bit more trial and error seemed todo it

goofansu · April 20, 2022, 3:34am

I have the same error:

❯ fly deploy
==> Verifying app config
--> Verified app config
==> Building image
Error failed to fetch an image or build from source: error connecting to docker: failed building options: failed probing "personal": context deadline exceeded

❯ fly version
fly v0.0.320 darwin/amd64 Commit: 58aae1e BuildDate: 2022-04-13T15:58:08Z

Tried all the methods above but no luck, is there any information I can provide to help? Thanks

jsierles · April 20, 2022, 6:33am

Make sure you have the latest flyctl, then try fly wireguard websockets enable. After that, try fly deploy again and see if it helped. This will connect to Fly over TCP 443 instead of the default UDP used by Wireguard.

goofansu · April 20, 2022, 10:18am

Thank you! It works.

mfilej · April 28, 2022, 8:29am

For anyone who arrives here by googling the same problem.

I was getting this error and none of the above helped. What fixed it in the end:

fly wireguard list
fly wireguard reset
fly wireguard remove

When removing, select the one that was initially listed as the output of the list command.

Topic		Replies	Views
fly deploy fails - docker context deadline exceeded	1	305	August 2, 2022
Flyctl deploy errors after creating build context Build debugging	1	336	October 28, 2022
error connecting to docker	12	529	July 18, 2022
failed to fetch an image or build from source: error connecting to docker Build debugging	3	326	July 12, 2023
Trouble Deploying: failed to fetch an image or build from source: error fetching docker server info: Get "http://[fdaa:1:1184:a7b:f4:eb3a:d214:2]:2375/v1.41/info": context deadline exceeded	2	322	June 15, 2023

Deployments not working: error connecting to docker

Related topics