Deployments not working: error connecting to docker

[19:24:31]phoenix/phx166» fly ips private
ID REGION IP

just to clarify:
I currently don’t have an app running or deployed. I nuked the project up to start over with a generic phoenix app. I just wanted to rule out any config errors on my end.

So my “main” problem is that I can’t deploy… and then I found this thread

[19:22:43]phoenix/phx166» fly launch --remote-only
An existing fly.toml file was found for app solitary-snow-2344
App is not running, deploy...
Deploying solitary-snow-2344
==> Validating app configuration
--> Validating app configuration done
Services
TCP 80/443 ⇢ 8080
Error error connecting to docker: failed building options: failed probing "personal": context deadline exceeded

:wave: FYI this is still a thing. Not critical though… but would be great to deploy eventually

fly launch --remote-only

An existing fly.toml file was found for app ancient-flower-5218
App is not running, deploy...
Deploying ancient-flower-5218
==> Validating app configuration
--> Validating app configuration done
Services
TCP 80/443 ⇢ 8080
Error error connecting to docker: failed building options: failed probing "personal": context deadline exceeded

fly doctor

TEST          	STATUS
Agent         	PASS
Docker (local)	PASS
Probe (app)   	failed probing personal: failed probing "personal": context deadline exceeded
Token         	PASS
Unix socket   	PASS

Hey! Please install the latest prerelease:

curl -L https://fly.io/install.sh | sh -s pre

Then try:

fly wireguard status

Select your personal org and, assuming you only have one ‘peer’ on the list, pick it and paste the output here.

I’m having the same problem.

Just tried the prerelease version. Here is the output from fly wireguard status:

Alive: true
Installed On Gateway At: 2022-02-11T09:27:30Z
Traffic: rx:0 tx:0

The deploy command still fails:

$ flyctl deploy
==> Verifying app config
--> Verified app config
==> Building image
Error failed to fetch an image or build from source: error connecting to docker: failed building options: failed probing "personal": context deadline exceeded

THX here you go!

:white_check_mark: (didn’t work at first, had to remove my brew install)

assuming you only have one ‘peer’ on the list

:grimacing: I have 3 peers in the list. Likley those got added when I tried to connect via livebook and wireguard tunnel to a remote app (this is for a future “halp” ticket :laughing: )

here is the output for all three:

[7:26:09]phoenix/phx166(main⚡)» fly wireguard status
Automatically selected personal organization: kim.asdf@gmail.com
? Select peer: interactive-kim-kim-asdf-gmail-com-214
Alive: true
Installed On Gateway At: 2022-02-09T03:14:52Z
Traffic: rx:0 tx:0
[7:26:37]phoenix/phx166(main⚡)» fly wireguard status
Automatically selected personal organization: kim.asdf@gmail.com
? Select peer: interactive-kim-kim-asdf-gmail-com-714
Alive: true
Installed On Gateway At: 2022-02-09T03:13:12Z
Traffic: rx:0 tx:0
[7:26:44]phoenix/phx166(main⚡)» fly wireguard status
Automatically selected personal organization: kim.asdf@gmail.com
? Select peer: interactive-kim-kim-asdf-gmail-com-274
Alive: true
Last Source Address: 206.128.71.128:25504
Installed On Gateway At: 2022-02-03T23:14:54Z
Traffic: rx:39060 tx:55692

@Hypermind

Are you on an Apple M1 machine? In that case, try flyctl deploy --remote-only
see this comment here → Error starting Hex application - #7 by kurt

2 Likes

OK - can you try fly ping -o personal once again?

its still timing out

[9:09:16]phoenix/phx166(main⚡)» LOG_LEVEL=debug fly ping -o personal
DEBUG Loaded flyctl config from/Users/kim/.fly/config.yml
DEBUG determined hostname: "kim-"
DEBUG determined working directory: "/Users/kim/Projects/phoenix/phx166"
DEBUG determined user home directory: "/Users/kim"
DEBUG determined config directory: "/Users/kim/.fly"
DEBUG ensured config directory exists.
DEBUG ensured config directory perms.
DEBUG cache loaded.
DEBUG config initialized.
DEBUG initialized task manager.
DEBUG started querying for new release
DEBUG client initialized.
DEBUG app config loaded from /Users/kim/Projects/phoenix/phx166/fly.toml
DEBUG --> POST https://api.fly.io/graphql {{"query":"query($slug: String!) { organization(slug: $slug) { id internalNumericId slug name type } }","variables":{"slug":"personal"}}
}
DEBUG <-- 200 https://api.fly.io/graphql (1.71s) {"data":{"organization":{"id":"60L7BO3NPGLMXHvjvN6nmoY6nAFq08","internalNumericId":"29800","slug":"personal","name":"kim.sdfd@gmail.com","type":"PERSONAL"}}}
DEBUG --> POST https://api.fly.io/graphql {{"query":"mutation($input: ValidateWireGuardPeersInput!) { validateWireGuardPeers(input: $input) { invalidPeerIps } }","variables":{"input":{"peerIps":["fdaa:0:4152:a7b:177d:0:a:2"]}}}
}
DEBUG <-- 200 https://api.fly.io/graphql (563.31ms) {"data":{"validateWireGuardPeers":{"invalidPeerIps":[]}}}

OK - are you using a VPN or anything like that?

Last try here - please download the release from this page that corresponds to your machine’s architecture: Release v0.0.269-dev-tqbf-tcp-proxy-48b8696 · superfly/flyctl-dev · GitHub

This is a special release that uses a different approach (websocket TCP instead of UDP) to connect to Wireguard. Try unpacking it, setting it executable with chmod 750 flyctl*, then:

fly agent stop
./flyctl-whatever deploy --remote-only

no VPN or any network restrictions

Unfortunately still no :game_die:. It tries to connect to the remote builder …
:point_down: output with debug flag

[11:29:23]phoenix/phx166(main⚡)» fly agent stop
Update available 0.0.269-dev-tqbf-tcp-proxy-48b8696 -> v0.0.297-pre-2.
Run "fly version update" to upgrade.
[11:29:42]phoenix/phx166(main⚡)» LOG_LEVEL=debug flyctl deploy --remote-only
DEBUG Loaded flyctl config from/Users/kim/.fly/config.yml
DEBUG determined hostname: "kimasfdsdf"
DEBUG determined working directory: "/Users/kim/Projects/phoenix/phx166"
DEBUG determined user home directory: "/Users/kim"
DEBUG determined config directory: "/Users/kim/.fly"
DEBUG cache loaded.
DEBUG config initialized.
DEBUG initialized task manager.
DEBUG skipped querying for new release
Update available 0.0.269-dev-tqbf-tcp-proxy-48b8696 -> v0.0.297-pre-2.
Run "flyctl version update" to upgrade.
DEBUG client initialized.
DEBUG Loading app config from/Users/kim/Projects/phoenix/phx166/fly.toml
DEBUG Working Directory: /Users/kim/Projects/phoenix/phx166
DEBUG App Config File: /Users/kim/Projects/phoenix/phx166/fly.toml
Deploying vanilla166
==> Validating app configuration
DEBUG --> POST https://api.fly.io/graphql {{"query":"query($appName: String!, $definition: JSON!) { app(name: $appName) { parseConfig(definition: $definition) { definition valid errors services { description } } } }","variables":{"appName":"vanilla166","definition":{"deploy":{"release_command":"/app/bin/migrate"},"env":{"PHX_HOST":"vanilla166.fly.dev","PORT":"8080"},"experimental":{"allowed_public_ports":[],"auto_rollback":true},"kill_signal":"SIGTERM","kill_timeout":5,"processes":[],"services":[{"concurrency":{"hard_limit":25,"soft_limit":20,"type":"connections"},"http_checks":[],"internal_port":8080,"ports":[{"handlers":["http"],"port":80},{"handlers":["tls","http"],"port":443}],"processes":["app"],"protocol":"tcp","script_checks":[],"tcp_checks":[{"grace_period":"1s","interval":"15s","restart_limit":0,"timeout":"2s"}]}]}}}
}
DEBUG <-- 200 https://api.fly.io/graphql (704.51ms) {"data":{"app":{"parseConfig":{"definition":{"kill_timeout":5,"kill_signal":"SIGTERM","processes":[],"deploy":{"release_command":"/app/bin/migrate"},"experimental":{"allowed_public_ports":[],"auto_rollback":true},"services":[{"processes":["app"],"protocol":"tcp","internal_port":8080,"concurrency":{"soft_limit":20,"hard_limit":25,"type":"connections"},"ports":[{"port":80,"handlers":["http"]},{"port":443,"handlers":["tls","http"]}],"tcp_checks":[{"interval":"15s","timeout":"2s","grace_period":"1s","restart_limit":0}],"http_checks":[],"script_checks":[]}],"env":{"PHX_HOST":"vanilla166.fly.dev","PORT":"8080"}},"valid":true,"errors":[],"services":[{"description":"TCP 80/443 ⇢ 8080"}]}}}}
--> Validating app configuration done
Services
TCP 80/443 ⇢ 8080
DEBUG trying remote docker daemon
DEBUG Trying 'Buildpacks' strategy
DEBUG no buildpack builder configured, skipping
DEBUG result image:<nil> error:<nil>
DEBUG Trying 'Dockerfile' strategy
DEBUG --> POST https://api.fly.io/graphql {{"query":"mutation($input: EnsureMachineRemoteBuilderInput!) { ensureMachineRemoteBuilder(input: $input) { machine { id state ips { nodes { family kind ip } } }, app { name organization { slug } } } }","variables":{"input":{"appName":"vanilla166","organizationId":null}}}
}
DEBUG <-- 200 https://api.fly.io/graphql (2.96s) {"data":{"ensureMachineRemoteBuilder":{"machine":{"id":"7175a0e0","state":"starting","ips":{"nodes":[{"family":"v6","kind":"privatenet","ip":"fdaa:0:4152:a7b:1a:0:901d:2"},{"family":"v6","kind":"public","ip":"2604:1380:4111:1e04::901d:3"},{"family":"v4","kind":"private","ip":"172.19.0.98"}]}},"app":{"name":"fly-builder-winter-wind-513","organization":{"slug":"personal"}}}}}
DEBUG checking ip &{Family:v6 Kind:privatenet IP:fdaa:0:4152:a7b:1a:0:901d:2 MaskSize:0}
DEBUG --> POST https://api.fly.io/graphql {{"query":"query ($appName: String!) { app(name: $appName) { id name hostname deployed status version appUrl config { definition } organization { id slug } services { description protocol internalPort ports { port handlers } } ipAddresses { nodes { id address type createdAt } } imageDetails { repository version } } }","variables":{"appName":"vanilla166"}}
}
Waiting for remote builder fly-builder-winter-wind-513... connecting ⣻ DEBUG <-- 200 https://api.fly.io/graphql (718.44ms) {"data":{"app":{"id":"vanilla166","name":"vanilla166","hostname":"vanilla166.fly.dev","deployed":false,"status":"pending","version":0,"appUrl":null,"config":{"definition":{"kill_timeout":5,"kill_signal":"SIGINT","processes":[],"experimental":{"allowed_public_ports":[],"auto_rollback":true},"services":[{"processes":["app"],"protocol":"tcp","internal_port":8080,"concurrency":{"soft_limit":20,"hard_limit":25,"type":"connections"},"ports":[{"port":80,"handlers":["http"]},{"port":443,"handlers":["tls","http"]}],"tcp_checks":[{"interval":"15s","timeout":"2s","grace_period":"1s","restart_limit":0}],"http_checks":[],"script_checks":[]}],"env":{}}},"organization":{"id":"60L7BO3NPGLMXHvjvN6nmoY6nAFq08","slug":"personal"},"services":[{"description":"TCP 80/443 ⇢ 8080","protocol":"TCP","internalPort":8080,"ports":[{"port":80,"handlers":["HTTP"]},{"port":443,"handlers":["TLS","HTTP"]}]}],"ipAddresses":{"nodes":[]},"imageDetails":{"repository":"unknown","version":"unknown"}}}}
DEBUG --> POST https://api.fly.io/graphql {{"query":"mutation($input: ValidateWireGuardPeersInput!) { validateWireGuardPeers(input: $input) { invalidPeerIps } }","variables":{"input":{"peerIps":["fdaa:0:4152:a7b:177d:0:a:2"]}}}
}
Waiting for remote builder fly-builder-winter-wind-513... connecting ⢿ DEBUG <-- 200 https://api.fly.io/graphql (149.08ms) {"data":{"validateWireGuardPeers":{"invalidPeerIps":[]}}}
DEBUG started agent process 46338
Waiting for remote builder fly-builder-winter-wind-513... connecting ⢿

Thanks to debugging help from @kim, we tracked this down to a regression in how flyctl handles wireguard peers that have been reaped in our backend, but are still active locally.

While we’re working on a fix, here’s a way you can try to fix it for now:

  1. Stop the fly agent with fly agent stop
  2. Remove the wireguard peer entry from ~/.fly/config.yml for the organization you’re having trouble with, including the org name key.
  3. Start the agent and try fly ping -a orgname

If this works, deploys should work as well.
2. Stop the

2 Likes

Silly me, I was using fly tool on Windows Subsystem for Linux. Your post about Apple M1 pointed me to the right direction. Once I’ve installed the native fly toolkit for Windows, everything started working.

2 Likes

I’m having exactly the same issue and the same errors as Kim, except when I do fly wireguard status I get Alive: false.

Also trying to fix it doesn’t seem to help. Or I’m doing it wrong:

access_token: x
wire_guard_state:
  personal:
    org: personal
    name: x
    region: ams
    localpublic: x
    localprivate: x
    dns: ""
    peer:
      peerip: x
      endpointip: ams1.gateway.6pn.dev
      pubkey: x

So I have to remove this part from config.yaml, right?

  personal:
    org: personal
    name: x
    region: ams
    localpublic: x
    localprivate: x
    dns: ""
    peer:
      peerip: x
      endpointip: ams1.gateway.6pn.dev
      pubkey: x

I’m not in a hurry btw. I’m currently just trying out fly.io :slightly_smiling_face:

Yeah, you can try: stop agent, remove that, deploy (agent will start on its own).

Also if you could provide your peerIP, we could take a look at something on our end. That’s safe to share.

getting similar issues:

❯ fly deploy
==> Verifying app config
→ Verified app config
==> Building image
Error failed to fetch an image or build from source: failed building options: failed probing “personal”: context deadline exceeded

❯ fly destroy fly-builder-holy-sunset-5566

❯ fly ping -o personal

ping seems to hang

CMD+C then I try deploying again, same as first output…

Last week, we deployed a fix that should have solved this problem for most people. Which OS are you running?

In the cases where it still happens, there is likely an issue with networking. Can you confirm if you are running over a VPN of some kind? If so, turn it off and try again. Also try these commands:

fly doctor
fly agent stop; fly agent start

Then trying fly ping -o personal again.

Probe (app) failed probing personal: failed probing "personal": context deadline exceeded

bit more trial and error seemed todo it

I have the same error:

❯ fly deploy
==> Verifying app config
--> Verified app config
==> Building image
Error failed to fetch an image or build from source: error connecting to docker: failed building options: failed probing "personal": context deadline exceeded
❯ fly version
fly v0.0.320 darwin/amd64 Commit: 58aae1e BuildDate: 2022-04-13T15:58:08Z

Tried all the methods above but no luck, is there any information I can provide to help? Thanks

1 Like