Deployments not working: error connecting to docker

OK - can you try fly ping -o personal once again?

its still timing out

[9:09:16]phoenix/phx166(main⚡)» LOG_LEVEL=debug fly ping -o personal
DEBUG Loaded flyctl config from/Users/kim/.fly/config.yml
DEBUG determined hostname: "kim-"
DEBUG determined working directory: "/Users/kim/Projects/phoenix/phx166"
DEBUG determined user home directory: "/Users/kim"
DEBUG determined config directory: "/Users/kim/.fly"
DEBUG ensured config directory exists.
DEBUG ensured config directory perms.
DEBUG cache loaded.
DEBUG config initialized.
DEBUG initialized task manager.
DEBUG started querying for new release
DEBUG client initialized.
DEBUG app config loaded from /Users/kim/Projects/phoenix/phx166/fly.toml
DEBUG --> POST https://api.fly.io/graphql {{"query":"query($slug: String!) { organization(slug: $slug) { id internalNumericId slug name type } }","variables":{"slug":"personal"}}
}
DEBUG <-- 200 https://api.fly.io/graphql (1.71s) {"data":{"organization":{"id":"60L7BO3NPGLMXHvjvN6nmoY6nAFq08","internalNumericId":"29800","slug":"personal","name":"kim.sdfd@gmail.com","type":"PERSONAL"}}}
DEBUG --> POST https://api.fly.io/graphql {{"query":"mutation($input: ValidateWireGuardPeersInput!) { validateWireGuardPeers(input: $input) { invalidPeerIps } }","variables":{"input":{"peerIps":["fdaa:0:4152:a7b:177d:0:a:2"]}}}
}
DEBUG <-- 200 https://api.fly.io/graphql (563.31ms) {"data":{"validateWireGuardPeers":{"invalidPeerIps":[]}}}

OK - are you using a VPN or anything like that?

Last try here - please download the release from this page that corresponds to your machine’s architecture: Release v0.0.269-dev-tqbf-tcp-proxy-48b8696 · superfly/flyctl-dev · GitHub

This is a special release that uses a different approach (websocket TCP instead of UDP) to connect to Wireguard. Try unpacking it, setting it executable with chmod 750 flyctl*, then:

fly agent stop
./flyctl-whatever deploy --remote-only

no VPN or any network restrictions

Unfortunately still no :game_die:. It tries to connect to the remote builder …
:point_down: output with debug flag

[11:29:23]phoenix/phx166(main⚡)» fly agent stop
Update available 0.0.269-dev-tqbf-tcp-proxy-48b8696 -> v0.0.297-pre-2.
Run "fly version update" to upgrade.
[11:29:42]phoenix/phx166(main⚡)» LOG_LEVEL=debug flyctl deploy --remote-only
DEBUG Loaded flyctl config from/Users/kim/.fly/config.yml
DEBUG determined hostname: "kimasfdsdf"
DEBUG determined working directory: "/Users/kim/Projects/phoenix/phx166"
DEBUG determined user home directory: "/Users/kim"
DEBUG determined config directory: "/Users/kim/.fly"
DEBUG cache loaded.
DEBUG config initialized.
DEBUG initialized task manager.
DEBUG skipped querying for new release
Update available 0.0.269-dev-tqbf-tcp-proxy-48b8696 -> v0.0.297-pre-2.
Run "flyctl version update" to upgrade.
DEBUG client initialized.
DEBUG Loading app config from/Users/kim/Projects/phoenix/phx166/fly.toml
DEBUG Working Directory: /Users/kim/Projects/phoenix/phx166
DEBUG App Config File: /Users/kim/Projects/phoenix/phx166/fly.toml
Deploying vanilla166
==> Validating app configuration
DEBUG --> POST https://api.fly.io/graphql {{"query":"query($appName: String!, $definition: JSON!) { app(name: $appName) { parseConfig(definition: $definition) { definition valid errors services { description } } } }","variables":{"appName":"vanilla166","definition":{"deploy":{"release_command":"/app/bin/migrate"},"env":{"PHX_HOST":"vanilla166.fly.dev","PORT":"8080"},"experimental":{"allowed_public_ports":[],"auto_rollback":true},"kill_signal":"SIGTERM","kill_timeout":5,"processes":[],"services":[{"concurrency":{"hard_limit":25,"soft_limit":20,"type":"connections"},"http_checks":[],"internal_port":8080,"ports":[{"handlers":["http"],"port":80},{"handlers":["tls","http"],"port":443}],"processes":["app"],"protocol":"tcp","script_checks":[],"tcp_checks":[{"grace_period":"1s","interval":"15s","restart_limit":0,"timeout":"2s"}]}]}}}
}
DEBUG <-- 200 https://api.fly.io/graphql (704.51ms) {"data":{"app":{"parseConfig":{"definition":{"kill_timeout":5,"kill_signal":"SIGTERM","processes":[],"deploy":{"release_command":"/app/bin/migrate"},"experimental":{"allowed_public_ports":[],"auto_rollback":true},"services":[{"processes":["app"],"protocol":"tcp","internal_port":8080,"concurrency":{"soft_limit":20,"hard_limit":25,"type":"connections"},"ports":[{"port":80,"handlers":["http"]},{"port":443,"handlers":["tls","http"]}],"tcp_checks":[{"interval":"15s","timeout":"2s","grace_period":"1s","restart_limit":0}],"http_checks":[],"script_checks":[]}],"env":{"PHX_HOST":"vanilla166.fly.dev","PORT":"8080"}},"valid":true,"errors":[],"services":[{"description":"TCP 80/443 ⇢ 8080"}]}}}}
--> Validating app configuration done
Services
TCP 80/443 ⇢ 8080
DEBUG trying remote docker daemon
DEBUG Trying 'Buildpacks' strategy
DEBUG no buildpack builder configured, skipping
DEBUG result image:<nil> error:<nil>
DEBUG Trying 'Dockerfile' strategy
DEBUG --> POST https://api.fly.io/graphql {{"query":"mutation($input: EnsureMachineRemoteBuilderInput!) { ensureMachineRemoteBuilder(input: $input) { machine { id state ips { nodes { family kind ip } } }, app { name organization { slug } } } }","variables":{"input":{"appName":"vanilla166","organizationId":null}}}
}
DEBUG <-- 200 https://api.fly.io/graphql (2.96s) {"data":{"ensureMachineRemoteBuilder":{"machine":{"id":"7175a0e0","state":"starting","ips":{"nodes":[{"family":"v6","kind":"privatenet","ip":"fdaa:0:4152:a7b:1a:0:901d:2"},{"family":"v6","kind":"public","ip":"2604:1380:4111:1e04::901d:3"},{"family":"v4","kind":"private","ip":"172.19.0.98"}]}},"app":{"name":"fly-builder-winter-wind-513","organization":{"slug":"personal"}}}}}
DEBUG checking ip &{Family:v6 Kind:privatenet IP:fdaa:0:4152:a7b:1a:0:901d:2 MaskSize:0}
DEBUG --> POST https://api.fly.io/graphql {{"query":"query ($appName: String!) { app(name: $appName) { id name hostname deployed status version appUrl config { definition } organization { id slug } services { description protocol internalPort ports { port handlers } } ipAddresses { nodes { id address type createdAt } } imageDetails { repository version } } }","variables":{"appName":"vanilla166"}}
}
Waiting for remote builder fly-builder-winter-wind-513... connecting ⣻ DEBUG <-- 200 https://api.fly.io/graphql (718.44ms) {"data":{"app":{"id":"vanilla166","name":"vanilla166","hostname":"vanilla166.fly.dev","deployed":false,"status":"pending","version":0,"appUrl":null,"config":{"definition":{"kill_timeout":5,"kill_signal":"SIGINT","processes":[],"experimental":{"allowed_public_ports":[],"auto_rollback":true},"services":[{"processes":["app"],"protocol":"tcp","internal_port":8080,"concurrency":{"soft_limit":20,"hard_limit":25,"type":"connections"},"ports":[{"port":80,"handlers":["http"]},{"port":443,"handlers":["tls","http"]}],"tcp_checks":[{"interval":"15s","timeout":"2s","grace_period":"1s","restart_limit":0}],"http_checks":[],"script_checks":[]}],"env":{}}},"organization":{"id":"60L7BO3NPGLMXHvjvN6nmoY6nAFq08","slug":"personal"},"services":[{"description":"TCP 80/443 ⇢ 8080","protocol":"TCP","internalPort":8080,"ports":[{"port":80,"handlers":["HTTP"]},{"port":443,"handlers":["TLS","HTTP"]}]}],"ipAddresses":{"nodes":[]},"imageDetails":{"repository":"unknown","version":"unknown"}}}}
DEBUG --> POST https://api.fly.io/graphql {{"query":"mutation($input: ValidateWireGuardPeersInput!) { validateWireGuardPeers(input: $input) { invalidPeerIps } }","variables":{"input":{"peerIps":["fdaa:0:4152:a7b:177d:0:a:2"]}}}
}
Waiting for remote builder fly-builder-winter-wind-513... connecting ⢿ DEBUG <-- 200 https://api.fly.io/graphql (149.08ms) {"data":{"validateWireGuardPeers":{"invalidPeerIps":[]}}}
DEBUG started agent process 46338
Waiting for remote builder fly-builder-winter-wind-513... connecting ⢿

Thanks to debugging help from @kim, we tracked this down to a regression in how flyctl handles wireguard peers that have been reaped in our backend, but are still active locally.

While we’re working on a fix, here’s a way you can try to fix it for now:

  1. Stop the fly agent with fly agent stop
  2. Remove the wireguard peer entry from ~/.fly/config.yml for the organization you’re having trouble with, including the org name key.
  3. Start the agent and try fly ping -a orgname

If this works, deploys should work as well.
2. Stop the

1 Like

Silly me, I was using fly tool on Windows Subsystem for Linux. Your post about Apple M1 pointed me to the right direction. Once I’ve installed the native fly toolkit for Windows, everything started working.

2 Likes

I’m having exactly the same issue and the same errors as Kim, except when I do fly wireguard status I get Alive: false.

Also trying to fix it doesn’t seem to help. Or I’m doing it wrong:

access_token: x
wire_guard_state:
  personal:
    org: personal
    name: x
    region: ams
    localpublic: x
    localprivate: x
    dns: ""
    peer:
      peerip: x
      endpointip: ams1.gateway.6pn.dev
      pubkey: x

So I have to remove this part from config.yaml, right?

  personal:
    org: personal
    name: x
    region: ams
    localpublic: x
    localprivate: x
    dns: ""
    peer:
      peerip: x
      endpointip: ams1.gateway.6pn.dev
      pubkey: x

I’m not in a hurry btw. I’m currently just trying out fly.io :slightly_smiling_face:

Yeah, you can try: stop agent, remove that, deploy (agent will start on its own).

1 Like

Also if you could provide your peerIP, we could take a look at something on our end. That’s safe to share.

getting similar issues:

❯ fly deploy
==> Verifying app config
→ Verified app config
==> Building image
Error failed to fetch an image or build from source: failed building options: failed probing “personal”: context deadline exceeded

❯ fly destroy fly-builder-holy-sunset-5566

❯ fly ping -o personal

ping seems to hang

CMD+C then I try deploying again, same as first output…

Last week, we deployed a fix that should have solved this problem for most people. Which OS are you running?

In the cases where it still happens, there is likely an issue with networking. Can you confirm if you are running over a VPN of some kind? If so, turn it off and try again. Also try these commands:

fly doctor
fly agent stop; fly agent start

Then trying fly ping -o personal again.

Probe (app) failed probing personal: failed probing "personal": context deadline exceeded

bit more trial and error seemed todo it

I have the same error:

❯ fly deploy
==> Verifying app config
--> Verified app config
==> Building image
Error failed to fetch an image or build from source: error connecting to docker: failed building options: failed probing "personal": context deadline exceeded
❯ fly version
fly v0.0.320 darwin/amd64 Commit: 58aae1e BuildDate: 2022-04-13T15:58:08Z

Tried all the methods above but no luck, is there any information I can provide to help? Thanks

1 Like

Make sure you have the latest flyctl, then try fly wireguard websockets enable. After that, try fly deploy again and see if it helped. This will connect to Fly over TCP 443 instead of the default UDP used by Wireguard.

1 Like

Thank you! It works.

For anyone who arrives here by googling the same problem.

I was getting this error and none of the above helped. What fixed it in the end:

fly wireguard list
fly wireguard reset
fly wireguard remove

When removing, select the one that was initially listed as the output of the list command.

7 Likes

For some reason, for me it helped running as sudo.
After everything else mentioned didn’t help.

macOS Monterey 12.0.1

1 Like

Running as sudo worked for me!

I just setup the cli and project yesterday, but running fly deploy today resulted in:

❯ fly deploy --verbose
Update available 0.0.328 -> v0.0.330.
Run "fly version update" to upgrade.
==> Verifying app config
--> Verified app config
==> Building image
Remote builder fly-builder-rough-sun-5961 ready
==> Creating build context
--> Creating build context done
==> Building image with Docker
--> docker host: 20.10.12 linux x86_64
[+] Building 8.8s (0/1)                                                                                
[+] Building 0.9s (11/11) FINISHED                                                                     
 => [internal] load remote build context                                                          0.0s
 => copy /context /                                                                               0.4s
 => [internal] load metadata for docker.io/library/python:3.7                                     0.3s
 => [1/7] FROM docker.io/library/python:3.7@sha256:11c044f967935362ef8291811ed01b0708a40668c43c8  0.0s
 => CACHED [2/7] RUN apt-get update && apt-get install -y     python3-pip     python3-venv     p  0.0s
 => CACHED [3/7] RUN mkdir -p /app                                                                0.0s
 => CACHED [4/7] WORKDIR /app                                                                     0.0s
 => CACHED [5/7] COPY requirements.txt .                                                          0.0s
 => CACHED [6/7] RUN pip install -r requirements.txt                                              0.0s
 => CACHED [7/7] COPY . .                                                                         0.0s
 => exporting to image                                                                            0.0s
 => => exporting layers                                                                           0.0s
 => => writing image sha256:34120b474913e62aa67f70f544979329c0daa8793bb6553d80a18e54621f041f      0.0s
 => => naming to registry.fly.io/justrecipe:deployment-1653614388                                 0.0s
--> Building image done
==> Pushing image to fly
The push refers to repository [registry.fly.io/justrecipe]
3a4016ee9c2c: Retrying in 1 second 
22c7b3c7d04d: Retrying in 1 second 
056d54c04b13: Retrying in 1 second 
5f70bf18a086: Retrying in 1 second 
7752ae6b5fd7: Retrying in 1 second 
00800d747256: Waiting 
2e0764ceda8b: Waiting 
3f5d38b4936d: Waiting 
7be8268e2fb0: Waiting 
b889a93a79dd: Waiting 
9d4550089a93: Waiting 
a7934564e6b9: Waiting 
1b7cceb6a07c: Waiting 
b274e8788e0c: Waiting 
78658088978a: Waiting 
Error failed to fetch an image or build from source: error rendering push status stream: received unexpected HTTP status: 500 Internal Server Error

Running the wirguard remove commands changed the error to:

❯ fly deploy --verbose
Update available 0.0.328 -> v0.0.330.
Run "fly version update" to upgrade.
==> Verifying app config
--> Verified app config
==> Building image
Error failed to fetch an image or build from source: error connecting to docker: failed building options: failed probing "personal": context deadline exceeded

But sudo worked, maybe because I installed fly with homebrew? not sure yet

Edit: removed the homebrew installed fly cli and installed it via the curl | sh method, now getting the following error which even sudo doesn’t fix:

❯ fly deploy                                                                                    33.06s
==> Verifying app config
--> Verified app config
==> Building image
Remote builder fly-builder-rough-sun-5961 ready
==> Creating build context
--> Creating build context done
==> Building image with Docker
--> docker host: 20.10.12 linux x86_64
[+] Building 4.1s (0/1)                                                                                
[+] Building 1.6s (11/11) FINISHED                                                                     
 => [internal] load remote build context                                                          0.0s
 => copy /context /                                                                               0.4s
 => [internal] load metadata for docker.io/library/python:3.7                                     0.1s
 => [1/7] FROM docker.io/library/python:3.7@sha256:11c044f967935362ef8291811ed01b0708a40668c43c8  0.0s
 => CACHED [2/7] RUN apt-get update && apt-get install -y     python3-pip     python3-venv     p  0.0s
 => CACHED [3/7] RUN mkdir -p /app                                                                0.0s
 => CACHED [4/7] WORKDIR /app                                                                     0.0s
 => CACHED [5/7] COPY requirements.txt .                                                          0.0s
 => CACHED [6/7] RUN pip install -r requirements.txt                                              0.0s
 => [7/7] COPY . .                                                                                0.4s
 => exporting to image                                                                            0.4s
 => => exporting layers                                                                           0.4s
 => => writing image sha256:ba4c850eb788d3bf272efc515c2f67f01e66a90267e5be3290e3a1c5dd7ec6d6      0.0s
 => => naming to registry.fly.io/justrecipe:deployment-1653615561                                 0.0s
--> Building image done
==> Pushing image to fly
The push refers to repository [registry.fly.io/justrecipe]
f66f68f9ff4f: Preparing 
22c7b3c7d04d: Preparing 
056d54c04b13: Preparing 
5f70bf18a086: Preparing 
7752ae6b5fd7: Preparing 
00800d747256: Waiting 
2e0764ceda8b: Waiting 
3f5d38b4936d: Waiting 
7be8268e2fb0: Waiting 
b889a93a79dd: Waiting 
9d4550089a93: Waiting 
a7934564e6b9: Waiting 
1b7cceb6a07c: Waiting 
b274e8788e0c: Waiting 
78658088978a: Waiting 
Error failed to fetch an image or build from source: error rendering push status stream: unauthorized: not allowed

Update: it’s working today :man_shrugging:t2:

2 Likes