Deployments not working: error connecting to docker

Also if you could provide your peerIP, we could take a look at something on our end. That’s safe to share.

getting similar issues:

❯ fly deploy
==> Verifying app config
→ Verified app config
==> Building image
Error failed to fetch an image or build from source: failed building options: failed probing “personal”: context deadline exceeded

❯ fly destroy fly-builder-holy-sunset-5566

❯ fly ping -o personal

ping seems to hang

CMD+C then I try deploying again, same as first output…

Last week, we deployed a fix that should have solved this problem for most people. Which OS are you running?

In the cases where it still happens, there is likely an issue with networking. Can you confirm if you are running over a VPN of some kind? If so, turn it off and try again. Also try these commands:

fly doctor
fly agent stop; fly agent start

Then trying fly ping -o personal again.

Probe (app) failed probing personal: failed probing "personal": context deadline exceeded

bit more trial and error seemed todo it

I have the same error:

❯ fly deploy
==> Verifying app config
--> Verified app config
==> Building image
Error failed to fetch an image or build from source: error connecting to docker: failed building options: failed probing "personal": context deadline exceeded
❯ fly version
fly v0.0.320 darwin/amd64 Commit: 58aae1e BuildDate: 2022-04-13T15:58:08Z

Tried all the methods above but no luck, is there any information I can provide to help? Thanks

1 Like

Make sure you have the latest flyctl, then try fly wireguard websockets enable. After that, try fly deploy again and see if it helped. This will connect to Fly over TCP 443 instead of the default UDP used by Wireguard.

3 Likes

Thank you! It works.

For anyone who arrives here by googling the same problem.

I was getting this error and none of the above helped. What fixed it in the end:

fly wireguard list
fly wireguard reset
fly wireguard remove

When removing, select the one that was initially listed as the output of the list command.

11 Likes

For some reason, for me it helped running as sudo.
After everything else mentioned didn’t help.

macOS Monterey 12.0.1

1 Like

Running as sudo worked for me!

I just setup the cli and project yesterday, but running fly deploy today resulted in:

❯ fly deploy --verbose
Update available 0.0.328 -> v0.0.330.
Run "fly version update" to upgrade.
==> Verifying app config
--> Verified app config
==> Building image
Remote builder fly-builder-rough-sun-5961 ready
==> Creating build context
--> Creating build context done
==> Building image with Docker
--> docker host: 20.10.12 linux x86_64
[+] Building 8.8s (0/1)                                                                                
[+] Building 0.9s (11/11) FINISHED                                                                     
 => [internal] load remote build context                                                          0.0s
 => copy /context /                                                                               0.4s
 => [internal] load metadata for docker.io/library/python:3.7                                     0.3s
 => [1/7] FROM docker.io/library/python:3.7@sha256:11c044f967935362ef8291811ed01b0708a40668c43c8  0.0s
 => CACHED [2/7] RUN apt-get update && apt-get install -y     python3-pip     python3-venv     p  0.0s
 => CACHED [3/7] RUN mkdir -p /app                                                                0.0s
 => CACHED [4/7] WORKDIR /app                                                                     0.0s
 => CACHED [5/7] COPY requirements.txt .                                                          0.0s
 => CACHED [6/7] RUN pip install -r requirements.txt                                              0.0s
 => CACHED [7/7] COPY . .                                                                         0.0s
 => exporting to image                                                                            0.0s
 => => exporting layers                                                                           0.0s
 => => writing image sha256:34120b474913e62aa67f70f544979329c0daa8793bb6553d80a18e54621f041f      0.0s
 => => naming to registry.fly.io/justrecipe:deployment-1653614388                                 0.0s
--> Building image done
==> Pushing image to fly
The push refers to repository [registry.fly.io/justrecipe]
3a4016ee9c2c: Retrying in 1 second 
22c7b3c7d04d: Retrying in 1 second 
056d54c04b13: Retrying in 1 second 
5f70bf18a086: Retrying in 1 second 
7752ae6b5fd7: Retrying in 1 second 
00800d747256: Waiting 
2e0764ceda8b: Waiting 
3f5d38b4936d: Waiting 
7be8268e2fb0: Waiting 
b889a93a79dd: Waiting 
9d4550089a93: Waiting 
a7934564e6b9: Waiting 
1b7cceb6a07c: Waiting 
b274e8788e0c: Waiting 
78658088978a: Waiting 
Error failed to fetch an image or build from source: error rendering push status stream: received unexpected HTTP status: 500 Internal Server Error

Running the wirguard remove commands changed the error to:

❯ fly deploy --verbose
Update available 0.0.328 -> v0.0.330.
Run "fly version update" to upgrade.
==> Verifying app config
--> Verified app config
==> Building image
Error failed to fetch an image or build from source: error connecting to docker: failed building options: failed probing "personal": context deadline exceeded

But sudo worked, maybe because I installed fly with homebrew? not sure yet

Edit: removed the homebrew installed fly cli and installed it via the curl | sh method, now getting the following error which even sudo doesn’t fix:

❯ fly deploy                                                                                    33.06s
==> Verifying app config
--> Verified app config
==> Building image
Remote builder fly-builder-rough-sun-5961 ready
==> Creating build context
--> Creating build context done
==> Building image with Docker
--> docker host: 20.10.12 linux x86_64
[+] Building 4.1s (0/1)                                                                                
[+] Building 1.6s (11/11) FINISHED                                                                     
 => [internal] load remote build context                                                          0.0s
 => copy /context /                                                                               0.4s
 => [internal] load metadata for docker.io/library/python:3.7                                     0.1s
 => [1/7] FROM docker.io/library/python:3.7@sha256:11c044f967935362ef8291811ed01b0708a40668c43c8  0.0s
 => CACHED [2/7] RUN apt-get update && apt-get install -y     python3-pip     python3-venv     p  0.0s
 => CACHED [3/7] RUN mkdir -p /app                                                                0.0s
 => CACHED [4/7] WORKDIR /app                                                                     0.0s
 => CACHED [5/7] COPY requirements.txt .                                                          0.0s
 => CACHED [6/7] RUN pip install -r requirements.txt                                              0.0s
 => [7/7] COPY . .                                                                                0.4s
 => exporting to image                                                                            0.4s
 => => exporting layers                                                                           0.4s
 => => writing image sha256:ba4c850eb788d3bf272efc515c2f67f01e66a90267e5be3290e3a1c5dd7ec6d6      0.0s
 => => naming to registry.fly.io/justrecipe:deployment-1653615561                                 0.0s
--> Building image done
==> Pushing image to fly
The push refers to repository [registry.fly.io/justrecipe]
f66f68f9ff4f: Preparing 
22c7b3c7d04d: Preparing 
056d54c04b13: Preparing 
5f70bf18a086: Preparing 
7752ae6b5fd7: Preparing 
00800d747256: Waiting 
2e0764ceda8b: Waiting 
3f5d38b4936d: Waiting 
7be8268e2fb0: Waiting 
b889a93a79dd: Waiting 
9d4550089a93: Waiting 
a7934564e6b9: Waiting 
1b7cceb6a07c: Waiting 
b274e8788e0c: Waiting 
78658088978a: Waiting 
Error failed to fetch an image or build from source: error rendering push status stream: unauthorized: not allowed

Update: it’s working today :man_shrugging:t2:

2 Likes

This issue occurs randomly for me, and theres nothign that can be done seemingly.

I think I will be switching to Cloudflare workers

Can you post your fly.toml here? Buildpacks are prone to causing this. We use buildpacks when we haven’t built specific support for a framework yet, so if you’re using something like node.js we probably defaulted to a buildpack.

We can help you work around these, though, usually pretty quickly.

But running JS on workers is totally reasonable too. Our happy path is definitely not node.js apps yet.

BTW, I ran into this “recently” (I think it started a while ago but I worked around it by doing flyctl deploy --local-only).

Deploys would fail with

Error failed to fetch an image or build from source: error connecting to docker: failed building options: failed probing "personal": context deadline exceeded

while flyctl doctor would say

Pinging WireGuard gateway (give us a sec)... FAILED
(Error: ping gateway: no response from gateway received)

Running flyctl wireguard status failed with

? Select peer: interactive-Bistromath-M1-fly-bip-filippo-io-273
Error unhandled upstream service error: nats wg stat: sql: no rows in result set

while flyctl wireguard remove failed with

Error Could not find

so it sounds a lot like

Running fly wireguard reset made flyctl doctor happy, but the build would now timeout with

WARN Remote builder did not start in time. Check remote builder logs with `flyctl logs -a fly-builder-divine-violet-4151`
Error failed to fetch an image or build from source: error connecting to docker: remote builder app unavailable

Then running fly wireguard remove interactive-Bistromath-M1-fly-bip-filippo-io-273 and flyctl apps destroy fly-builder-divine-violet-4151 made it all work. Not sure if they were both necessary.

1 Like

The ‘unavailable builder’ issue may be related to a bug we just fixed. Builders could get into a race condition where our backend wouldn’t start them up.

1 Like

fly wireguard reset also fixed it for me.

3 Likes

Something was wrong with my ISP. So connecting over VPN and then running fly wireguard reset worked for me.

1 Like

This fly wireguard websockets enable solve my problem. I was facing the same issue: “failed to fetch an image or build from source: error connecting to docker: failed building options: failed probing “personal”: context deadline exceeded”

2 Likes

Thank you!