Blocked on deploying after upgrading flyctl

I upgraded flyctl this morning, and am blocked on deploying any of my apps (deploy was working prior to deploy). Output:

WARN Failed to start remote builder heartbeat: failed building options: failed probing “{project}-610”: context deadline exceeded
Error: failed to fetch an image or build from source: error connecting to docker: failed building options: failed probing “{project}-610”: context deadline exceeded

I tried fly auth logout/fly auth login, which did not fix it.

The builder machine seems to be in a tight loop; the monitoring output shows:

2023-06-08T14:23:47.017 app[3d8d0d2f13e289] mia [info] time=“2023-06-08T14:23:47.017415314Z” level=debug msg=“checking docker activity”

2023-06-08T14:23:47.025 app[3d8d0d2f13e289] mia [info] time=“2023-06-08T14:23:47.017749667Z” level=debug msg=“Calling GET /v1.41/containers/json?filters=%7B%22status%22%3A%7B%22running%22%3Atrue%7D%7D&limit=0”

2023-06-08T14:23:48.027 app[3d8d0d2f13e289] mia [info] time=“2023-06-08T14:23:48.026800122Z” level=debug msg=“checking docker activity”

2023-06-08T14:23:48.027 app[3d8d0d2f13e289] mia [info] time=“2023-06-08T14:23:48.027233485Z” level=debug msg=“Calling GET /v1.41/containers/json?filters=%7B%22status%22%3A%7B%22running%22%3Atrue%7D%7D&limit=0”

2023-06-08T14:23:49.028 app[3d8d0d2f13e289] mia [info] time=“2023-06-08T14:23:49.028393064Z” level=debug msg=“checking docker activity”

2023-06-08T14:23:49.028 app[3d8d0d2f13e289] mia [info] time=“2023-06-08T14:23:49.028684666Z” level=debug msg=“Calling GET /v1.41/containers/json?filters=%7B%22status%22%3A%7B%22running%22%3Atrue%7D%7D&limit=0”

2023-06-08T14:23:50.030 app[3d8d0d2f13e289] mia [info] time=“2023-06-08T14:23:50.029793714Z” level=debug msg=“checking docker activity”

2023-06-08T14:23:50.030 app[3d8d0d2f13e289] mia [info] time=“2023-06-08T14:23:50.030231577Z” level=debug msg=“Calling GET /v1.41/containers/json?filters=%7B%22status%22%3A%7B%22running%22%3Atrue%7D%7D&limit=0”

Based on the monitoring output you provided, it seems like there might be an issue with the remote builder’s connection to the Docker daemon.

Here are a few additional steps you can try to resolve the issue. If 1st step doesn’t fix it for you, try the next step and so on.

  1. Double-check your internet connection and ensure that it’s stable.

  2. Ensure that your local Docker daemon is running, accessible and up-to-date.

  3. Check if there are any firewalls or security settings that might be blocking the connection between the remote builder and the Docker daemon.

  4. Clear any Docker cache or temporary files that might be causing issues.

  5. Try restarting your Docker daemon.

  6. Try building using remote builder by adding the --remote-only flag:

fly deploy --remote-only
  1. If the issue persists, you can try downgrading flyctl to the previous version that was working for you.

  2. If the issue persists, try deploying your application from a different machine or network to see if the issue persists.

You can also try the steps listed here:

All the way down to:

fly deploy --remote-only

…fails.

How do I do this?

And I have tried deleting the builder - didn’t work.

What version are you on? Run flyctl version to see current version.

To downgrade use:

curl -L https://fly.io/install.sh | sh -s -- v0.1.27

Replace v0.1.27 with desired version

And just to confirm, you stopped and restarted Docker locally and it still didn’t fix it?

:wave:t3: I had the same issue, and this worked for me:

fly wireguard reset
fly agent restart

I hope it helps :crossed_fingers:t3:

for me it’s not working, still same error;/ only local build is working

I upgraded flyctl again this morning to 0.1.32 and that seemed to fix it. I think it was a bug in 0.1.31?

I’m having similar issues, on flyctl 0.1.32, unfortunately. I deployed fine yesterday, but for some reason today I can’t deploy. Similar error.

I ran into this issue again on 0.1.32, so I guess the upgrade only fixed it by restarting the agent or resetting some state. Ricardo’s steps fixed it for me this time.

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.