I upgraded flyctl this morning, and am blocked on deploying any of my apps (deploy was working prior to deploy). Output:
WARN Failed to start remote builder heartbeat: failed building options: failed probing “{project}-610”: context deadline exceeded
Error: failed to fetch an image or build from source: error connecting to docker: failed building options: failed probing “{project}-610”: context deadline exceeded
I tried fly auth logout/fly auth login, which did not fix it.
The builder machine seems to be in a tight loop; the monitoring output shows:
2023-06-08T14:23:47.017 app[3d8d0d2f13e289] mia [info] time=“2023-06-08T14:23:47.017415314Z” level=debug msg=“checking docker activity”
2023-06-08T14:23:47.025 app[3d8d0d2f13e289] mia [info] time=“2023-06-08T14:23:47.017749667Z” level=debug msg=“Calling GET /v1.41/containers/json?filters=%7B%22status%22%3A%7B%22running%22%3Atrue%7D%7D&limit=0”
2023-06-08T14:23:48.027 app[3d8d0d2f13e289] mia [info] time=“2023-06-08T14:23:48.026800122Z” level=debug msg=“checking docker activity”
2023-06-08T14:23:48.027 app[3d8d0d2f13e289] mia [info] time=“2023-06-08T14:23:48.027233485Z” level=debug msg=“Calling GET /v1.41/containers/json?filters=%7B%22status%22%3A%7B%22running%22%3Atrue%7D%7D&limit=0”
2023-06-08T14:23:49.028 app[3d8d0d2f13e289] mia [info] time=“2023-06-08T14:23:49.028393064Z” level=debug msg=“checking docker activity”
2023-06-08T14:23:49.028 app[3d8d0d2f13e289] mia [info] time=“2023-06-08T14:23:49.028684666Z” level=debug msg=“Calling GET /v1.41/containers/json?filters=%7B%22status%22%3A%7B%22running%22%3Atrue%7D%7D&limit=0”
2023-06-08T14:23:50.030 app[3d8d0d2f13e289] mia [info] time=“2023-06-08T14:23:50.029793714Z” level=debug msg=“checking docker activity”
2023-06-08T14:23:50.030 app[3d8d0d2f13e289] mia [info] time=“2023-06-08T14:23:50.030231577Z” level=debug msg=“Calling GET /v1.41/containers/json?filters=%7B%22status%22%3A%7B%22running%22%3Atrue%7D%7D&limit=0”
I ran into this issue again on 0.1.32, so I guess the upgrade only fixed it by restarting the agent or resetting some state. Ricardo’s steps fixed it for me this time.