Failed to start remote builder heartbeat

1

Hi, I am unable to deploy my application today. I deleted the builder and it doesn’t seem to be able to create a new one.

$ fly deploy -c fly-staging.toml --remote-only
==> Verifying app config
--> Verified app config
==> Building image
WARN Failed to start remote builder heartbeat: You hit a Fly API error with request ID: 01GQVR588P4KEFXVQNM5AWHV8W-sin
Error failed to fetch an image or build from source: error connecting to docker: You hit a Fly API error with request ID: 01GQVR59HYKP164A1JECWCSKYK-sin


error Command failed with exit code 1.

$ fly doctor -a omitted
Testing authentication token... PASSED
Testing flyctl agent... PASSED
Testing local Docker instance... Nope
Pinging WireGuard gateway (give us a sec)... PASSED

App specific checks for omitted:
Checking that app has ip addresses allocated... PASSED
Checking AAAA record for omitted.fly.dev... PASSED
Oops, something went wrong! Could you try that again?

Does anyone have an idea on how to fix this issue? or is one of fly’s services down? I can’t see any issues at https://status.flyio.net/

1 Like
2

Fly services could be down… but if they aren’t, see if any of these steps help:

3

Same. Can’t seem to deploy anything. Have been trying for the last 2-3h

4

@ignoramous thanks for the reply.

Update here: I didn’t do anything but tried again today and a new builder is now created but gets stuck on sin [info] Pulling container image.

2 Likes
5

The issue doesn’t look like its resolved yet. Still seeing this:
fly deploy
==> Verifying app config
→ Verified app config
==> Building image
WARN Failed to start remote builder heartbeat: You hit a Fly API error with request ID: 01GQXVZ20B595FP7ZB3N6MH80D-maa

even after deleting the builder and my app

6

Update: my builds are working again.

Response from Eli from Fly.io support regarding my specific issue which may help with future issues if you are outside the US:

You may have already seen this (Fly.io Status - SIN deployment / connectivity issues) but we recently were able to resolve some significant packet loss out of SIN with our upstream providers, so it’s possible that your deploys are working better now-- it looks like your current builder is in SIN, so it might have been affected by this issue.

If not, you may want to try moving your machine to a region in the US, closer to our registry origin. Currently, builders are created in the region closest to you. We’re currently working on a built-in solution, but at the moment you might have better luck manually moving your builder to the US:

  1. Make sure you have a recent flyctl (v0.0.443 at least, safer to get the very latest though)
  2. Find your builder machine ID via: fly machine ls -a <your-fly-builder-app-name>
  3. Clone the machine in a new region: fly machine clone <machine-id> --region <some-US-region> -a <your-fly-builder-app-name>
  4. Destroy your previous machine (fly machine remove <old-machine-id>)
  5. Destroy your old volume (fly volume delete <old-volume-id>)
4 Likes
7

awesome, trashing the builder worked for me. I needed to add --force to the destroy steps to clean things up.

1 Like
8

Destroying the builder worked for me too!

To find the name of your builder:
fly apps list

Mine was something like “fly-builder-strange-feather-2712”.

Then, I ran:
fly apps destroy fly-builder-strange-feather-2712

and redeployed.

Yay!