Failed to start remote builder heartbeat: server returned a non-200 status code: 500

Getting an error with fly deploy happening on both local and github actions:

WARN Failed to start remote builder heartbeat: server returned a non-200 status code: 500

Error: failed to fetch an image or build from source: error connecting to docker: server returned a non-200 status code: 500

I tried fly apps destroy the fly-builder, same result.

This is the debug log where errors occurred

==> Building image
DEBUG trying remote docker daemon
DEBUG --> POST https://api.fly.io/graphql

DEBUG {
  "query": "mutation($input: EnsureMachineRemoteBuilderInput!) { ensureMachineRemoteBuilder(input: $input) { machine { id state ips { nodes { family kind ip } } }, app { name organization { id slug } } } }",
  "variables": {
    "input": {
      "appName": "xxxxxxx",
      "organizationId": null
    }
  }
}


DEBUG {}
DEBUG <-- 500 https://api.fly.io/graphql (4.62s)

DEBUG {
  "errors": [
    {
      "message": "You hit a Fly API error with request ID: 01HCJDCDG18P9T4WRG0RRH0SQ0-sea",
      "extensions": {
        "code": "SERVER_ERROR",
        "fly_request_id": "01HCJDCDG18P9T4WRG0RRH0SQ0-sea"
      }
    }
  ],
  "data": {}
}

WARN Failed to start remote builder heartbeat: server returned a non-200 status code: 500
DEBUG <-- 500 https://api.fly.io/graphql (2.57s)

DEBUG {
  "errors": [
    {
      "message": "You hit a Fly API error with request ID: 01HCJDCJ59Z85QYVNNBAB308BP-sea",
      "extensions": {
        "code": "SERVER_ERROR",
        "fly_request_id": "01HCJDCJ59Z85QYVNNBAB308BP-sea"
      }
    }
  ],
  "data": {}
}

DEBUG result image:<nil> error:error connecting to docker: server returned a non-200 status code: 500
DEBUG {
  "query": "\n# @genqlient\nmutation ResolverFinishBuild ($input: FinishBuildInput!) {\n\tfinishBuild(input: $input) {\n\t\tid\n\t\tstatus\n\t\twallclockTimeMs\n\t}\n}\n",
  "variables": {
    "input": {
      "appName": "xxxxxxxxxx",
      "buildId": "3849606",
      "builderMeta": {
        "builderType": "",
        "buildkitEnabled": false,
        "dockerVersion": "",
        "platform": "",
        "remoteAppName": "",
        "remoteMachineId": ""
      },
      "clientMutationId": "",
      "finalImage": {
        "id": "",
        "sizeBytes": 0,
        "tag": ""
      },
      "logs": "error connecting to docker: server returned a non-200 status code: 500",
      "machineId": "",
      "status": "failed",
      "strategiesAttempted": [
        {
          "error": "",
          "note": "no buildpack builder configured, skipping",
          "result": "failed",
          "strategy": "Buildpacks"
        },
        {
          "error": "error connecting to docker: server returned a non-200 status code: 500",
          "note": "",
          "result": "failed",
          "strategy": "Dockerfile"
        }
      ],
      "timings": {
        "buildAndPushMs": 2577,
        "buildMs": 2577,
        "builderInitMs": 2577,
        "contextBuildMs": -1,
        "imageBuildMs": -1,
        "pushMs": -1
      }
    }
  },
  "operationName": "ResolverFinishBuild"
}
6 Likes

Having same issue on my NodeJS project

I have similar issue, also with NodeJS project. My app is broken in production and I’m unable to fix it because of this.

WARN Failed to start remote builder heartbeat: Could not find Volume

Error: failed to fetch an image or build from source: error connecting to docker: Could not find Volume

Same here unfortunately! Service outage?

Hi! Just wanted to pop in and say we’re looking into this problem.

2 Likes

Thank you :smiley:

Same issue over here

Same here in NRT region…

If it helps anyone, I was able to work around this by building the docker image for my app locally (docker build -t exampleapp .) and then deploying it by doing flyctl deploy --image exampleapp --local-only

It didn’t work just by pointing it to the image I built, also had to specify --local-only.

2 Likes

Step-by-step guide on how to resolve the issues and deploy your docker image locally

  • Check Docker Images:
    First, verify that you have the Docker image you want to deploy. You can do this by running:
docker images

This command should display a list of your Docker images. Make sure the image you want to deploy is listed.

If you don’t have the Fly.io CLI (Command Line Interface) installed, you can download and install it using the following commands for Linux:

curl -L https://fly.io/install.sh | sh

This command above will install the Fly.io CLI to your system.

After installation, you may need to add the CLI’s directory to your PATH. You can do this by running the following commands:

export FLYCTL_INSTALL="/home/pencil/.fly"
export PATH="$FLYCTL_INSTALL/bin:$PATH"

Once you have the Fly.io CLI installed and your Docker image is ready, you can deploy it to Fly.io using the flyctl deploy command. Use the following command to deploy your image:

flyctl deploy --image relovista --local-only

Replace relovista with the name of your Docker image.

  • Review Deployment Output:
  1. The flyctl deploy command will provide you with information about the deployment process. Review the output for any errors or confirmations.

By following these steps, you should be able to deploy your docker image locally using fly.io. Make sure your Docker image is configured correctly for deployment, and ensure that you have the necessary dependencies installed and configured as specified in your Dockerfile and docker-compose.yml if applicable.

If you encounter any issues during the deployment process, review the error messages for troubleshooting or post your it here also for help.

Yes, experiencing same issue. Fly does not show any incidents or status issues (neither in the public nor the personal view)…

Same here. Glad to hear it is being worked on, this is currently breaking our CI.

According to status.flyio.net everything works fine but evidently we are many who are experiencing this.

Same here.

Same issue here.

same here with Golang and Docker apps. Tried both Madrid and Paris region, same result.

Restarted flyctl daemon → no change.

Removed all apps via dashboard, “launched” again → no change.

giving up for now, waiting for you to resolve the issue.

(thanx btw for your awesome app, site and service)

Same, using Go. Cannot deploy.

As an update, I was able to build and deploy later in the day yesterday, but I have not tried again. So it’s clear that it was something on fly.io’s end and not an installation or configuration problem on my side.

Hi all, sorry for the issues you’ve experienced with builders failing to create/start. We deployed some changes over the last few days that resulted in volumes getting created on hosts that were actually ineligible for them which would then prevent the machine from being created to use the volume. We’ve rolled out a final/hopeful fix for the issue and if you’re still seeing a problem, the first thing to try is destroying the builder app so it will get re-created on the next deploy.

fyi: I’m currently working from Spain.

  1. deleted all apps from fly.io via Dashboard

  2. deleted fly.toml

  3. deleted all local Docker containers somewhat related to my app

  4. flyctl deploy + launch → still the same error:

Choose a region for deployment: Madrid, Spain (mad)
App will use ‘mad’ region as primary

Created app 'arc42-xyz' in organization 'personal'
Admin URL: https://fly.io/apps/arc42-xyz
Hostname: arc42-xyz.fly.dev
? Would you like to set up a Postgresql database now? No
? Would you like to set up an Upstash Redis database now? No
Wrote config file fly.toml
? Would you like to deploy now? Yes
Validating /Users/gernotstarke/projects/arc42/xyz/fly.toml
Platform: machines
✓ Configuration is valid
==> Building image
WARN Failed to start remote builder heartbeat: server returned a non-200 status code: 500

Ok, taking a look to see what could still be happening.