GitHub action WARN Failed to start remote builder heartbeat: server returned a non-200 status code: 504

Questions / Help
1

This is consistently happening now.

I’ve tried fixing the flyctl version in my workflow:

      - name: Load flyctl
        uses: superfly/flyctl-actions/setup-flyctl@master
        with:
          version: 0.1.58

Here’s the output:

Secrets have been staged, but not set on VMs. Deploy or update machines in this app for the secrets to take effect.
==> Verifying app config
--> Verified app config
Validating /home/runner/work/XXX/XXX/fly.toml
Platform: machines
✓ Configuration is valid
==> Building image
WARN Failed to start remote builder heartbeat: server returned a non-200 status code: 504

Error: failed to fetch an image or build from source: error connecting to docker: server returned a non-200 status code: 504

Error: Process completed with exit code 1.
1 Like
2

Hi there, sorry about this. We opened a statuspage incident a few moments ago and are investigating an issue with our secrets vault availability. I would guess this is related. We’ll post updates here: https://status.flyio.net/

3

Thanks @jphenow, would this explain why this was happening for over 2 days though?

Has the secrets vault issue been happening for days?

4

Aha no it hasn’t - that’s strange.

Could post the output of going to https://debug.fly.dev?

5

This is running through GitHub actions though, I enabled LOG_LEVEL debug in my workflow:

==> Building image
DEBUG --> POST https://api.fly.io/graphql

DEBUG {
  "query": "mutation($input: EnsureMachineRemoteBuilderInput!) { ensureMachineRemoteBuilder(input: $input) { machine { id state ips { nodes { family kind ip } } }, app { name organization { id slug } } } }",
  "variables": {
    "input": {
      "appName": "XXX",
      "organizationId": null
    }
  }
}


DEBUG {}
DEBUG <-- 504 https://api.fly.io/graphql (1m0.04s)

DEBUG <html><body><h1>504 Gateway Time-out</h1>
WARN Failed to start remote builder heartbeat: server returned a non-200 status code: 504
The server didn't respond in time.
</body></html>

DEBUG --> POST https://api.fly.io/graphql

DEBUG {
  "query": "\nmutation ResolverCreateBuild ($input: CreateBuildInput!) {\n\tcreateBuild(input: $input) {\n\t\tid\n\t\tstatus\n\t}\n}\n",
  "variables": {
    "input": {
      "appName": "XXX",
      "builderType": "remote",
      "clientMutationId": "",
      "imageOpts": {
        "buildArgs": {
          "PHP_VERSION": "8.2"
        },
        "buildPacks": null,
        "builder": "",
        "builtIn": "",
        "builtInSettings": null,
        "dockerfilePath": "/home/runner/work/XXX/XXX/.fly/apps/XXX/Dockerfile",
        "extraBuildArgs": null,
        "imageLabel": "",
        "imageRef": "",
        "noCache": false,
        "publish": true,
        "tag": "registry.fly.io/XXX:deployment-01H5WQQQQDZYFJQRWSG0Y6BSGV",
        "target": ""
      },
      "machineId": "",
      "strategiesAvailable": [
        "Buildpacks",
        "Dockerfile",
        "Builtin"
      ]
    }
  },
  "operationName": "ResolverCreateBuild"
}

DEBUG {0xc00015ac60}

DEBUG <-- 200 https://api.fly.io/graphql (78.52ms)

DEBUG {
  "data": {
    "createBuild": {
      "id": "2737349",
      "status": "started"
    }
  }
}

DEBUG --> POST https://api.fly.io/graphql

DEBUG {
  "query": "mutation($input: EnsureMachineRemoteBuilderInput!) { ensureMachineRemoteBuilder(input: $input) { machine { id state ips { nodes { family kind ip } } }, app { name organization { id slug } } } }",
DEBUG Trying 'Buildpacks' strategy

DEBUG no buildpack builder configured, skipping
DEBUG result image:<nil> error:<nil>

DEBUG Trying 'Dockerfile' strategy

  "variables": {
    "input": {
      "appName": "XXX",
      "organizationId": null
    }
  }
}


DEBUG {}
DEBUG <-- 504 https://api.fly.io/graphql (1m0.04s)
DEBUG result image:<nil> error:error connecting to docker: server returned a non-200 status code: 504


DEBUG <html><body><h1>504 Gateway Time-out</h1>
The server didn't respond in time.
</body></html>

DEBUG --> POST https://api.fly.io/graphql

DEBUG {
  "query": "\nmutation ResolverFinishBuild ($input: FinishBuildInput!) {\n\tfinishBuild(input: $input) {\n\t\tid\n\t\tstatus\n\t\twallclockTimeMs\n\t}\n}\n",
  "variables": {
    "input": {
      "appName": "XXX",
      "buildId": "2737349",
      "builderMeta": {
        "builderType": "",
        "buildkitEnabled": false,
        "dockerVersion": "",
        "platform": "",
        "remoteAppName": "",
        "remoteMachineId": ""
      },
      "clientMutationId": "",
      "finalImage": {
        "id": "",
        "sizeBytes": 0,
        "tag": ""
      },
      "logs": "error connecting to docker: server returned a non-200 status code: 504",
      "machineId": "",
      "status": "failed",
      "strategiesAttempted": [
        {
          "error": "",
          "note": "no buildpack builder configured, skipping",
          "result": "failed",
          "strategy": "Buildpacks"
        },
        {
          "error": "error connecting to docker: server returned a non-200 status code: 504",
          "note": "",
          "result": "failed",
          "strategy": "Dockerfile"
        }
      ],
      "timings": {
        "buildAndPushMs": 60043,
        "buildMs": 60043,
        "builderInitMs": 60043,
        "contextBuildMs": -1,
        "imageBuildMs": -1,
        "pushMs": -1
      }
    }
  },
  "operationName": "ResolverFinishBuild"
}

DEBUG {0xc0008d6900}
DEBUG <-- 200 https://api.fly.io/graphql (121.9ms)

DEBUG {
  "data": {
    "finishBuild": {
      "id": "2737349",
      "status": "failed",
      "wallclockTimeMs": 60169
    }
  }
}

DEBUG Task manager done
Error: failed to fetch an image or build from source: error connecting to docker: server returned a non-200 status code: 504
6

Awesome that’s helpful, thank you

Could try going to that https://debug.fly.dev and paste the output here?

Edit: you said this is on github actions whoops

Edit2: give me a moment, trying to reproduce

7

Yeah, this is inside GitHub actions. I edited my debug message to include more info. It seems both the remote builder and the connection to docker time out.

8

Are you using the default GitHub runners or do you have some private ones configured?

By chance could you share some more of the workflow yml so we can get a look at the deploy command etc.?

9

Default runners. Is there a way to DM you on here so I can send the yml files? It’s a private repo

10

Essentially it’s just this, though:

env:
  FLY_API_TOKEN: ${{ secrets.FLY_API_TOKEN }}

# jobs: steps:  etc...
      - name: Load flyctl
        uses: superfly/flyctl-actions/setup-flyctl@master
        with:
          version: 0.1.58

      - name: Deploy app
        run: |
          LOG_LEVEL=debug flyctl secrets set -a ${{ inputs.app_name }} --stage KEY_1=${{ secrets.KEY_1 }} KEY_2=${{ secrets.KEY_2 }}
          LOG_LEVEL=debug flyctl deploy --app ${{ inputs.app_name }} --remote-only --env QUEUE_WORKERS=${{ inputs.queue_workers }} --env APP_URL=${{ env.APP_URL }} --env OCTANE_SERVER=${{ inputs.octane_server }} --build-arg PHP_VERSION=${{ inputs.php_version }}
11

Thanks for that - I’m not sure we have DMs setup on this forum but this should be enough to run with!

One potentially quickish option to get unblocked while I keep looking - does --local-only instead of --remote-only work for you from GitHub actions? We may need to find the action that sets up docker on your runner

I can hunt a bit to see what might have changed recently to cause this for you - I haven’t heard of this happening for others, haven’t reproduced just yet.

12

I’ll try local building, but obviously I’m trying to point out an issue with remote building where that worked consistently up until 2 days or so ago.

Am I supposed to have a remote builder machine in my account (used to with apps v1), but I’m not seeing one now.

13

--local-only does work and the deploy completes, but it’s obviously much slower than remote building.

14

Remote building just worked on my last deploy and now I also see a free builder in my account again, so I guess that got deleted and added back some how?

image

15

Yep was just hoping to rule some things out and possibly unblock your deploys for now!

Yes those builders should still work - that’s odd it popped back into place after the local build :thinking: well I’m glad it’s working now.

Will take a look to see if I can understand what happened

16

Much appreciated, @jphenow. Let me know so we can avoid this in the future :slight_smile:

17

I’ll let you know if I learn anything. Definitely post again if it comes back up in the mean time

1 Like
18

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.