flyctl deploy --local-only - "error building: failed to dial gRPC: http: invalid Host header"

Questions / Help Build debugging
1

Hey!

Up until a few days ago, fly deploy --local-only was working just fine.

Not sure what exactly changed but as of 2 days ago, i keep getting this error: “error building: failed to dial gRPC: http: invalid Host header.”

Here is an output of the logs just when it crashes:

 "query": "\nmutation ResolverFinishBuild ($input: FinishBuildInput!) {\n\tfinishBuild(input: $input) {\n\t\tid\n\t\tstatus\n\t\twallclockTimeMs\n\t}\n}\n",
  "variables": {
    "input": {
      "appName": "****",
      "buildId": "2642672",
      "builderMeta": {
        "builderType": "local",
        "buildkitEnabled": true,
        "dockerVersion": "24.0.2",
        "platform": "linux/aarch64/",
        "remoteAppName": "",
        "remoteMachineId": ""
      },
      "clientMutationId": "",
      "finalImage": {
        "id": "",
        "sizeBytes": 0,
        "tag": ""
      },
      "logs": "error building: failed to dial gRPC: http: invalid Host header",
      "machineId": "",
      "status": "failed",
      "strategiesAttempted": [
        {
          "error": "",
          "note": "no buildpack builder configured, skipping",
          "result": "failed",
          "strategy": "Buildpacks"
        },
        {
          "error": "error building: failed to dial gRPC: http: invalid Host header",
          "note": "",
          "result": "failed",
          "strategy": "Dockerfile"
        }
      ],
      "timings": {
        "buildAndPushMs": 7527,
        "buildMs": 7522,
        "builderInitMs": 0,
        "contextBuildMs": 2,
        "imageBuildMs": 7519,
        "pushMs": -1
      }
    }
  },
  "operationName": "ResolverFinishBuild"
}

Please note that i can build with docker locally directly but cant seem to build locally with fly.

I deleted my builder machine to see if that helps but no luck.

flyctl deploy works (i.e. with remote builder).

normally id be fine with this workaround but it takes so much more time to deploy with remote builder than with --local-only so would love to get this resolved.

Thank you!

1 Like
2

I’m facing a similar issue when running the command: fly deploy --local-only

=> ERROR resolve image config for docker.io/docker/dockerfile:1 5.8s

resolve image config for docker.io/docker/dockerfile:1:

Error: failed to fetch an image or build from source: error building: failed to dial gRPC: http: invalid Host header

Please suggest how to resolve this. Thanks

3

@Hardik Unfortunately, I am still facing the same issue.

I’ve tried it on new apps and other existing apps and it’s the same problem.

Perhaps someone from the fly team can help. It’s not the end of the world, but --local-only is a much faster option to deploy so would love to get that back.

1 Like
4

Yes It would be great if someone from the Fly team can help us resolve this. Thanks

5

Hey what’s the output of fly version?

Once you check that, worth trying fly version update.

We had a version just over a week ago that accidentally bumped a patch version of Go that introduced a slight regression on some of the Docker tooling

If you’re experiencing it using local-only, I believe I saw that Docker itself was affected at one point, it might be worth making sure you’re up to date on Docker as well if you’re looking to use local-only.

1 Like
6

Hi!

fly version: fly v0.1.58 darwin/arm64 Commit: 82ebecb2bf83647d20c0e11e51272418f40a9b2c-dirty BuildDate: 2023-07-14T18:28:59Z

When I run “fly version update”, i get: “Already running latest flyctl v0.1.58”

I just tried fly deploy --local-only and I’m still getting the error described in my original post.

Regarding docker, I’m on the latest version of docker desktop for mac.

Version - 4.21.1 (114176)
Engine: 24.0.2

Let me know if there’s anything else you need. Would love to get this issue resolved.

Thanks for your reply @jphenow

1 Like
7

Very strange! What happens if you use the remote builder? You can do that by not explicitly using --local-only or using --remote-only

1 Like
8

Using the remote builder works fine.

The only issue i have with it is that most often it’s much slower. Specifically the phase where it’s “sending build context to docker daemon”. Even with no changes to any docker layers, that phase can sometimes take 100-300 seconds. Sometimes it’s almost as fast as local-only but most often it’s slower / much slower. Is there a reason why this might be?

And regarding the local-only issue, any suggestions i can try on my end?

1 Like
9

fly doctor OUTPUT:

Testing authentication token... PASSED
Testing flyctl agent... PASSED
Testing local Docker instance... PASSED
Pinging WireGuard gateway (give us a sec)... PASSED

App specific checks for spring-sunset-4701:
Checking that app has ip addresses allocated... Nope
        No ip addresses assigned to this app. If the app is not intended to receive traffic, this is fine.
        Otherwise, it likely means that the services configuration is not correctly setup to receive http, tls, tcp, or udp traffic.
        https://fly.io/docs/reference/configuration/#the-services-sections
No ipv4 or ipv6 ip addresses allocated to app spring-sunset-4701
Build checks for spring-sunset-4701:
Checking docker context size (this may take little bit)... PASSED (238 kB)
Checking for .dockerignore... PASSED

fly version update output:

Already running latest flyctl v0.1.58

I’m using the latest version of docker for apple silicon chip.

I’m still getting the same error.

10 
=> ERROR [build 3/4] RUN npm ci                                                                                       0.9s
------                                                                                                                      
 > [build 3/4] RUN npm ci:                                                                                                  
#10 0.900 npm ERR! code ERESOLVE                                                                                            
#10 0.903 npm ERR! ERESOLVE could not resolve                                                                               
#10 0.903 npm ERR!                                                                                                          
#10 0.903 npm ERR! While resolving: mongoose-currency@0.2.0                                                                 
#10 0.903 npm ERR! Found: mongoose@7.3.4
#10 0.903 npm ERR! node_modules/mongoose
#10 0.903 npm ERR!   mongoose@"^7.3.1" from the root project
#10 0.903 npm ERR! 
#10 0.903 npm ERR! Could not resolve dependency:
#10 0.903 npm ERR! peer mongoose@"~> 4.x" from mongoose-currency@0.2.0
#10 0.903 npm ERR! node_modules/mongoose-currency
#10 0.904 npm ERR!   mongoose-currency@"^0.2.0" from the root project
#10 0.904 npm ERR! 
#10 0.904 npm ERR! Conflicting peer dependency: mongoose@4.13.21
#10 0.904 npm ERR! node_modules/mongoose
#10 0.904 npm ERR!   peer mongoose@"~> 4.x" from mongoose-currency@0.2.0
#10 0.904 npm ERR!   node_modules/mongoose-currency
#10 0.904 npm ERR!     mongoose-currency@"^0.2.0" from the root project
#10 0.904 npm ERR! 
#10 0.904 npm ERR! Fix the upstream dependency conflict, or retry
#10 0.904 npm ERR! this command with --force or --legacy-peer-deps
#10 0.904 npm ERR! to accept an incorrect (and potentially broken) dependency resolution.
#10 0.904 npm ERR! 
#10 0.904 npm ERR! 
#10 0.904 npm ERR! For a full report see:
#10 0.904 npm ERR! /root/.npm/_logs/2023-07-18T19_26_13_063Z-eresolve-report.txt
#10 0.905 
#10 0.906 npm ERR! A complete log of this run can be found in:
#10 0.906 npm ERR!     /root/.npm/_logs/2023-07-18T19_26_13_063Z-debug-0.log
------
Error: failed to fetch an image or build from source: error building: executor failed running [/bin/sh -c npm ci]: exit code: 1

I’m getting this error when I try using the remote builder. Can you suggest a way to fix this?

11

Hey I was trying to see if I could get it reproducing locally - I haven’t been able to yet.

Out of curiosity, does it build fine if you do a local docker build directly using docker? Just want to narrow our scope a bit

12

Yep, it builds fine.

What version of docker are you on and is it mac or windows?

For further context, i just git cloned the repo used here: Run a Node App · Fly Docs

And tried fly --local-only and ran into the same issue.

13

Mac with M1! I updated my Docker yesterday to 4.20.1 but I see I have an update now again, let me update again.

14

M2 pro here, with latest Docker Desktop (4.21.1 (114176)) and finding the same issue;

15

@Moreno @fittd How did you install flyctl? From Homebrew or https://fly.io/install.sh?

16

Similar issue here. There has been no changes to the fly.toml or Dockerfiles. Both Docker & Fly are on the latest version, and neither remote nor --local-only builds work since around 2pm PT. It did work last time 12 hours ago ~2am PT.

Error on Local:
Error: failed to fetch an image or build from source: error building: failed to dial gRPC: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: http: invalid Host header"

Error on remote:
Error: failed to fetch an image or build from source: error building: failed to solve: rpc error: code = Canceled desc = grpc: the client connection is closing

UPDATE: Issue has been resolved as mysteriously as it appeared. I assume hotfixed?

17

@kaz I had flyctl installed via brew.

I just uninstalled it and used the install script and --local-only seems to complete (in terms of building) but it fails at the very end after all layers are pushed. This is the error im getting now:

DEBUG {0x140016546f0}
DEBUG <-- 200 https://api.fly.io/graphql (1.43s)

DEBUG {
  "data": {
    "createRelease": null
  },
  "errors": [
    {
      "message": "Validation failed: Image must be amd64 architecture for linux us",
      "locations": [
        {
          "line": 4,
          "column": 2
        }
      ],
      "path": [
        "createRelease"
      ],
      "extensions": {
        "code": "UNPROCESSABLE"
      }
    }
  ]
}

DEBUG Task manager done
Error: input:4: createRelease Validation failed: Image must be amd64 architecture for linux us

For more context - i re-installed fly with brew and tried again. I’m getting the same general error but with a slightly different message:

Error: failed to fetch an image or build from source: error building: failed to dial gRPC: rpc error: code = Unavailable desc = connection error: desc = “transport: Error while dialing: http: invalid Host header”

18

That last validation error appears to be more specific to docker+m2.

Are you specifying a platform on your Dockerfile?

19

Thanks for the information!

So I believe Homebrew’s flyctl is built with Go 1.20.5 and having http: invalid Host header issue. flyctl >= v0.1.59 broke --local-only on Apple silicon Macs. I’m fixing the second issue right now.

Meanwhile,

flyctl >= v0.1.59 made this context building part much faster than the previous versions. Could you try remote builder once again with the latest flyctl?

20

No, I wasn’t but that fixed the issue. Thanks a ton!