`fly deploy` stuck on “Waiting for remote builder”

1 
Waiting for remote builder fly-builder-wandering-fire-1326... 🌏
WARN Remote builder did not start in time. Check remote builder logs with `flyctl logs -a fly-builder-wandering-fire-1326`
WARN Failed to start remote builder heartbeat: remote builder app unavailable

My builder isn’t doing any work now. I tried delete it and auto create it and it not work.

I try fly deploy --verbose but nothing more output.

fly version
fly v0.1.0 darwin/arm64 Commit: f39689b8 BuildDate: 2023-05-08T13:30:28Z
2

I read

but nothing help

3

Looking at fly-builder-wandering-fire-1326 logs I see that the builder started just fine.
I wonder if this might be a Wireguard connectivity issue.
Can you try to switch flyctl agent from UDP to websockets for Wireguard and try again? Like this:

$ fly agent stop
$ fly wg websockets enable
4

@pavel Thanks for your reply. I tried this but it doesn’t help.

Because of your information, I captured a little packets and thought maybe the this is the problem:

Every request

{
  "query": "mutation($input: ValidateWireGuardPeersInput!) { validateWireGuardPeers(input: $input) { invalidPeerIps } }",
  "variables": {
    "input": {
      "peerIps": [
        "xxxxx"
      ]
    }
  }
}

returns a

{
  "data": {
    "validateWireGuardPeers": {
      "invalidPeerIps": [
      ]
    }
  }
}

So I think this may not be a network problem on my side.

5

This part is correct. flyctl checks if Wireguard peers listed in its config are valid and prunes the ones that no longer exist on the backend.

Can you try this?

# stop the agent to make sure it starts a new log file
$ fly agent stop
# remove previous agent log files
$ rm ~/.fly/agent-logs/*.log
# start the builder machine
$ fly -a fly-builder-wandering-fire-1326 m start --select
# ssh into the machine
$ fly -a fly-builder-wandering-fire-1326 ssh console

SSH also goes via Wireguard tunnel. Does this work? If not, can you share agent’s log file, please?

6

@pavel This can work

image
image1540×410 78.7 KB

7

Okay. This rules out Wireguard issue.

One more thing to give more info.

When you deploy with debug log level:

$ LOG_LEVEL=debug fly deploy

there will be a lot of debug info printed, including connection attempts to the builder.
There should be some debug output after the line that looks like: Waiting for remote builder ...

Something like DEBUG Remote builder available... or DEBUG Remote builder unavailable....

Can you post them here, please? They should include the error message.

8

@pavel Hi, this is the log

Waiting for remote builder fly-builder-wandering-fire-1326... 🌏DEBUG Remote builder unavailable, retrying in 200ms (err: error during connect: Get "http://[fdaa:1:8c65:a7b:7f47:5aee:fd59:2]:2375/_ping": proxyconnect tcp: connect tcp 127.0.0.1:6152: bad local address)
DEBUG Remote builder unavailable, retrying in 200ms (err: error during connect: Get "http://[fdaa:1:8c65:a7b:7f47:5aee:fd59:2]:2375/_ping": proxyconnect tcp: connect tcp 127.0.0.1:6152: bad local address)

And I know the reason :joy: It’s because I have an HTTP_PROXY environment variable set!

However, is there any way I can use it with proxy?

9

After run HTTP_PROXT= fly deploy I build successfully!

10

Okay. That’s a great finding. This is something we should probably fix on the flyctl side.

However, is there any way I can use it with proxy?

Connections to the builder happen over the private network that flyctl connects to via its agent. It’s possible to connect your computer directly to this private network via Wireguard 1 and tell flyctl to not start its own tunnel to the builder (by setting FLY_REMOTE_BUILDER_HOST_WG=true env var), this will allow the proxy running on the same machine to talk to the builder as well.
Why do you want to connect to the builder via proxy?

11

@pavel Got it! Thank you.

Just because… I’m in China and I’m afraid one day I can’t connect to the builder without a proxy.

12

I understand.
As I said, connections to the builder happen over encrypted Wireguard tunnel automatically established by flyctl.
If it can connect to the tunnel, you don’t need HTTP proxy as the traffic already goes via VPN. If it can’t connect to the tunnel, HTTP proxy wouldn’t help unfortunately.

13

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.