INFO Waiting for remote builder to become available...hangs

$ fly deploy
INFO Local docker unavailable, hooking you up with a remote Docker builder…
INFO Waiting for remote builder to become available…

5 minutes pass

^C
WARN Canceled
Error error during connect: Get “http://docker/_ping”: command [ssh -o StrictHostKeyChecking=no -l -p 10000 – fly-builder-patient-feather-6343.fly.dev docker system dial-stdio] has exited with exit status 255, please make sure the URL is valid, and Docker 18.09 or later is installed on the remote host: stderr=ControlPath too long (’/home/gavin/.ssh/master-thompson-@fly-builder-patient-feather-6343.fly.dev:10000’ >= 108 bytes)

When I removed the ControlPath option from my ssh config and ran deploy again, it worked. If there’s a problem with my ssh config I should at least get an error, not a hang. Also, what are you doing with ssh in the first place?

We expose SSH from remote builders, Docker proxies to the remote daemon through it. The Docker internals use system SSH for this, which is a bit silly. @michael was looking at workarounds for this.

Also, it’s possible the remote builder took too long to start listening. Will you give it a try with the ControlPath option and see if it happens again?

Oh gosh I typed up a reply last night then Wandavision came on and I forgot to hit send :man_facepalming:

We’re exposing the docker daemon port via an ssh proxy for auth inspired by the docs. The code is on github if you’re curious. The docker client uses your system ssh and it’s config which can cause problems like you saw. We’ll have a fix in a few days. Sorry about that!

Will you give it a try with the ControlPath option and see if it happens again?

Still hangs.

Docker […] is a bit silly.

Indeed. I don’t even have it installed so the remote builder is great.

We’ll have a fix today.

@gavinwahl The latest flyctl has a fix that doesn’t use the system’s ssh anymore. When you have a moment upgrade and try again. Sorry about that!

INFO Local docker unavailable, hooking you up with a remote Docker builder…
INFO Waiting for remote builder (fly-builder-autumn-glade-1777) to become available…
Error Could not ping remote builder within 5 minutes, aborting.

Could you update flyctl to the latest version and try again?

Working now

Got the same timeout error again

Looks like all your builder VMs were receiving connections and processing builds, so I think this is either an actual timeout where the VM couldn’t start within 5 minutes or an error that’s being handled wrong. We’ll have an update soon that gives granular status while builder VMs launch and I’ll do some more testing on failures to make sure there’s not a bug in there.

@gavinwahl We’ve made a bunch of improvements to flyctl’s build and deploy process this week. Let me know how things look after updating to v0.0.197+.