Are there any issues with the remote builder at the moment? Trying a deploy and getting an error:
INFO Remote only, hooking you up with a remote Docker builder...
INFO Waiting for remote builder (fly-builder-lively-frog-1262) to become available...
INFO Remote builder is ready to build!
Error failed to dial gRPC: cannot connect to the Docker daemon. Is 'docker daemon' running on this host?: Error creating ssh client: ssh: handshake failed: EOF
Yep, assuming this is it? I added a check in the CI script to show the version after the last issue deploying where I wasn’t sure:
Check it is the latest version
Executing command: /root/.fly/bin/flyctl version
flyctl v0.0.186 linux/amd64 Commit: 5cd602b BuildDate: 2021-03-08T20:28:34Z
Hmm failed again. For a different reason this time though:
INFO Remote only, hooking you up with a remote Docker builder…
INFO Waiting for remote builder (fly-builder-lively-frog-1262) to become available…
INFO Remote builder is ready to build!
Error error during connect: Post “http://%2Fvar%2Frun%2Fdocker.sock/v1.40/build?buildargs=null&buildid=upload-request%3A681da772fb5fa53223675a1e03833b9724302649d
d36c5c9eee6e751ff2d6b03&cachefrom=null&cgroupparent=&cpuperiod=0&cpuquota=0&cpusetcpus=&cpusetmems=&cpushares=0&dockerfile=&labels=null&memory=0&memswap=0&networ
kmode=&rm=0&shmsize=0&target=&ulimits=null&version=2”: Error creating ssh client: ssh: handshake failed: EOF
We changed how flyctl connects to remote builders this past week to get buildpacks working, among other fixes. It’s using ssh now through a proxy in the builder instead of an http proxy. It seems to be working nicely except for intermittent issues like this a few people have reported. I’d like to get those fixed asap though.
Not in my env. I don’t know what the CI system adds though behind the scenes.
Can you restart the remote builder app it gets assigned? That frog one? Or kill it and so make a new one next time it tries to deploy? Since I’m using the latest flyctl, could that old remote builder be the issue? Not sure how that bit works at your end.
These builders are just fly apps created in your org, you can delete it with flyctl apps destroy fly-builder-lively-frog-1262 and it’ll create a new one on the next build.
I think there’s a bug that’s causing flyctl to create a remote builder because --remote-only is on but infer settings for the local daemon because Codefresh has docker installed locally. Out of curiosity, why do you have the --remote-only flag in CI if it has docker?
I think, unless it has changed, that their docker daemon is not available in the way fly needs it to be.
In that … Codefresh can build docker images and deploy them, but only to a registry. And so under that set up, the flyctl would need to support being passed an already-made image (in a remote registry). So, not a fly-managed registry. And I don’t believe that currently is allowed? At least it wasn’t supported a while back.
The docker daemon within Codefresh is apparently not intended to be used via a system-style call, ie directly e.g I noticed they say “By default, and for security reasons, the Docker Daemon is not exposed to freestyle steps running on our SaaS environments. In other words, this approach cannot be used in our SaaS Runtime Environments.”
So maybe it is causing flyctl confusion too.
But the good news is that I’ve tried another deploy, which will have made a new remote builder to replace that one I just destroyed, and the deploy worked
I’m glad a new builder worked but I’m not sure why I’m setting up a test on Codefresh now, hopefully I can get to the bottom of this.
You can use the system docker to build if you’d like. Not sure this works on Codefresh, but it does on BuildKite, CircleCI, and GitHub. Here’s an example :
# authenticate docker to fly
flyctl docker auth
# build the app with docker & tag for your repo
docker build -t registry.fly.io/your-app:build-id ./path/to/app
# push the build to fly
docker push registry.fly.io/your-app:build-id
# deploy the image
flyctl deploy --image registry.fly.io/your-app:build-id
That image flag can be a local image or an image in a registry that fly can pull from to deploy. We need to document that better.
Interesting, yes, I wondered how fly integrates/authenticates with a private registry.
So:
I spoke too soon about it being fixed. That deploy did work. But the next one I tried did not. So maybe if I keep deleting the remote builder app, then deploy, it will work. At least for now. As I got the same error:
INFO Remote only, hooking you up with a remote Docker builder…
INFO Waiting for remote builder (fly-builder-black-dust-4017) to become available…
INFO Remote builder is ready to build!
Error error during connect: Post “http://%2Fvar%2Frun%2Fdocker.sock/v1.40/build?buildargs=%7B%22STAGE%22%3A%22dev%22%7D&buildid=6d047673fd1e73f1e576b4bbde705c0db
00451d582fdae20c09e4a224fe5602d&cachefrom=null&cgroupparent=&cpuperiod=0&cpuquota=0&cpusetcpus=&cpusetmems=&cpushares=0&dockerfile=&labels=null&memory=0&memswap=
0&networkmode=&remote=upload-request&rm=0&session=w6mvmoyta8e5yvot2zwm614cj&shmsize=0&t=registry.fly.io…&ulimits
=null&version=2”: Error creating ssh client: ssh: handshake failed: EOF
Yeah, your commands make sense but I’m not quite sure how that approach works with Codefresh. Unless it is simple as translating them into Codefresh-speak. Since … as I read their docs, you can’t do a system call like docker xyz. Instead you have to turn it into their yml, which they then run on their docker daemon. You aren’t supposed to call docker directly. At least from their warning. So maybe that would work. It would be good to get the remote builder working, but like you say, if Codefresh provide docker and you can use it, it would avoid the need for the builder.
As a bonus, I did try removing that –remote-only flag to see what would happen. And yes, as suspected, Codefresh does not provide direct access to its docker daemon. As you get this message:
INFO Local docker unavailable, hooking you up with a remote Docker builder…
@greg just letting you know we shipped a bunch of improvements to flyctl’s build+deploy code and remote builders this week. Try it out and let us know if you’re still having issues.
Or kill it and so make a new one next time it tries to deploy? Since I’m using the latest flyctl, could that old remote builder be the issue? Not sure how that bit works at your end.
So maybe if I keep deleting the remote builder app, then deploy, it will work. At least for now. As I got the same error: