Deploys failing


Are there any issues with the remote builder at the moment? Trying a deploy and getting an error:

INFO Remote only, hooking you up with a remote Docker builder...                                                                                                 
INFO Waiting for remote builder (fly-builder-lively-frog-1262) to become available...                                                                            
INFO Remote builder is ready to build!                                                                                                                           
Error failed to dial gRPC: cannot connect to the Docker daemon. Is 'docker daemon' running on this host?: Error creating ssh client: ssh: handshake failed: EOF

Are you using the latest version of flyctl?

Yep, assuming this is it? I added a check in the CI script to show the version after the last issue deploying where I wasn’t sure:

Check it is the latest version                                                                                                                                                                                                                                                                 
Executing command: /root/.fly/bin/flyctl version                                                                                                                 
flyctl v0.0.186 linux/amd64 Commit: 5cd602b BuildDate: 2021-03-08T20:28:34Z

Yeah that’s the latest, but I can’t reproduce. Does it continue to error if you try again?

I’ll try it again.

I wondered if something had changed, but sounds like not. Maybe it’s a random error with the builder if you aren’t getting it too.

Hmm failed again. For a different reason this time though:

INFO Remote only, hooking you up with a remote Docker builder…
INFO Waiting for remote builder (fly-builder-lively-frog-1262) to become available…
INFO Remote builder is ready to build!
Error error during connect: Post “http://%2Fvar%2Frun%2Fdocker.sock/v1.40/build?buildargs=null&buildid=upload-request%3A681da772fb5fa53223675a1e03833b9724302649d
kmode=&rm=0&shmsize=0&target=&ulimits=null&version=2”: Error creating ssh client: ssh: handshake failed: EOF

We changed how flyctl connects to remote builders this past week to get buildpacks working, among other fixes. It’s using ssh now through a proxy in the builder instead of an http proxy. It seems to be working nicely except for intermittent issues like this a few people have reported. I’d like to get those fixed asap though.

Do you have any DOCKER_* variables in your env?

Not in my env. I don’t know what the CI system adds though behind the scenes.

Can you restart the remote builder app it gets assigned? That frog one? :slight_smile: Or kill it and so make a new one next time it tries to deploy? Since I’m using the latest flyctl, could that old remote builder be the issue? Not sure how that bit works at your end.

Oh interesting, what CI system is this?

These builders are just fly apps created in your org, you can delete it with flyctl apps destroy fly-builder-lively-frog-1262 and it’ll create a new one on the next build.

It’s Codefresh. Trying it out.

Ah … interesting. I didn’t know I controlled that. Ok, great, I’ll try deleting it and see what happens on the next deploy.

I think there’s a bug that’s causing flyctl to create a remote builder because --remote-only is on but infer settings for the local daemon because Codefresh has docker installed locally. Out of curiosity, why do you have the --remote-only flag in CI if it has docker?

Ah. Well this caused me some confusion too!

I think, unless it has changed, that their docker daemon is not available in the way fly needs it to be.

In that … Codefresh can build docker images and deploy them, but only to a registry. And so under that set up, the flyctl would need to support being passed an already-made image (in a remote registry). So, not a fly-managed registry. And I don’t believe that currently is allowed? At least it wasn’t supported a while back.

The docker daemon within Codefresh is apparently not intended to be used via a system-style call, ie directly e.g I noticed they say “By default, and for security reasons, the Docker Daemon is not exposed to freestyle steps running on our SaaS environments. In other words, this approach cannot be used in our SaaS Runtime Environments.”

So maybe it is causing flyctl confusion too.

But the good news is that I’ve tried another deploy, which will have made a new remote builder to replace that one I just destroyed, and the deploy worked :slight_smile:

1 Like

I’m glad a new builder worked but I’m not sure why :slight_smile: I’m setting up a test on Codefresh now, hopefully I can get to the bottom of this.

You can use the system docker to build if you’d like. Not sure this works on Codefresh, but it does on BuildKite, CircleCI, and GitHub. Here’s an example :

# authenticate docker to fly
flyctl docker auth

# build the app with docker & tag for your repo
docker build -t ./path/to/app

# push the build to fly
docker push

# deploy the image
flyctl deploy --image

That image flag can be a local image or an image in a registry that fly can pull from to deploy. We need to document that better.

Interesting, yes, I wondered how fly integrates/authenticates with a private registry.


  1. I spoke too soon about it being fixed. That deploy did work. But the next one I tried did not. :frowning: So maybe if I keep deleting the remote builder app, then deploy, it will work. At least for now. As I got the same error:

INFO Remote only, hooking you up with a remote Docker builder…
INFO Waiting for remote builder (fly-builder-black-dust-4017) to become available…
INFO Remote builder is ready to build!
Error error during connect: Post “http://%2Fvar%2Frun%2Fdocker.sock/v1.40/build?buildargs=%7B%22STAGE%22%3A%22dev%22%7D&buildid=6d047673fd1e73f1e576b4bbde705c0db
=null&version=2”: Error creating ssh client: ssh: handshake failed: EOF

  1. Yeah, your commands make sense but I’m not quite sure how that approach works with Codefresh. Unless it is simple as translating them into Codefresh-speak. Since … as I read their docs, you can’t do a system call like docker xyz. Instead you have to turn it into their yml, which they then run on their docker daemon. You aren’t supposed to call docker directly. At least from their warning. So maybe that would work. It would be good to get the remote builder working, but like you say, if Codefresh provide docker and you can use it, it would avoid the need for the builder.

As a bonus, I did try removing that –remote-only flag to see what would happen. And yes, as suspected, Codefresh does not provide direct access to its docker daemon. As you get this message:

INFO Local docker unavailable, hooking you up with a remote Docker builder…

Good to know, thanks! We’re digging through a few possible causes, I’ll keep you posted.

@greg just letting you know we shipped a bunch of improvements to flyctl’s build+deploy code and remote builders this week. Try it out and let us know if you’re still having issues.

@michael Thanks.

I tried some deploys earlier today. The first failed:

Remote builder fly-builder-cold-glitter-7018 ready                                                                                                               
Error error connecting to docker: Error waiting for remote builder app: context canceled 

… so I deleted that builder app, as I’ve done before. And since then all deploys have worked :slight_smile:

Thanks for following up! Looks like some builders weren’t getting the latest changes. I just purged them so you should be good next time.