https-failover is now set to true by default in flyctl!

Fresh Produce
1

Hey everyone, you might have seen my last post announcing the --https-failover flag. The TLDR is that you can automatically retry to build an image if your remote builder fails. We’ve now set that flag to true as of v0.2.55. Since then, we’ve noticed a solid reduction in issues connecting to remote builders. This won’t fix all issues, of course, but tackling annoying issues stopping people from deploying their apps is always great.

3 Likes
2

I have no idea what is going on in this error, other than it seems related to Docker, but since this change was two days ago, it seems like perhaps relevant.

==> Building image
==> Building image with Docker
--> docker host: 25.0.2 linux x86_64
==> Building image
==> Building image with Docker
--> docker host: 25.0.2 linux x86_64
Error: failed to fetch an image or build from source: error building: failed to get status: rpc error: code = Unavailable desc = rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: unable to upgrade to h2c, received 403"

any ideas?

Also, is it strange that the first three lines are exactly duplicated in the next three lines? Is that evidence of this --https-failover happening?

3

Interestingly, when I deploy locally it works fine, but the output is significantly different, and it is using a different remote build machine, which is odd, because there is only one build machine on my account. Why would it be 25.0.2 from CI but 24.0.7 from local? What do those numbers mean?

Also, 403 error usually indicates a permissions or authorization issue… but I’m using the same token for CI as for local.

Example from local:

==> Building image
Waiting for remote builder fly-builder-falling-glade-6980...
 🌏WARN The running flyctl agent (v0.2.45) is older than the current flyctl (v0.2.55).
Remote builder fly-builder-falling-glade-6980 ready
Remote builder fly-builder-falling-glade-6980 ready
==> Building image with Docker
--> docker host: 24.0.7 linux x86_64
4

Also, just noticing how strange this is…

 🌏WARN The running flyctl agent (v0.2.45) is older than the current flyctl (v0.2.55).

This is your builder tool. Do you not have them instrumented to auto-update?

5

Looking at the logs of the build machine (fly-builder-falling-glade-6980) on my account there are a lot of warnings and some errors. The most concerning was:

2024-05-19T14:43:09.570 app[732871574a9d85] sin [info] time="2024-05-19T14:43:09.570359638Z" level=error msg="Could not add route to IPv6 network 2001:db8:1::1/64 via device docker0: network is down"
6

It looks like in your first response you were building locally, and in this one you were building remotely (explaining the Docker host version mismatch). I’ll look into your builder a bit later, but in the meantime, can you please try deleting your org’s builder app and trying to deploy again?

1 Like
7

As for this specific log message, it’s definitely interesting but I have the same message in my working builder, so it’s likely not the issue in this case.

2 Likes
8

I briefly flirted with --remote-only but I can’t understand the benefit since the App Builder has none of my secrets, and I don’t want to set that up, so I am going to continue with local deploys for now. No idea how to get CI working, other than perhaps configuring the App Builder with all my secrets. As for the version mismatches, I suppose one version could have been from a local docker, either for me locally, or in CI. :confused: