Improve remote builder issues with --http-failover!

Fresh Produce
1

Hey everyone,
We added a flag to flyctl deployments called -http-failover. When set to true, it’ll automatically failover to using http to connect to a remote builder (provided that we aren’t doing that already!)

As an example, you’d just run fly deploy --http-failover=true

Why should I care?
Remote builder failures are one of the most common errors that deploys bump into. Currently, --http-failover is false by default, but once we’re sure the flag doesn’t have any issues, we’ll flip the default to true. This should help anyone who experiences these types of errors often to have smoother deploys.

7 Likes
2

I think it’s a very interesting idea :slight_smile:

1 Like
3

I’m still regularly seeing many different issues, especially when doing hundreds of builds per day. A few examples:

  • Error: failed to fetch an image or build from source: Not authorized to access this firecrackerapp
  • Error: failed to fetch an image or build from source: server returned a non-200 status code: 504
  • Error: failed to fetch an image or build from source: error building: failed to receive status: rpc error: code = Unavailable desc = error reading from server: EOF
  • cannot create /x/y/z: No space left on device

I’ve been personally moving all builds from Fly to simply building on GHA:

jobs:
  build:
    runs-on: ubuntu-22.04
    steps:
      - uses: actions/checkout@v4
      - name: Setup Docker BuildX
        id: buildx
        uses: docker/setup-buildx-action@master
        with:
          install: true
      - name: Log in to Fly.io Docker registry
        uses: docker/login-action@v3
        with:
          registry: registry.fly.io
          username: x
          password: ${{ secrets.FLY_API_TOKEN }}
      - name: Build image
        uses: docker/build-push-action@v3
        with:
          builder: ${{ steps.buildx.outputs.name }}
          push: true
          tags: ${{ env.IMAGE_TAG }}
          cache-from: type=gha
          cache-to: type=gha,mode=max

  deploy:
    needs: build
    runs-on: ubuntu-22.04
    steps:
      - uses: actions/checkout@v4
      - uses: superfly/flyctl-actions/setup-flyctl@master
      - run: flyctl deploy --image=${{ env.IMAGE_TAG }}

This is more reliable and considering you pay for Github actions per minute, it doesn’t have impact on cost. Perhaps Fly should consider letting someone else take care of building, e.g. integrating with some SaaS like https://depot.dev/ :eyes:

2 Likes
4

Hey @Yaeger , can you please link your fly app name? If you can’t publicly, could you email support@fly.io with your app name and mention Billy? I want to look into a few of the errors you mentioned.

5

Done, sent it to our support address. Hope that works for you!

1 Like