Every deploy through Github Actions is failing

1

I’m not sure if anyone else ie experiencing this but for the past few days almost every github action deploy has been failing with these logs:

==> Validating app configuration
--> Validating app configuration done
Services
TCP 80/443 ⇢ 5000
Waiting for remote builder fly-builder-red-glitter-####...
Creating WireGuard peer "interactive-e3ad59969f94-myemail-gmail-com-613" in region "iad" for organization ludicrous
Error error connecting to docker: Could not ping remote builder within 5 minutes, aborting.

Here is what I do in the action

      - uses: superfly/flyctl-actions@1.1
        env:
          FLY_API_TOKEN: ${{ env.FLY_API_TOKEN }}
        with:
          args: 'deploy --remote-only'
2

Using the command manually (with the remote flag) works fine

3

Will be looking into this shortly.

Can you run it with the environment variable LOG_LEVEL=debug ? This might give us more information.

I think we might have a repo setup to test this with.

4

@nahtnam I just setup a test repo for our action and it’s connecting to the builder just fine using the ord gateway. I see in the logs it’s using IAD for you. I wonder if there’s an issue there.

Are you able to ssh into a running app for your org like this?

flyctl ssh console -a app-name --region iad
5

@nahtnam we found a bug that was probably causing this. Our API was accepting wireguard peer pubkeys that aren’t valid, so wireguard wasn’t able to reload and pick up new peers. This should work in iad now, but we’re still cleaning up the bug.

6

Awesome, thanks for looking into it. Is there any way I can change where the build happens? I would prefer it to be in SJC, I had no idea it was happening in IAD

7

Builders are created close to wherever flyctl deploy originated and are reused for all builds in an org after that. Since IAD is close to us-east-1 you may have gotten there if GH actions kicked off the deploy that created the builder. We don’t have an easy way to move them right now. Simplest is to delete the existing builder then run flyctl deploy from your machine so it gets setup closer to you.

8

Hello @michael @kurt , they are failing again:

Waiting for remote builder fly-builder-red-glitter-1316...
Creating WireGuard peer "interactive-a55c661a6a12-myemail-gmail-com-810" in region "iad" for organization ludicrous
Error error connecting to docker: Error waiting for remote builder app: context deadline exceeded
9

Hey there.

This is unrelated to GH Actions this time. I’ve created a statuspage incident: Fly.io Status - Remote builders are not detected as "up"