Remote builders stuck in pending

Questions / Help Build debugging
1

Hey all!

I recently started using fly.io and have been enjoying the service immensely so far. However, I the remote builders seem to very hit or miss. Most recently, all my builder VMs appear to be stuck in “pending” without any way to diagnose what the issue is.

DOCKER_BUILDKIT=1 flyctl deploy --remote-only -v
Deploying <REDACTED>
==> Validating app configuration
--> Validating app configuration done
Services
TCP 80 ⇢ 8080
Remote builder <REDACTED> ready
Error error connecting to docker: Error waiting for remote builder app: context canceled
make: *** [deploy] Error 1

flyctl version
flyctl v0.0.211 darwin/amd64 Commit: babb333 BuildDate: 2021-04-26T19:53:31Z

Nothing too fancy in my fly.toml:

app = "<REDACTED>"

kill_signal = "SIGINT"
kill_timeout = 5

[[services]]
  internal_port = 8080
  protocol = "tcp"

  [[services.ports]]
    handlers = ["http"]
    port = "80"

[[services.http_checks]]
  interval = 10000
  grace_period = "5s"
  method = "get"
  path = "/"
  protocol = "http"
  timeout = 2000
  tls_skip_verify = false
  [services.http_checks.headers]

nor my Dockerfile

FROM golang:1.16

WORKDIR /app

COPY go.* ./

RUN go mod download

COPY . .

RUN --mount=type=cache,target=/root/.cache/go-build \
  CGO_ENABLED=0 go build -o /app .

FROM alpine:latest  
RUN apk --no-cache add ca-certificates
WORKDIR /root/
COPY --from=0 /app .
CMD ["./app"]

Any ideas? I’ve had a couple successful builds and then suddenly timeouts.

2

Hey there, glad you’re enjoying our service!

We have some troubles with capacity in Virginia right now. Currently adding more servers there.

Would you mind sending me your builder name by private message here? I’ll confirm this is because of our IAD region.

3

@chriskseto could you try again when you have a minute? We’ve added more capacity and adjusted builder priority, so hopefully this won’t happen again.