Error connecting to docker: remote builder app unavailable

Hi, yesterday I deployed just fine with fly deploy.
Today there is this error after some minutes trying to contact builder.

WARN Remote builder did not start in time. Check remote builder logs with `flyctl logs -a fly-builder-still-morning-9002`
Error failed to fetch an image or build from source: error connecting to docker: remote builder app unavailable

I tried deleting the builder and creating a new one in the dashboard, but the result is the same.

I also tried removing and recreating a wireguard peer connections, enabling wireguard websockets and other suggestions posted in other relevant topics.

What steps should I take now?

I have the same issue. Is this an issue from fly.io?

Same here. Every deploy is timing out for me today.

==> Verifying app config
--> Verified app config
==> Building image
WARN Remote builder did not start in time. Check remote builder logs with `flyctl logs -a fly-builder-sparkling-violet-5498`
Error failed to fetch an image or build from source: remote builder app unavailable

I’ve tried deleting the builder, so it automatically creates a new one on next deploy attempt, but it still fails to deploy with the new builder.

I also tried scaling the builder up (are we allowed to do that??) but no luck.

My builder and the app I’m trying to deploy are both in the lhr region (London, UK).

I wanted to add that I can run fly ssh console and other fly commands successfully on the running app.
(Even start the observer (!) and observe the app remotely b/c it’s a Elixir/Phoenix app)

Only the command fly deploy fails for me.

@gtaranti can you try fly apps destroy fly-builder-still-morning-9002 and retry?

@callum same, but with the app fly-builder-sparkling-violet-5498.

@hmontazeri I don’t know what your builder app name is, but if you can try and destroy it and then retry a deploy, that might help.

@Jerome Thanks, but the same with new builder : fly-builder-fragrant-smoke-1646

Same, I have tried destroying the builder. Just tried again now too. Deploy always fails.

When I monitor the builder’s logs, it always gets to Configuring firecracker and then nothing.

Also, not sure if this is related, but: my postgresdb app (also in lhr region) has just died after I attempted to scale it, and it last log line is Configuring firecracker, same as on the builder’s logs. The monitoring page says: ‘Failed deployment v7 … Failed due to unhealthy allocations. … 1 desired, 1 placed, 0 healthy, 1 unhealthy"’

Thanks @callum for letting us know the region!
@hmontazeri and @gtaranti what region(s) are y’all deploying in?

I’m trying to deploy an app in region fra
For the builder I don’t know. Does it matter which region is the builder?

The logs show the builder not booting correctly.

I’ve create an incident as we’re working on resolving this: Fly.io Status - Remote builders are not working properly in London

Can all of you try again now? I think it’ll work better.

FRA

==> Building image with Docker
--> docker host: 20.10.12 linux x86_64
[+] Building 597.2s (0/1)                                                       
 => [internal] load remote build context                                 597.2s
Error failed to fetch an image or build from source: error building: error during connect: Post "http://[fdaa:0:6775:a7b:23c4:1:183b:2]:2375/v1.41/build?buildargs=null&buildid=upload-request%3Abcd8662ef86ccf001c4ccbd4ea35255276a06ee47c87a41e4ee85f9782557f56&cachefrom=null&cgroupparent=&cpuperiod=0&cpuquota=0&cpusetcpus=&cpusetmems=&cpushares=0&dockerfile=&labels=null&memory=0&memswap=0&networkmode=&rm=0&shmsize=0&target=&ulimits=null&version=2": write unix ->/Users/montazha/.fly/fly-agent.sock: write: broken pipe

Only after destroying the builder and trying again fly deploy , I was able to deploy successfully.

Issue resolved for me.

Thanks, I am able to deploy again now. And my postgres db app is back online.