fly deploy is hanging during building. can i switch remote builder?

==> Verifying app config
Validating fly.toml
✓ Configuration is valid
--> Verified app config
==> Building image
Remote builder fly-builder-morning-frost-5200 ready
Remote builder fly-builder-morning-frost-5200 ready
==> Building image with Docker
--> docker host: 24.0.7 linux x86_64
[+] Building 85.3s (6/15)
 => CACHED docker-image://docker.io/docker/dockerfile:1@sha256:a57df69d0  0.0s
 => [internal] load metadata for docker.io/library/node:20.6.1-slim       0.7s
 => [internal] load build context                                         0.5s
 => => transferring context: 83.77kB                                      0.5s
 => [base 1/2] FROM docker.io/library/node:20.6.1-slim@sha256:2dab2d0e8  82.7s
 => => resolve docker.io/library/node:20.6.1-slim@sha256:2dab2d0e8813ee1  0.0s
 => => sha256:9fe74cf2f95f2c249e48b8b604cac9abdbe11c7b31 7.06kB / 7.06kB  0.0s
 => => sha256:360eba32fa65016e0d558c6af176db31a202e9a6 29.12MB / 29.12MB  0.2s
 => => sha256:24f632c8bcc8f838302a408474f63c842f77e774d0 3.36kB / 3.36kB  0.0s
 => => sha256:6beeb28cf70c25b3fa252633de23de8a30a1e5d 15.73MB / 48.10MB  82.7s
 => => sha256:6b93b28e5314813bed2421b0c2a470cae8909217af 2.74MB / 2.74MB  0.1s
 => => sha256:6356377ba599c35b902251cc0dd1b24f62737778029b23 450B / 450B  0.0s
 => => sha256:2dab2d0e8813ee1601f8d25a8e4aa5530ffc4d0cc1 1.21kB / 1.21kB  0.0s
 => => sha256:c2c8591f8face6346d00cd6d3353913311af17755c 1.37kB / 1.37kB  0.0s
 => => extracting sha256:360eba32fa65016e0d558c6af176db31a202e9a6071666f  1.0s
 => => extracting sha256:24f632c8bcc8f838302a408474f63c842f77e774d053285  0.0s

I’ve been trying to deploy for the past 30 min and it always hang during this stage. I have waited for over 700 seconds in one of the attempts so I think there’s definitely an issue.

Is there a way for me to switch the assigned remote builder to see if that fixes the issue?

Try:

fly apps destroy fly-builder-morning-frost-5200

A new builder will be created for you the next time you deploy.

1 Like

thanks for the quick response. I successfully assigned a new builder but unfortunately did not resolve the issue.

would you have any other suggestions for me?

Can I see the Dockerfile?

Hi, build finally finished in 4136.0s. Before this incident it used to be consistently less than 2 min if I remember correctly tho.

I’m going to try a few more times throughout the day.

for your reference, here is my dockerfile

# syntax = docker/dockerfile:1

# Adjust NODE_VERSION as desired
ARG NODE_VERSION=20.6.1
FROM node:${NODE_VERSION}-slim as base

LABEL fly_launch_runtime="Remix"

# Remix app lives here
WORKDIR /app

# Set production environment
ENV NODE_ENV="production"


# Throw-away build stage to reduce size of final image
FROM base as build

# Install packages needed to build node modules
RUN apt-get update -qq && \
    apt-get install -y build-essential pkg-config python-is-python3

# Install node modules
COPY --link package-lock.json package.json ./
RUN npm ci --include=dev

# Copy application code
COPY --link . .

# Build application
RUN npm run build

# Remove development dependencies
RUN npm prune --omit=dev


# Final stage for app image
FROM base

# Copy built application
COPY --from=build /app /app

# Start the server by default, this can be overwritten at runtime
EXPOSE 3000
CMD [ "npm", "run", "start" ]

and the successful build terminal output

==> Building image
✓ compatible remote builder found
Waiting for remote builder fly-builder-holy-violet-7843...
 🌍INFO Override builder host with: https://fly-builder-holy-violet-7843.fly.dev (was tcp://[fdaa:4:1c82:a7b:2cd:f6f2:c1c8:2]:2375)
Remote builder fly-builder-holy-violet-7843 ready
Waiting for remote builder fly-builder-holy-violet-7843...
 🌍INFO Override builder host with: https://fly-builder-holy-violet-7843.fly.dev (was tcp://[fdaa:4:1c82:a7b:2cd:f6f2:c1c8:2]:2375)
Remote builder fly-builder-holy-violet-7843 ready
==> Building image with Docker
--> docker host: 24.0.7 linux x86_64
[+] Building 4136.0s (16/16) FINISHED
 => [internal] load .dockerignore                                         0.2s
 => => transferring context: 88B                                          0.1s
 => [internal] load build definition from Dockerfile                      0.2s
 => => transferring dockerfile: 953B                                      0.1s
 => resolve image config for docker.io/docker/dockerfile:1                1.6s
 => docker-image://docker.io/docker/dockerfile:1@sha256:a57df69d0ea827fb  0.2s
 => => resolve docker.io/docker/dockerfile:1@sha256:a57df69d0ea827fb7266  0.0s
 => => extracting sha256:96918c57e42509b97f10c074d80672ecdbd3bb7dcd38c1b  0.1s
 => => sha256:b5f3b260a9678e1d83d2fce86eeddf79420b79147eaba2 482B / 482B  0.0s
 => => sha256:68ebc061390d9a7d6e194f9d58309c754a53cb8b4e 1.26kB / 1.26kB  0.0s
 => => sha256:96918c57e42509b97f10c074d80672ecdbd3bb7d 11.98MB / 11.98MB  0.1s
 => => sha256:a57df69d0ea827fb7266491f2813635de6f17269be 8.40kB / 8.40kB  0.0s
 => [internal] load metadata for docker.io/library/node:20.6.1-slim       0.6s
 => [base 1/2] FROM docker.io/library/node:20.6.1-slim@sha256:2dab2d0  4065.0s
 => => resolve docker.io/library/node:20.6.1-slim@sha256:2dab2d0e8813ee1  0.0s
 => => sha256:6356377ba599c35b902251cc0dd1b24f62737778029b23 450B / 450B  0.0s
 => => sha256:2dab2d0e8813ee1601f8d25a8e4aa5530ffc4d0cc1 1.21kB / 1.21kB  0.0s
 => => sha256:c2c8591f8face6346d00cd6d3353913311af17755c 1.37kB / 1.37kB  0.0s
 => => sha256:9fe74cf2f95f2c249e48b8b604cac9abdbe11c7b31 7.06kB / 7.06kB  0.0s
 => => sha256:360eba32fa65016e0d558c6af176db31a202e9a6 29.12MB / 29.12MB  0.2s
 => => sha256:24f632c8bcc8f838302a408474f63c842f77e774d0 3.36kB / 3.36kB  0.0s
 => => sha256:6beeb28cf70c25b3fa252633de23de8a30a1e 48.10MB / 48.10MB  4063.0s
 => => sha256:6b93b28e5314813bed2421b0c2a470cae8909217af 2.74MB / 2.74MB  0.0s
 => => extracting sha256:360eba32fa65016e0d558c6af176db31a202e9a6071666f  1.1s
 => => extracting sha256:24f632c8bcc8f838302a408474f63c842f77e774d053285  0.0s
 => => extracting sha256:6beeb28cf70c25b3fa252633de23de8a30a1e5d00d3ee1d  1.9s
 => => extracting sha256:6b93b28e5314813bed2421b0c2a470cae8909217af784ff  0.1s

Check your .dockerignore file. If it is missing, or doesn’t include node-modules, you will be uploading a large number of files. Here’s an example you can use as a starting point:

here is my dockerignore

node_modules

/.cache
/build
/public/build
.env

This is not normal, nor what I or others are seeing. My current theory remains that you have some file or set of files that are large that you are uploading. One way to test this is to create a file by the name of Dockerfile.test, with the following contents:

FROM ubuntu
WORKDIR /app
COPY . .

Then run the following command:

fly console --dockerfile Dockerfile.test -C bash

If my theory is correct, that will take a while. Once you get a command prompt from the ephemeral machine that this creates, you can find where the problem is with the following command:

du -s * .* | sort -n
1 Like

It was the unused images that were getting pushed to the remote builder.

Of course, since fly.io uses a remote builder, it should grab everything from my project, as it doesn’t have any context of which assets would eventually get imported by my code.

I’m not entirely sure if this was the sole issue when this happened, since it only cut down the total size by <30% (<300mb). I guess this process doesn’t scale linearly with the total size?
But I do not have issues with the remote builder anymore.

Thank you!