Deploy hangs for long time and fail

Hi, I have a pretty small rust binary that compiles on my machine under a minute. Local docker build is similarly fast. When deploying on fly, it takes ages and transfer huge amount of data. Any idea what can I try?

✓ Configuration is valid
→ Verified app config
==> Building image
==> Building image with Depot
→ build: ()
[+] Building 1787.9s (9/16)
=> [internal] load build definition from Dockerfile 0.3s
=> => transferring dockerfile: 1.52kB 0.3s
=> [internal] load metadata for Docker Hub Container Image Library | App Containerization 0.3s
=> [internal] load metadata for Docker Hub Container Image Library | App Containerization 0.3s
=> [internal] load .dockerignore 0.2s
=> => transferring context: 2B 0.2s
=> [builder 1/6] FROM Docker Hub Container Image Library | App Containerization 0.0s
=> => resolve Docker Hub Container Image Library | App Containerization 0.0s
=> [internal] load build context 1787.0s
=> => transferring context: 4.82GB 1787.0s
=> [stage-1 1/5] FROM Docker Hub Container Image Library | App Containerization 0.0s
=> => resolve Docker Hub Container Image Library | App Containerization 0.0s
=> CACHED [stage-1 2/5] WORKDIR /app 0.0s
=> CACHED [builder 2/6] WORKDIR /usr/src/app 0.0s
=> [stage-1 3/5] RUN apt-get update && apt-get install -y libssl3 libsnappy1v5 3.7s

This time it passed, but after a very long time. I am afraid that this will limit my ability to update working production, so any help will be beneficial. This is the whole output:

✓ Configuration is valid
--> Verified app config
==> Building image
==> Building image with Depot
--> build:  ()
[+] Building 2388.8s (17/17) FINISHED
 => [internal] load build definition from Dockerfile                                          0.3s
 => => transferring dockerfile: 1.52kB                                                        0.3s
 => [internal] load metadata for docker.io/library/debian:bookworm-slim                       0.3s
 => [internal] load metadata for docker.io/library/rust:1.83-slim                             0.3s
 => [internal] load .dockerignore                                                             0.2s
 => => transferring context: 2B                                                               0.2s
 => [builder 1/6] FROM docker.io/library/rust:1.83-slim@sha256:540c902e99c384163b688bbd8b5b8  0.0s
 => => resolve docker.io/library/rust:1.83-slim@sha256:540c902e99c384163b688bbd8b5b8520e94e7  0.0s
 => [internal] load build context                                                          2140.6s
 => => transferring context: 5.81GB                                                        2140.6s
 => [stage-1 1/5] FROM docker.io/library/debian:bookworm-slim@sha256:f70dc8d6a8b6a06824c9247  0.0s
 => => resolve docker.io/library/debian:bookworm-slim@sha256:f70dc8d6a8b6a06824c92471a1a2580  0.0s
 => CACHED [stage-1 2/5] WORKDIR /app                                                         0.0s
 => CACHED [builder 2/6] WORKDIR /usr/src/app                                                 0.0s
 => [stage-1 3/5] RUN apt-get update && apt-get install -y     libssl3     libsnappy1v5       3.7s
 => [builder 3/6] COPY . .                                                                    5.4s
 => [builder 4/6] RUN apt-get update && apt-get install -y     pkg-config     libssl-dev     19.0s
 => [builder 5/6] RUN cargo build --release                                                 214.5s
 => [builder 6/6] RUN strip target/release/nanostitch                                         0.3s
 => [stage-1 4/5] COPY --from=builder /usr/src/app/target/release/nanostitch /app/nanostitch  0.0s
 => [stage-1 5/5] RUN mkdir -p /app/data && chmod 777 /app/data                               0.1s
 => exporting to image                                                                        7.9s
 => => exporting layers                                                                       0.1s
 => => exporting manifest sha256:318441517dba38a79c323ae6e97003ffd77f0b0f98bb308ec9e00e240cb  0.0s
 => => exporting config sha256:2875a9c96f779e897ded0702412b1352694d4e72550237c7fe4367f4e5936  0.0s
 => => pushing layers for registry.fly.io/nanostitch:deployment-01JJKG4WWKN8AC55XS0FHQRTEA@s  6.8s
 => => pushing layer sha256:62519c733b389f0af33308fe007cbb3dc0667d5905143aa2e69217b70110a095  4.2s
 => => pushing layer sha256:e4a9cee759b2c14f64f6890ab762771b4fb95e0d8ef50230b72e4c239e1198db  5.3s
 => => pushing layer sha256:5743c39633be8e26d8cc629d4582ff15818334cb24a260d405c4c38ae8dd0b20  2.3s
 => => pushing layer sha256:fdc7a7e1dffb0e8bb7874830c96647c3e77bc11a1b645eb8c57a444cae1d99b5  2.4s
 => => pushing layer sha256:2875a9c96f779e897ded0702412b1352694d4e72550237c7fe4367f4e5936b0f  4.0s
 => => pushing layer sha256:af302e5c37e9dc1dbe2eadc8f5059d82a914066b541b0d1a6daa91d0cc55057d  6.8s
 => => pushing manifest for registry.fly.io/nanostitch:deployment-01JJKG4WWKN8AC55XS0FHQRTEA  1.0s
--> Build Summary:  ()
--> Building image done
image: registry.fly.io/nanostitch:deployment-01JJKG4WWKN8AC55XS0FHQRTEA
image size: 37 MB

would you mind sharing your Dockerfile and perhaps your fly.toml as well? how big is your locally built docker image?

in the mean time, since you already have the local docker image built, you could try and use that directly instead of having fly’s remote builder rebuild it from scratch.

Tell fly to use your image and then deploy using fly deploy --local-only

1 Like

I changed my fly.toml meanwhile, but here it is:


app = "nanostitch"
primary_region = "ams"

[build]
dockerfile = "Dockerfile"

[http_service]
internal_port = 8080
force_https = true
auto_start_machines = true
min_machines_running = 1
processes = ["primary", "replica"]

[[vm]]
memory = "512mb"
cpu_kind = "shared"
cpus = 1

[processes]
primary = "./nanostitch"
replica = "./nanostitch"

[[services]]
protocol = "tcp"
internal_port = 8080
processes = ["primary", "replica"]

  [[services.ports]]
  port = 80
  handlers = ["http"]
  force_https = true

  [[services.ports]]
  port = 443
  handlers = ["tls", "http"]

  [[services.tcp_checks]]
  interval = "15s"
  timeout = "2s"
  grace_period = "1s"

[[mounts]]
source = "nanostitch_data_ams"
destination = "/app/data"
processes = ["primary"]

[[mounts]]
source = "nanostitch_data_ams_backup"
destination = "/app/data"
processes = ["replica"]

[env]
RUST_LOG = "info"
DATABASE_PATH = "/app/data/events.db"
SEARCH_PATH = "/app/data/search_index"
EVENT_RETENTION_SECS = "2592000"
CLEANUP_INTERVAL_SECS = "3600"
ENABLE_SEARCH = "true"

and the docker:

# Builder stage
FROM rust:1.83-slim AS builder

WORKDIR /usr/src/app
COPY . .

RUN apt-get update && apt-get install -y \
    pkg-config \
    libssl-dev \
    libclang-dev \
    clang \
    librocksdb-dev \
    libsnappy-dev \
    liblz4-dev \
    libzstd-dev \
    libgflags-dev

# Build with release profile
RUN cargo build --release
# Strip debug symbols to reduce binary size
RUN strip target/release/nanostitch

# Runtime stage
FROM debian:bookworm-slim

WORKDIR /app

# Install runtime dependencies for RocksDB and SSL
RUN apt-get update && apt-get install -y \
    libssl3 \
    libsnappy1v5 \
    liblz4-1 \
    libzstd1 \
    && rm -rf /var/lib/apt/lists/*

# Copy the binary from builder
COPY --from=builder /usr/src/app/target/release/nanostitch /app/nanostitch

# Create directory for RocksDB with appropriate permissions
RUN mkdir -p /app/data && chmod 777 /app/data

# Set environment variables
ENV RUST_LOG=info
ENV DATABASE_PATH=/app/data/events.db
ENV SEARCH_PATH=/app/data/search_index
ENV ENABLE_SEARCH=true

# Set some system parameters for RocksDB (optional, might need privileged container)
# ENV ROCKSDB_MAX_OPEN_FILES=1000
# ENV ROCKSDB_WRITE_BUFFER_SIZE=512MB

# These can also be set via ulimit in your fly.toml if needed
# RUN ulimit -n 1024

EXPOSE 8080

# Health check
HEALTHCHECK --interval=30s --timeout=30s --start-period=5s --retries=3 \
    CMD curl -f http://localhost:8080/health || exit 1

CMD ["./nanostitch"]

I’d try to build the docker image locally (say with name "nanosnitch_image"), then edit the fly.toml so that the build section becomes:

[build]
image = 'nanosnitch_image'

and finally deploy using: fly deploy --local-only

so that the locally built image is sent to Fly rather than rebuilt remotely

(if you are on an Apple Silicon machine you will have to specify the platform otherwise the resulting image won’t run once deployed: docker build --platform=linux/amd64 -t nanosnitch_image . )

That improve the build time significantly, thx. I ran into showstopper anyway, Fly does not have shared volumes, the whole concept I was trying to deploy is not going to work.

depending on what you need to store and how big each of those things are… you could consider storing them in SQLite and use LiteFS since :point_down:

SQLite reads and writes small blobs 35% faster than the same blobs can be read from or written to individual files on disk using fread() or fwrite().

(but do note LiteFS status in the doc).

I decided to follow a completely different path. Appreciate the help though. BTW, next build was way faster even when I forgot to build locally first.

1 Like

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.