Pushing to registry slow

1

Pushing to the registry is painfully slow. A deploy that would take 3 minutes at most sometimes takes an hour when trying to push layers to http://registry.fly.io/

The same deployment sometimes only takes a minute to push to the same registry.

What is going on with the registry performance?

I had this problem previously and was able to mitigate it with --remote-only, but this is failing as of lately as well: Deploy stuck sending build context to Docker daemon - #9 by hb9cwp

Deployments are arguably a very critical thing so it’s upsetting that I can’t trust this part of fly.io. Sometimes I am able to deploy a hotfix within 5 minutes and sometimes it takes an hour because the registry upload is so slow.

2

Something really weird is going on.

So I opened this thread because I was frustrated waiting for an hour for the upload to complete. Opened another terminal and tried another deploy at the same time with --remote-only. And while the second deploy was “Sending build context to Docker daemon” I could see how the first deployment was finally uploading with 100x of the speed I was seeing for the previous hour to the registry. It finished within a minute.

3

Hi @jascha are you still seeing this issue pushing to registry and what region are you pushing to?

We had a previous incident yesterday with registry being unresponsive in the LHR region that has been resolved now, this maybe related to that.

Fly.io Status - Registry is unresponsive in LHR

4

I have seen it a lot of times over the past few weeks and I am pushing to EWR region so I don’t think that’s related. It is still slow (even if occasionally it works).

Here: 13.66 MB in 3400s with --remote-only:

image
image3024×504 75.1 KB

5

Hey there,

Pushing the build context to the remote builder is not the same as pushing to the registry. Are you seeing problems with both, or just with pushing the Docker context?

One reason the Docker context push could be a problem is that your remote builder is hosted in Paris. Which region are you in? If you’re in ewr, pushes could be slow due to that distance.

If you have one region where you deploy from, it’s better to have your builder there. To make that happen, you can destroy your builder app and run a remote deployment from your region. The builder will be created closer to you.

Pushing to the registry should, in general, be fast from any region, but there we’re dependent on AWS S3, our source of truth currently for storing images. Sometimes it can slow down without warning. In the future we hope to run our own storage and be able to make more specific claims about latency.

6

As I said it unfortunately happens with both pushing to the registry and pushing the docker context to the builder.

It just happened that I had a screenshot from the latter and that’s why I posted it.

I am located in Europe, so Paris is fine. But even then pushing from Virginia to Paris should not take 1 hour for 16MB.

It just happened again. It pushed four layers just fine but the last one is taking forever.

7

And again this weird thing happens, if pushing to registry is grinding to a halt for --local-only, and I run another fly deploy --remote-only in a separate terminal window it magically starts pushing the image.

8

Can you share your Dockerfile?

9 

FROM node:16 AS development

RUN mkdir /home/node/app && chown node:node /home/node/app

WORKDIR /home/node/app

COPY package*.json ./

ENV PUPPETEER_SKIP_CHROMIUM_DOWNLOAD true

RUN npm i --quiet

COPY ../ ./

RUN npm run build

FROM node:16-slim AS production

RUN apt-get update \
 && apt-get install -y chromium \
    fonts-ipafont-gothic fonts-wqy-zenhei fonts-thai-tlwg fonts-kacst fonts-freefont-ttf libxss1 \
    --no-install-recommends

ENV PUPPETEER_SKIP_CHROMIUM_DOWNLOAD true
ENV PUPPETEER_EXECUTABLE_PATH /usr/bin/chromium

WORKDIR /home/node/app

RUN chown -R node:node /home/node/app

USER node

COPY --chown=node:node ../packages ./packages

COPY --chown=node:node ../package*.json ./

COPY --chown=node:node ../patches ./patches

RUN npm i --quiet

COPY --from=development --chown=node:node /home/node/app/dist ./dist

CMD [ "npm", "run", "start" ]
10

Just deployed again, now it’s fast with --local-only. No special tricks necessary, no changes to the Dockerfile.

11

What’s the total size of the final image being built? It’s hard to say what could have been happening there.

12

1.4 GB but I have seen it happen as well with an image that’s half the size