Global registry: now in production 🚀

Fresh Produce
1

registry.fly.io now uses global storage to improve reliability of image pushes! Pushes in Sydney are stored in Sydney. Pushes near Mumbai use storage in Mumbai. And so on. This improves reliability when pushing docker images to our registry.

Every Fly deployment interacts with registry.fly.io. That’s where we store the docker image that gets transmogrified into a running firecracker VM. fly deploy will push an image to registry.fly.io and then flyd or nomad will pull that image when creating or updating an instance.

Previously, the registry stored images in the Ashburn, Virginia USA region. fly deploy would struggle to push the image to the registry when run far away from Virginia USA, like say from Sydney. That’s because pushing gigabytes, or even lots of megabytes, of docker image content from Sydney to Virginia is… difficult. Packets get lost, timeouts occur, retries are needed all of which could lead to failed deployments.

So, we brought the registry storage closer to you. After a successful preview of this new architecture, we now have storage in all regions, and have released this work to production.

Having the docker images close to where they are pushed doesn’t change the fact that pulls may cross the globe. For example, an image pushed near Seattle will be stored there. An app running that image in Seattle, Tokyo, and Ezeiza will still have to pull it from Seattle to those three locations. We’ve seen pushes be more sensitive to network conditions, which is why we started in this way. We’re working on improving the reliability of pulls, so stay tuned!

Keep on deploying like you always have, and let us know if you run into any issues with the registry or have other feedback.

23 Likes
2

I wonder if that’s the reason for this error https://community.fly.io/t/deploy-failed-once-more/13681/17

My CI suddenly gives the same error. Meanwhile the same command works from my laptop. Although after deleting the builder app (as suggested in that thread) and trying again local, using remote builder, it took some seconds to pick up the registry and start working. But my CI still fails

3

Yeah. I think that’s why i get the deploy error.

–local-only work but remote build not work. It’s so sad. I just first time use fly.io I has been use the heroku. But it’s should not have same issue.

fly deploy: failed

image
image2490×336 111 KB

fly deploy --local-only: work fine.

image
image2512×854 476 KB

In github action: failed

image
image3128×802 377 KB

Not anyone testing your change?

4

I think this might have been an api issue, since it has a request id. Are you still seeing this failure?

If so, please try running with LOG_LEVEL=debug and let us know what shows up. Something like this should work for github actions:

...
      - run: flyctl deploy --remote-only
        env:
          FLY_API_TOKEN: ${{ secrets.FLY_API_TOKEN }}
          LOG_LEVEL: debug
5

Pushing a fairly large image (6.02GB), with one layer that makes up most of its size (5.59GB) now seems to be failing after this change.

Docker push output:

The push refers to repository [registry.fly.io/musicbrainz-production]
1222cb43c6f2: Mounted from musicbrainz-staging
6dba0c30060e: Mounted from musicbrainz-staging
98fb2c3c0209: Pushed
c2fe3df43305: Pushed
a4c3c0273921: Pushed
9334dd897c59: Pushing [==================================================>]  5.598GB
4b3e77b7cdac: Pushed
5f70bf18a086: Layer already exists
ca6083bb7ee3: Pushed
dcbc06995385: Pushed
308d33c9936f: Pushed
c7699a1fbefa: Pushed
730fe4386c4a: Pushed
9edb40afd798: Pushed
d67c4070ec0d: Pushed
ca56acf03095: Pushed
fe4fa537f778: Pushed
921828208720: Pushed
78a822fe2a2d: Pushed
received unexpected HTTP status: 502 Bad Gateway

Docker daemon log:

time="2023-07-11T13:08:00.466921580Z" level=error msg="Upload failed, retrying: received unexpected HTTP status: 502 Bad Gateway"
time="2023-07-11T13:10:43.106738700Z" level=error msg="Not continuing with push after error: context canceled"
7

Does docker-hub-mirror.fly.io behave the same way? I imagine it caches the pulled images on the closest region of the puller. Is it deployed on all the fly regions?

I ask because I want to use the fastest possible way to pull images from dockerhub for a project called dockerphobia, where you can inspect any Docker image files sizes with a treemap chart.