Preview: a more reliable global registry

Fresh Produce
1

Every Fly deployment interacts with registry.fly.io. That’s where we store the docker image that gets transmogrified into a running firecracker VM. fly deploy will push an image to registry.fly.io and then flyd or nomad will pull that image when creating or updating an instance.

The registry stores images in the iad (Ashburn, Virginia USA) region. fly deploy can struggle to push the image to the registry when run far away from Virginia USA, like say from Sydney. That’s because pushing gigabytes, or even lots of megabytes, of docker image content from Sydney to Virginia is… difficult. Packets get lost, timeouts occur, retries are needed all resulting in slow or even failed deployments.

We’re changing that today! We’re bringing the registry storage closer to you using our global network. Deploying from Sydney? We’ll store the docker image in Sydney. Deploy from Singapore and we’ll store the image in Singapore. Virginia => Virginia. This makes it more reliable and faster to deploy your applications.

Deploy using registry2.flyio.net to test out this preview:

export FLY_REGISTRY_HOST=registry2.flyio.net
fly deploy

Make sure to run FLY_REGISTRY_HOST=registry2.flyio.net fly auth docker if you directly push images with docker push.

Let us know how it goes! We’d love to hear about bugs, performance & reliability changes, and anything else you encounter in the thread below.

Notes about the preview

The preview is starting with registry storage in iad (Ashburn, Virginia USA), sin (Singapore, Singapore), and syd (Sydney, Australia) regions. We’ll be adding more regions during the preview. The goal is to have storage in every region so deployments can be fast and reliable around the globe. We’re working on it!

You can switch back to the default registry at anytime by unsetting the FLY_REGISTRY_HOST environment variable and then re-running deployments. It is safe to switch between the default registry and registry2 as much as you need.

Other global registry use cases

Do you have a use case for a global docker registry outside of fly deploy? We’d love to hear from you! Post a message below or reach out to hello@fly.io to share those use cases.

We’re considering making a standalone registry product. No promises or anything yet. If we hear some real use cases that will definitely help us prioritize standalone registry work, though!

11 Likes
2

Giving this a quick test from github actions shows no improvement to deploy times. Tested with fly builder app both in iad and in syd.

Also, if the storage location of the docker image depends on the location of the fly CLI that runs the command, it’ll fail for anyone using pipelines such as github actions since the closest region is usually dfw.

It would be better if the docker image was replicated to any region you were deploying to transparently on first deploy of that image.

1 Like
3

Thanks for trying this out!

Pushes will get stored in whatever region is closest. For example, a fly deploy run in dfw would have the docker image stored in iad—that is currently the closest storage location to dfw. The registry then handles routing to wherever the image is stored for pulls (iad in this example).

I might have misunderstood this comment. What failure mode are you thinking about for github actions?

We want to do this. We’ll get there. Give us some time. :slight_smile:

1 Like
4

Sorry, fail was the wrong word. I meant that there wouldn’t be a benefit since dfw would be the closest region, which means the image storage would be in iad but the machines exist in syd or sin so you still end up with the cross planet transfers.

1 Like
5

Thanks for clarifying! You’re right, the current architecture does still have cross planet transfers. We have seen the pushes be a bit more sensitive to adverse network conditions, which is why we started this way.

We’re planning to automatically copy the images to more regions based on where apps are deployed to help improve pull performance. We’re working on it!

2 Likes
6

Is there any ETA for the global replication of images?

I have a 1 GB image hosted in registry.fly.io - it takes, <1 secs to download it in AMS.

But when I start a machine in LAX (US) it takes around 50 seconds to start a machine.

LAX

2023-08-04T22:31:06.228 runner[4d891d17f77d08] lax [info] Pulling container image registry.fly.io/my-image
2023-08-04T22:31:59.778 runner[4d891d17f77d08] lax [info] Successfully prepared image registry.fly.io/my-image (53.549659372s)
2023-08-04T22:32:00.237 runner[4d891d17f77d08] lax [info] Configuring firecracker

AMS

2023-08-04T22:35:52.069 runner[683d922f724d68] ams [info] Pulling container image registry.fly.io/my-image
2023-08-04T22:35:52.399 runner[683d922f724d68] ams [info] Successfully prepared image registry.fly.io/my-image (330.255621ms)
2023-08-04T22:35:53.140 runner[683d922f724d68] ams [info] Configuring firecracker