Using the Fly Machines API with private images

[x7ryan]

For context I’m working on a POC for a setup where I’m going to use the Machines API to provision a new instance of an app on demand when a customer signs up. @fideloper-fly was very helpful on X in helping me to setup Secrets and IP’s programmatically even though those are on the Machines API.

However I ran into a wall. The app and secret provisioning is working smoothly in this POC I built, but the machine build is failing because it cannot pull the image. So I have the image being build in GH Actions and stored in the GH Container Registry, but since its from a private repo the image is private too so you need to authenticate to pull it - which is exactly what I want anyway, but apparently according to @kurt private third party registries are not supported by fly. Though this would be so much easier if fly would just let me authenticate to an external image registry like GHCR.

They seem to push you to just use the fly registry instead. Since images on the fly registry have to be tied to an app, I’d have to create an app without deploying it, just to house that image. That also has a problem that images won’t be persisted beyond a few days, which makes sense in a lot of cases because Fly is paying for that storage and it predominately is used as a temporary registry for images built using fly builders from the CLI. But for my use case makes it a no go, I also kinda suspect it cannot have multiple versions of an image, it will only keep the latest though I cannot confirm that.

The problem is for my use case I need to access the latest version, potentially the last few versions, of an image essentially in perpetuity. Putting my image on a public registry is not at all ideal, even if its hosted at a super obscure URL I’d much rather keep the image private.

Does anyone have any other suggestions?

[x7ryan]

So I did some testing today. I found an article on running a Registry as a pull through cache. I was able to deploy that as a fly app, and get it working where it would authenticate to GHCR for me but make the images available from its local cache without authentication. This does also gives the benefit for a local cache for the images for even less latency. But it is publicly available, I can somewhat offset that by hosting it at a long obscure domain, but still far from ideal.

I did test if I could get it to work though fly’s local networking, I removed the public IPs from that app and used (app-name).internal for the hostname, unfortunately I found that didn’t quite work, I got an “Unsupported registry” error. I assume the infrastructure that is pulling it technically exists outside my org so it cannot route to that app.

So I am back at the beginning. Still open to any other ideas anyone may have.

EDIT: I think for now given this limitation, I will skip using Fly and might revisit this in another year or 2 to see if there’s a better story around using private images hosted outside the fly registry, or else a way to pay to persist images in the fly registry at least.