Direct push to registry.fly.io // OCI Configs/Manifests not supported ?

Hi,

I am trying to find a “minimalist” way to upload and run OCI container images to Fly.io. No docker, no builder, no Dockerfile, just a rootfs tar.gz together with OCI manifest and configuration JSONs. Directly “pushed” to registry.fly.io, without using any other Docker container image repository.

My app on Fly is named “test-lachine”.

I pushed an OCI image manifest (content-type application/vnd.oci.image.manifest.v1+json), and its two blobs (config, content-type application/vnd.oci.image.config.v1+json, and rootfs tar.gz, content-type application/vnd.oci.image.layer.v1.tar+gzip) to registry.fly.io/test-lachine, using HTTP basic authentification x:[[fly access token]] and OCI distribution REST API.

The manifest, as well as the blobs it references, are accepted, and can be fetched (see curl logs below).

However, when I try to deploy the app with “flyctl deploy”, I get:

==> Verifying app config
--> Verified app config
==> Building image
Searching for image 'registry.fly.io/test-lachine' remotely...
Error failed to fetch an image or build from source: You ran into an error connecting to the Fly API. If the error persists, keep this request ID to help Fly Support track it down: 01G685BENR82YGHP4TEFF99Q49-cdg

When trying to see the request sent (on a different container) I can see that a GET request on /v2/test/manifests/latest is made with Accept: application/vnd.docker.distribution.manifest.v2+json, application/json by User-Agent: rest-client/2.1.0 (linux x86_64) ruby/3.1.2p20), suggesting that OCI manifests are not supported ?

Or maybe the problem comes from that the fact that the fetch requests are not authenticated ? If so, is it possible to deploy images already pushed to the internal Fly.io registry, without using any other registry ?

Thanks,

cURL logs showing that the manifests and blobs can be fetched:

> GET /v2/test-lachine/manifests/latest HTTP/2
> Host: registry.fly.io
> user-agent: curl/7.83.1
> authorization: Basic **REDACTED**
> accept: application/vnd.oci.image.manifest.v1+json
>
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* Connection state changed (MAX_CONCURRENT_STREAMS == 4294967295)!
< HTTP/2 200
< content-length: 419
< content-type: application/vnd.oci.image.manifest.v1+json
< docker-content-digest: sha256:1fbb69be2e87c74dfd7e513c12ad218f44d686ddfec4e9e3b109d402a3fc14e9
< docker-distribution-api-version: registry/2.0
< etag: "sha256:1fbb69be2e87c74dfd7e513c12ad218f44d686ddfec4e9e3b109d402a3fc14e9"
< date: Thu, 23 Jun 2022 11:34:47 GMT
< server: Fly/9ece5bcd (2022-06-21)
< via: 2 fly.io
< fly-request-id: 01G685FMBQCCSQFH8DEXTZDVPS-cdg
<
{
  "schemaVersion": 2,
  "config": {
    "mediaType": "application/vnd.oci.image.config.v1+json",
    "size": 291,
    "digest": "sha256:c1cab53b10fa47a29918028e3d4679e9b38ef5002477470de3d56c72fe7a0942"
  },
  "layers": [
    {
      "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
      "size": 717142,
      "digest": "sha256:bf075635a2e2a31f602096193389716776c9d8de7d0f7dd925caec89c65ae089"
    }
  ]
}
* Connection #0 to host registry.fly.io left intact

> GET /v2/test-lachine/blobs/sha256:c1cab53b10fa47a29918028e3d4679e9b38ef5002477470de3d56c72fe7a0942 HTTP/2
> Host: registry.fly.io
> user-agent: curl/7.83.1
> accept: */*
> authorization: Basic **REDACTED**
>
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* Connection state changed (MAX_CONCURRENT_STREAMS == 4294967295)!
< HTTP/2 200
< accept-ranges: bytes
< cache-control: max-age=31536000
< content-length: 291
< content-type: application/octet-stream
< docker-content-digest: sha256:c1cab53b10fa47a29918028e3d4679e9b38ef5002477470de3d56c72fe7a0942
< docker-distribution-api-version: registry/2.0
< etag: "sha256:c1cab53b10fa47a29918028e3d4679e9b38ef5002477470de3d56c72fe7a0942"
< date: Thu, 23 Jun 2022 11:39:43 GMT
< server: Fly/9ece5bcd (2022-06-21)
< via: 2 fly.io
< fly-request-id: 01G685RGSH0Y3HZ8NWBX22YERA-cdg
<
{
  "architecture": "amd64",
  "os": "linux",
  "config": {
    "ExposedPorts": { "8080/tcp":{} },
    "Cmd": [ "/sbin/httpd","-p","8080","-f","-v" ]
  },
  "rootfs": {
    "diff_ids": [ "sha256:4ea65d4a3ad2286ef38d081c5d85825af7411ab1490b71f93f8f2264ef3fca55" ],
    "type": "layers"
  }
}

> GET /v2/test-lachine/blobs/sha256:bf075635a2e2a31f602096193389716776c9d8de7d0f7dd925caec89c65ae089 HTTP/2
> Host: registry.fly.io
> user-agent: curl/7.83.1
> accept: */*
> authorization: Basic REDACTED
>
{ [5 bytes data]
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
{ [81 bytes data]
* Connection state changed (MAX_CONCURRENT_STREAMS == 4294967295)!
} [5 bytes data]
< HTTP/2 200
< accept-ranges: bytes
< cache-control: max-age=31536000
< content-length: 717142
< content-type: application/octet-stream
< docker-content-digest: sha256:bf075635a2e2a31f602096193389716776c9d8de7d0f7dd925caec89c65ae089
< docker-distribution-api-version: registry/2.0
< etag: "sha256:bf075635a2e2a31f602096193389716776c9d8de7d0f7dd925caec89c65ae089"
< date: Thu, 23 Jun 2022 11:41:41 GMT
< server: Fly/9ece5bcd (2022-06-21)
< via: 2 fly.io
< fly-request-id: 01G685W911ZGD71FCAHJAZ3X6R-cdg
<
[[[ ... data .... with correct SHA256.]]]
1 Like

I’m trying to do the same thing. I don’t have time to respond in detail but my post a while back might be useful Does image transmogrification support `vnd.oci.image.layer.v1.tar+zstd`?

1 Like

Thank you, will check it ! Looks the same indeed :wink: What MIME type do you use for the config.json ? Does the OCI MIME works for configs ?

Now with this manifest:

< HTTP/2 200
< content-length: 490
< content-type: application/vnd.docker.distribution.manifest.v2+json
< docker-content-digest: sha256:61d3ab621bea865bb370865bbc17e2969efc12add76f9556760c927de3364711
< docker-distribution-api-version: registry/2.0
< etag: "sha256:61d3ab621bea865bb370865bbc17e2969efc12add76f9556760c927de3364711"
< date: Thu, 23 Jun 2022 12:49:43 GMT
< server: Fly/9ece5bcd (2022-06-21)
< via: 2 fly.io
< fly-request-id: 01G689SR815DMPS3HTXSPFA4ZV-cdg
<
{
  "schemaVersion": 2,
  "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
  "config": {
    "mediaType": "application/vnd.oci.image.config.v1+json",
    "size": 291,
    "digest": "sha256:c1cab53b10fa47a29918028e3d4679e9b38ef5002477470de3d56c72fe7a0942"
  },
  "layers": [
    {
      "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
      "size": 717142,
      "digest": "sha256:bf075635a2e2a31f602096193389716776c9d8de7d0f7dd925caec89c65ae089"
    }
  ]
}

I still have the same error.

On the host I control I see the config file is fetched, maybe the error is with it ?

Quoting the solution @nokome shared,

@alex14fr Is that what you attempted?

1 Like

I never have 404 issues when pushing images.

What I have today is the following:

  • when using OCI v1 manifest MIME, push is ok but starting machine fails with " Could not find image “registry.fly.io/monalisa1:latest

  • when using Docker manifest V2, I get " You ran into an error connecting to the Fly API. If the error persists, keep this request ID to help Fly Support track it down: 01G6D3YRR5Q62CHKA4WGPJN1KM-cdg"

Command to start the machine:

fly machines run registry.fly.io/monalisa1 -a monalisa1

Manifest JSON (with docker mime, similar for OCI mime):

{"schemaVersion":2,"mediaType":"application/vnd.docker.distribution.manifest.v2+json","config":{"mediaType":"application/vnd.oci.image.config.v1+json","size":358,"digest":"sha256:80d0128fceb373b6019dc4871fa96eb4bc8109fe9e39f19300c8c641bf4f2793"},"layers":[{"mediaType":"application/vnd.oci.image.layer.v1.tar+gzip","size":717142,"digest":"sha256:bf075635a2e2a31f602096193389716776c9d8de7d0f7dd925caec89c65ae089"}]}

Config JSON:

{"created":"2022-06-23T21:37:04+02:00","author":"xxx <xxx@gmail.com>","architecture":"amd64","os":"linux","rootfs":{"type":"layers","diff_ids":["sha256:4ea65d4a3ad2286ef38d081c5d85825af7411ab1490b71f93f8f2264ef3fca55"]},"config":{"Env":["HOME=/tmp"],"WorkingDir":"/","Entrypoint":["/sbin/httpd"],"Cmd":["-p","8080","-f","-v"],"ExposedPorts":{"8080/tcp":{}}}}

However, using the following Dockerfile:

FROM registry.fly.io/monalisa1:latest
RUN echo 'hello world' > index.html

and building with flyctl deploy works, as it transmogrifies my image, pushes registry.fly.io/monalisa1:deployment-1656150630 , with a Docker V2 manifest:

{
   "schemaVersion": 2,
   "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
   "config": {
      "mediaType": "application/vnd.docker.container.image.v1+json",
      "size": 865,
      "digest": "sha256:0df0c2eff78465916ca80d221aeb1c2ffdc6d9ce41f71e6dc563e2b1a6d4e559"
   },
   "layers": [
      {
         "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
         "size": 717142,
         "digest": "sha256:bf075635a2e2a31f602096193389716776c9d8de7d0f7dd925caec89c65ae089"
      },
      {
         "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
         "size": 248,
         "digest": "sha256:34ea0c494e5888e7602ddba3b32e99916c0c7362732d603a2f71fb5d4ca3f237"
      }
   ]
}

referencing a, different, Docker V1 config:

{"architecture":"amd64","author":"xxx \u003cxxx@gmail.com\u003e","config":{"ExposedPorts":{"8080/tcp":{}},"Env":["HOME=/tmp","PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"],"Entrypoint":["/sbin/httpd"],"Cmd":["-p","8080","-f","-v"],"WorkingDir":"/","OnBuild":null},"created":"2022-06-25T09:50:51.121912362Z","history":[{"created":"2022-06-25T09:50:50.865262654Z","created_by":"RUN /bin/sh -c echo 'hello world' \u003e index.html # buildkit","comment":"buildkit.dockerfile.v0"},{"created":"2022-06-25T09:50:51.121912362Z","created_by":"mount / from exec /bin/sh -c echo 'hello world' \u003e index.html","comment":"buildkit.exporter.image.v0"}],"os":"linux","rootfs":{"type":"layers","diff_ids":["sha256:4ea65d4a3ad2286ef38d081c5d85825af7411ab1490b71f93f8f2264ef3fca55","sha256:0d0118d5b6316752613a6c8075de8605a1b06a468310c583d68a5db13ecef623"]}}

Hence my guess is that Docker V2 schema 2 configs or OCI configs are not supported by machine start (they are indirectly if you use a “stub” docker build, as they seem to use a different “fetcher” for the FROM image of the Dockerfile).

PS. The code I use to push, fetch to docker repository, and generate manifest.json/config.json is here : GitHub - alex14fr/oci-imgtools

1 Like