Does image transmogrification support `vnd.oci.image.layer.v1.tar+zstd`?

Questions / Help
1

Hi,

I’m writing a CNB platform which generates OCI images that conform to the spec (as part of a Rust CLI tool which implements buildpacks as well). I want to deploy those images on Fly.

The spec allows for the application/vnd.oci.image.layer.v1.tar+zstd media type for layers. zstd is roughly twice as fast a gz and I’ve implemented that as the default compression algorithm. Does Fly’s image transmogrification support zstd?

Docker client v20.10.7 does not “failed to register layer: Error processing tar file(exit status 1): archive/tar: invalid tar header”

I can try it out on Fly (and will report back here when I do) but thought I’d ask here first in case the answer is “Yes it should”.

2

Can’t say definitively, but since Fly uses containerd (which supports zstd) to “run” Docker, I can’t see why not. Let us know how it goes!

3

Thanks for the pointers @ignoramous (although I think that first link should be Docker without Docker · Fly).

Unfortunately, I haven’t been able to get to the point of testing support for tar+zstd layers because the manifest media type application/vnd.oci.image.manifest.v1+json does not seem to be supported…

  1. I can build and push an image for my test app to registry.fly.io:
> cargo run images -- build --from ubuntu --tag registry.fly.io/stencila-test ../../fixtures/projects/apt/names/

<snip>

  2022-05-30T00:20:12.003088Z  INFO images::distribution: Pushing blob `sha256:4b7c4f557aa5865c89ef4e681ed3331cdff8c1cb16748df3a36f8b89c9b3f53e` to `registry.fly.io/stencila-test`
    at rust/images/src/distribution.rs:497

  2022-05-30T00:20:20.023523Z  INFO images::cli: Image built and pushed to `registry.fly.io/stencila-test:latest`.
    at rust/images/src/cli.rs:157

working_dir: "../../fixtures/projects/apt/names/"
ref:
  registry: registry.fly.io
  repository: stencila-test
  tag: latest
  digest: ~
base:
  registry: registry.hub.docker.com
  repository: library/ubuntu
  tag: ~
  digest: ~
layer_dirs:
  - "../../fixtures/projects/apt/names"
layer_diffs: true
layer_format: application/vnd.oci.image.layer.v1.tar+gzip
layout_dir: temp
  1. I can confirm that the image was successfully pushed by getting it’s manifest:
> xh -A bearer -a $FLY_TOKEN GET registry.fly.io/v2/stencila-test/manifests/latest Accept:application/vnd.oci.image.manifest.v1+json

HTTP/1.1 200 OK
Content-Length: 676
Content-Type: application/vnd.oci.image.manifest.v1+json
Date: Mon, 30 May 2022 00:20:35 GMT
Docker-Content-Digest: sha256:fab73b98f5c4e35d3e0a76bda597bdfb566912fa7702122b4452cd3bc89cc415
Docker-Distribution-Api-Version: registry/2.0
Etag: "sha256:fab73b98f5c4e35d3e0a76bda597bdfb566912fa7702122b4452cd3bc89cc415"
Fly-Request-Id: 01G495BYY8BZ4FAK1KMYWG9DVT-syd
Server: Fly/ccc539245 (2022-05-20)
Via: 1.1 fly.io

{
    "schemaVersion": 2,
    "mediaType": "application/vnd.oci.image.manifest.v1+json",
    "config": {
        "mediaType": "application/vnd.oci.image.config.v1+json",
        "digest": "sha256:6e147ff4b27659d4e5f9c9be6267bb7202d69f1c4ba1238eabe86d1c225d5312",
        "size": 1188
    },
    "layers": [
        {
            "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
            "digest": "sha256:125a6e411906fe6b0aaa50fc9d600bf6ff9bb11a8651727ce1ed482dc271c24c",
            "size": 30421006
        },
        {
            "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
            "digest": "sha256:4b7c4f557aa5865c89ef4e681ed3331cdff8c1cb16748df3a36f8b89c9b3f53e",
            "size": 938
        }
    ]
}
  1. But when I try to create a Fly Machine with that image, I get a 404
> xh -A bearer -a $FLY_TOKEN POST :4280/v1/apps/stencila-test/machines name=machine-1 config:='{"image": "registry.fly.io/stencila-test"}'

HTTP/1.1 404 Not Found
Content-Length: 69
Content-Type: application/json; charset=utf-8
Date: Mon, 30 May 2022 00:19:34 GMT
Fly-Trace-Id: a0ca8a2717b863f072539697756c6028

{
    "error": "Could not find image \"registry.fly.io/stencila-test\""
}

Note that if I leave the Accept:application/vnd.oci.image.manifest.v1+json header off the request in (2) I get a 404 also:

> xh -A bearer -a $FLY_TOKEN GET registry.fly.io/v2/stencila-test/manifests/latest

HTTP/1.1 404 Not Found
Content-Type: application/json; charset=utf-8
Date: Mon, 30 May 2022 00:22:41 GMT
Docker-Distribution-Api-Version: registry/2.0
Fly-Request-Id: 01G495FT3WQEVM6QP5YDWVNQZJ-syd
Server: Fly/ccc539245 (2022-05-20)
Transfer-Encoding: chunked
Via: 1.1 fly.io

{
    "errors": [
        {
            "code": "MANIFEST_UNKNOWN",
            "message": "OCI manifest found, but accept header does not support OCI manifests"
        }
    ]
}

Also note that if I do all of by pushing the image using docker push registry.fly.io/stencila-test I get a manifest from the server with media type application/vnd.docker.distribution.manifest.v1+prettyjws and POST ../machines works fine.

I’m going to try to see if pushing the manifest to the registry using application/vnd.docker.distribution.manifest.v2+json works (equivalent to application/vnd.oci.image.manifest.v1+json I think) and if that fails might look into generating application/vnd.docker.distribution.manifest.v1+json. But if Fly could support the current OCI spec, that’d of course be better :slight_smile:

1 Like
4

Can confirm that simply using application/vnd.docker.distribution.manifest.v2+json instead of application/vnd.oci.image.manifest.v1+json when pushing images to registry.fly.io fixes the 404 issue and I can start Fly machines with the images I’m pushing, including layers that are tar+zstd encoded :tada:

Still have to confirm that they are functioning properly (a bit of image config to do) and will report back when/if I get that sorted.

2 Likes