Fly API errors when deploying a Docker image built with GNU Guix

Hey there,

I’ve read people have had success deploying Docker images built with Nix. I’m trying to deploy a simple test image I built with GNU Guix, which is a package manager/distribution based on Nix.

On my first attempt I pushed this image to registry.fly.io myself:

$ podman push --creds x:<fly-api-token> redir-server:latest docker://registry.fly.io/redir-server:34abqrndvs0fy5kf4gxgn7s7rbmckhvg
Getting image source signatures
Copying blob sha256:868e42a984a31ad39185251b1e12cafdcc1a52891b7985a195a3e78e48b23684
Copying config sha256:dbe7b14ba0688f40706250bb48effd2af14c8fd8729e5e5ae1e5ef2d5d7fdb7b
Writing manifest to image destination
Storing signatures

I then continued with flyctl deploy, but this resulted in a Fly API error without further detail:

$ flyctl deploy --image registry.fly.io/redir-server:34abqrndvs0fy5kf4gxgn7s7rbmckhvg
==> Verifying app config
--> Verified app config
==> Building image
Searching for image 'registry.fly.io/redir-server:34abqrndvs0fy5kf4gxgn7s7rbmckhvg' remotely...
Error failed to fetch an image or build from source: You hit a Fly API error with request ID: 01GH880GK6K3XNA36S929P7WZP-fra

Rather than pushing the image manually, I tried getting flyctl deploy to push the image automatically, however this resulted in a similar error:

$ flyctl deploy --image localhost/redir-server:latest --local-only
==> Verifying app config
--> Verified app config
==> Building image
Searching for image 'localhost/redir-server:latest' locally...
image found: sha256:dbe7b14ba0688f40706250bb48effd2af14c8fd8729e5e5ae1e5ef2d5d7fdb7b
==> Pushing image to fly
The push refers to repository [registry.fly.io/redir-server:deployment-01GHA4GHFFF3DTAH1TPPFP77F7]
deployment-01GHA4GHFFF3DTAH1TPPFP77F7: digest: sha256:a8d4cf02e90069ff2716659366f28c5ef7181d35cba5843e55f6419c8a81251a size: 428
--> Pushing image done
==> Creating release
Error You hit a Fly API error with request ID: 01GHA4GNMEVNG58JE4B1B6QM27-fra

I’ve tried these approaches both yesterday and today in case this was a temporary error, but it seems this is persistent, and only happens with these images. I’ve been able to deploy other Docker images successfully, e.g. alpine:latest.

Even though both podman and docker are able to run these images, the internal structure of the images built by GNU Guix is slightly different from other images I downloaded from Docker Hub.

I’m thinking maybe it could be some difference in the image configuration. I’ve attached some of the relevant files from Alpine and my test image below, in case you’re able to spot something. I’m also able to provide a copy of this image if necessary.

Some things I’ve already tried:

  • Adding a CMD to the image in addition to an ENTRYPOINT. I used [experimental] in fly.toml to do this.
  • Importing the image into Docker, and re-exporting it. This produced an image with a format more similar to the Alpine image, but this resulted in the same error.

fly.toml

app = "redir-server"

kill_signal = "SIGTERM"

[[services]]
internal_port = 3000
protocol = "tcp"

[[services.ports]]
force_https = true
handlers = ["http"]
port = 80

[[services.ports]]
handlers = ["http", "tls"]
port = 443

Image tar archive

Here’s alpine:latest:

$ tar -tf alpine.tar 
994393dc58e7931862558d06e46aa2bb17487044f670f310dffe1d24e4d1eec7.tar
9c6f0724472873bb50a2ae67a9e7adcb57673a183cea8b06eb778dca859181b5.json
9b7240956cfbfefddcd91a2195bfb2ed2cd17bdff81f21111849d643dfaf8131/layer.tar
9b7240956cfbfefddcd91a2195bfb2ed2cd17bdff81f21111849d643dfaf8131/VERSION
9b7240956cfbfefddcd91a2195bfb2ed2cd17bdff81f21111849d643dfaf8131/json
manifest.json
repositories

Here’s my test image:

$ tar -tf 34abqrndvs0fy5kf4gxgn7s7rbmckhvg-redir-server-docker-pack.tar.gz 
./
./config.json
./2e9b4c2b54d610d4218761daee6e2ae0e5357485307f965932d51a4fe1321685/
./2e9b4c2b54d610d4218761daee6e2ae0e5357485307f965932d51a4fe1321685/VERSION
./2e9b4c2b54d610d4218761daee6e2ae0e5357485307f965932d51a4fe1321685/layer.tar
./2e9b4c2b54d610d4218761daee6e2ae0e5357485307f965932d51a4fe1321685/json
./repositories
./manifest.json

Image manifest.json

Here’s alpine:latest:

$ tar -xf alpine.tar -O manifest.json | jq .
[
  {
    "Config": "9c6f0724472873bb50a2ae67a9e7adcb57673a183cea8b06eb778dca859181b5.json",
    "RepoTags": [],
    "Layers": [
      "994393dc58e7931862558d06e46aa2bb17487044f670f310dffe1d24e4d1eec7.tar"
    ]
  }
]

Here’s my test image:

$ tar -xf 34abqrndvs0fy5kf4gxgn7s7rbmckhvg-redir-server-docker-pack.tar.gz -O manifest.json | jq . 
[
  {
    "Config": "config.json",
    "RepoTags": [
      "redir-server:latest"
    ],
    "Layers": [
      "2e9b4c2b54d610d4218761daee6e2ae0e5357485307f965932d51a4fe1321685/layer.tar"
    ]
  }
]

Image config

Here’s alpine:latest:

$ tar -xf alpine.tar -O 9c6f0724472873bb50a2ae67a9e7adcb57673a183cea8b06eb778dca859181b5.json | jq .
{
  "architecture": "amd64",
  "config": {
    "Hostname": "",
    "Domainname": "",
    "User": "",
    "AttachStdin": false,
    "AttachStdout": false,
    "AttachStderr": false,
    "Tty": false,
    "OpenStdin": false,
    "StdinOnce": false,
    "Env": [
      "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
    ],
    "Cmd": [
      "/bin/sh"
    ],
    "Image": "sha256:c0261ca8a4a79627f3e658c0c2b1e3166f56713a58e1411b1e3ab1e378962e75",
    "Volumes": null,
    "WorkingDir": "",
    "Entrypoint": null,
    "OnBuild": null,
    "Labels": null
  },
  "container": "0c4b20968eb1d804460f8612bc52be4f5a8e65eb14190b5ae30fec94d2fb4f50",
  "container_config": {
    "Hostname": "0c4b20968eb1",
    "Domainname": "",
    "User": "",
    "AttachStdin": false,
    "AttachStdout": false,
    "AttachStderr": false,
    "Tty": false,
    "OpenStdin": false,
    "StdinOnce": false,
    "Env": [
      "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
    ],
    "Cmd": [
      "/bin/sh",
      "-c",
      "#(nop) ",
      "CMD [\"/bin/sh\"]"
    ],
    "Image": "sha256:c0261ca8a4a79627f3e658c0c2b1e3166f56713a58e1411b1e3ab1e378962e75",
    "Volumes": null,
    "WorkingDir": "",
    "Entrypoint": null,
    "OnBuild": null,
    "Labels": {}
  },
  "created": "2022-08-09T17:19:53.47374331Z",
  "docker_version": "20.10.12",
  "history": [
    {
      "created": "2022-08-09T17:19:53.274069586Z",
      "created_by": "/bin/sh -c #(nop) ADD file:2a949686d9886ac7c10582a6c29116fd29d3077d02755e87e111870d63607725 in / "
    },
    {
      "created": "2022-08-09T17:19:53.47374331Z",
      "created_by": "/bin/sh -c #(nop)  CMD [\"/bin/sh\"]",
      "empty_layer": true
    }
  ],
  "os": "linux",
  "rootfs": {
    "type": "layers",
    "diff_ids": [
      "sha256:994393dc58e7931862558d06e46aa2bb17487044f670f310dffe1d24e4d1eec7"
    ]
  }
}

Here’s my test image:

$ tar -xf 34abqrndvs0fy5kf4gxgn7s7rbmckhvg-redir-server-docker-pack.tar.gz -O config.json | jq .
{
  "architecture": "amd64",
  "comment": "Generated by GNU Guix",
  "created": "1970-01-01T00:00:01Z",
  "config": {
    "env": [
      "PATH=/gnu/store/dgydn8f9kcxr8msra2ims21l4n8h0jck-profile/bin"
    ],
    "entrypoint": [
      "/gnu/store/dgydn8f9kcxr8msra2ims21l4n8h0jck-profile/bin/redir-server"
    ]
  },
  "container_config": null,
  "os": "linux",
  "rootfs": {
    "type": "layers",
    "diff_ids": [
      "sha256:868e42a984a31ad39185251b1e12cafdcc1a52891b7985a195a3e78e48b23684"
    ]
  }
}

Hey, can you try this again? We’ve issued a fix that should address the problem.

@jsierles Thanks for the reply! Still seeing errors unfortunately, I’ve got some new request IDs for you:

$ flyctl deploy --image registry.fly.io/redir-server:latest                          
==> Verifying app config
--> Verified app config
==> Building image
Searching for image 'registry.fly.io/redir-server:latest' remotely...
Error failed to fetch an image or build from source: You hit a Fly API error with request ID: 01GHC1EQZ0VBCMWATCAZPYWANG-fra

$ flyctl deploy --image localhost/redir-server:latest --local-only      
==> Verifying app config
--> Verified app config
==> Building image
Searching for image 'localhost/redir-server:latest' locally...
image found: sha256:dbe7b14ba0688f40706250bb48effd2af14c8fd8729e5e5ae1e5ef2d5d7fdb7b
==> Pushing image to fly
The push refers to repository [registry.fly.io/redir-server:deployment-01GHC1F96Q40BHJSPTY1D9BM54]
deployment-01GHC1F96Q40BHJSPTY1D9BM54: digest: sha256:a8d4cf02e90069ff2716659366f28c5ef7181d35cba5843e55f6419c8a81251a size: 428
--> Pushing image done
==> Creating release
Error You hit a Fly API error with request ID: 01GHC1FCK70CEPEE9PN01PH1ZC-fra

Thanks - can you try again now?

It works, thanks a lot for the help! Out of curiosity, was this an issue with some of the image metadata?

Yes - we dig around in image metadata, and images created by nix or guix tend to be quite sparse :slight_smile:

1 Like