Cannot deploy docker app: Could not find image

Questions / Help
1

I’m trying to deploy a pre-built docker image:

flyctl deploy --remote-only --config packaging/fly.toml
==> Verifying app config
--> Verified app config
==> Building image
Searching for image 'evcc/evcc:latest' remotely...
Error failed to fetch an image or build from source: Could not find image "docker.io/evcc/evcc:latest"

The image itself exists and can be pulled:

docker pull evcc/evcc:latest
latest: Pulling from evcc/evcc
47517142f6ba: Already exists
83857b36862e: Pull complete
2acba76af065: Pull complete
6571068c986a: Pull complete
5724833c0436: Pull complete
443cfd89df8f: Pull complete
Digest: sha256:ef4b24b487215e1883c53c826a129d544d14c1ed7ddb6d10c847ee1cfd3795a2
Status: Downloaded newer image for evcc/evcc:latest
docker.io/evcc/evcc:latest

Why does the deploy fail?

2

Hi @andig ,

I see that the image gets pulled

=> [builder 1/8] FROM docker.io/evcc/evcc:latest@sha256:92554c8e0f4ec89abf679062ac56941b13554ba03692a13b00401906 1.7s
=> => resolve docker.io/evcc/evcc:latest@sha256:92554c8e0f4ec89abf679062ac56941b13554ba03692a13b0040190686a11dfe 0.0s
=> => sha256:a982297b7ac7d05dba9585914a828a11b62de8cee44579a4e402abad98b8b8bd 1.43kB / 1.43kB

Can you try again?

3

Same result:

flyctl deploy --remote-only --config packaging/fly.toml
==> Verifying app config
--> Verified app config
==> Building image
Searching for image 'evcc/evcc:latest' remotely...
Error failed to fetch an image or build from source: Could not find image "docker.io/evcc/evcc:latest"

This is actually funny:

flyctl deploy --local-only --config packaging/fly.toml
==> Verifying app config
--> Verified app config
==> Building image
Searching for image 'evcc/evcc:latest' locally...
image found: sha256:db321f5eadee1fa29fb8a9c5211ade2691519575c735e8c8daa6c6331b854a47
==> Pushing image to fly
The push refers to repository [registry.fly.io/evcc-demo]
e6721f06c4a4: Pushed
1d08dae34e30: Pushed
8580ddf2a516: Pushed
267c0b504279: Pushed
d0b778eb6859: Pushed
9833f26c6bda: Pushed
deployment-01GRNAHANDREGQFAB6RFX70FNE: digest: sha256:2c88171b6522b03642d75759aabda2b6c7448d796b1096ffe01568e53977751a size: 1573
--> Pushing image done
==> Creating release
Error Validation failed: Image must be amd64 architecture for linux us

Seems its looking at a different registry here?

4

Hi @andig ,

Can you post the contents of your packaging/fly.toml & Dockerfile?

Also, I see that in your local, you have tagged a arm64 image and the fly VM is amd64. Are you building locally on a Mac M1/RaspberryPi?

I think there is a difference in the architecture for what you have built and what you are pushing.

fly.io doesn’t support arm64 as of now.

I think the Image may be cached in the remote builder. Also, there can be connectivity issues between the remote builder and docker.io. Try building with a new builder.

I would also suggest you to update to the latest flyctl, if you are still running the old one.

5

The image ist multi-arch and has the required platform:

docker pull --platform linux/amd64 evcc/evcc
Using default tag: latest
latest: Pulling from evcc/evcc
9621f1afde84: Already exists
170c23284a81: Pull complete
1c704c3c1044: Pull complete
e03bb08a5d3a: Pull complete
c4ea6022b1f4: Pull complete
2b95cb32f24f: Pull complete
Digest: sha256:92554c8e0f4ec89abf679062ac56941b13554ba03692a13b0040190686a11dfe
Status: Downloaded newer image for evcc/evcc:latest
docker.io/evcc/evcc:latest

I really started seeing this problem on CI where we’re using the fly action: https://github.com/evcc-io/evcc/blob/master/.github/workflows/release.yml#L118. I have then started investigating locally on Mac M1. This is the CI log: Revert "Debian/Ubuntu: use non-root evcc user (#4901)" · evcc-io/evcc@b06eae9 · GitHub

Can you post the contents of your packaging/fly.toml & Dockerfile?

Sure, they are https://github.com/evcc-io/evcc/blob/master/packaging/fly.toml and https://github.com/evcc-io/evcc/blob/master/Dockerfile

I think the Image may be cached in the remote builder.

Shouldn’t the local build work then?

Also, there can be connectivity issues between the remote builder and docker.io. Try building with a new builder.

Not sure how I would do that? I must admit that the builder part is quite confusing as we’re really not building anything, only deploying the already-built image.

I would also suggest you to update to the latest flyctl, if you are still running the old one.

Done, no change.

1 Like
6

See:

7

I suspect Fly builders aren’t multi-arch aware or some regression has caused it…

Btw, one can push a prebuilt docker image to Fly (in your case, you’d probably want to push only the amd64 variant):

1 Like
8

No this won’t work, since you are pushing from M1 (arm64) and deploying to an amd64 VM. The image architecture are different. It is trying to deploy an arm64 image on an amd64 VM.

As @ignoramous pointed out the remote builders are not multi-arch aware and only supports amd64. Deleting the existing remote builder should launch you a new one.

I ran using your DockerHub image and the arch is correctly getting selected in my builder without any issues.

Remote builder fly-builder-***** ready
==> Creating build context
→ Creating build context done
==> Building image with Docker
→ docker host: 20.10.12 linux x86_64
Sending build context to Docker daemon 23.25kB
[+] Building 1.3s (6/6) FINISHED
=> [internal] load remote build context 0.0s
=> copy /context / 0.1s
=> [internal] load metadata for docker.io/evcc/evcc:latest 0.9s
=> CACHED [1/2] FROM docker.io/evcc/evcc:latest@sha256:92554c8e0f4ec89abf679062ac56941b13554ba03692a13b004019068 0.0s

The issue started to happen due to the changes in the Dockerfile to introduce BUILDPLATFORM. I would suggest use remote builder (x86-64) to push to the registry and not use your local.

1 Like
9

No builder that I see:

flyctl apps list
NAME     	OWNER   	STATUS 	PLATFORM	LATEST DEPLOY
evcc     	personal	running	nomad   	2023-02-03T09:14:50Z	
evcc-api 	personal	running	nomad   	2022-11-20T16:18:57Z	
evcc-mail	personal	pending	        	                    	
evcc-demo	evcc    	running	nomad   	2023-01-14T12:15:53Z

So still unsure how to recreate the builder.

I suspect Fly builders aren’t multi-arch aware or some regression has caused it…

What I do notice though is that the demo app has a different owner (i.e. org instead of personal GH account). I’m not sure how that happened or if it was intentional.

The issue started to happen due to the changes in the Dockerfile to introduce BUILDPLATFORM. I would suggest use remote builder (x86-64) to push to the registry and not use your local.

I’m still confused, what role (any remote/local) builder plays here. There is nothing to build; the image is ready. Why would the BUILDPLATFORM change that?

I would suggest use remote builder (x86-64) to push to the registry and not use your local.

Funny enough I just gave it another try locally and that is working now:

    fly deploy --local-only

    ==> Verifying app config
    --> Verified app config
    ==> Building image
    Searching for image 'evcc/evcc:latest' locally...
    image found: sha256:355dbe063b2cedf635daf22c1159fba1d757f932cc628e0b9069a29ca6ba9279
    ==> Pushing image to fly
    The push refers to repository [registry.fly.io/evcc-demo]
    fa96e5061e5a: Pushed
    89e30b2e6462: Pushed
    08a2ff92d5f1: Pushed
    ce6e84bf6850: Pushed
    f87ea68c3406: Pushed
    34d5ebaa5410: Mounted from evcc
    deployment-01GRNXM9JDEWWCJHYBQH73QD0R: digest: sha256:91b6350ee11b9d7b27b812c3dc065233bbc4eab06872d12837d2c224cce2d7c3 size: 1573
    --> Pushing image done
    ==> Creating release
    --> release v2 created

    --> You can detach the terminal anytime without stopping the deployment
    ==> Monitoring deployment
    Logs: https://fly.io/apps/evcc-demo/monitoring

    1 desired, 1 placed, 1 healthy, 0 unhealthy [health checks: 1 total, 1 passing]
    --> v2 deployed successfully

and

    flyctl deploy --local-only --config packaging/fly.toml
    ==> Verifying app config
    --> Verified app config
    ==> Building image
    Searching for image 'evcc/evcc:latest' locally...
    image found: sha256:355dbe063b2cedf635daf22c1159fba1d757f932cc628e0b9069a29ca6ba9279
    ==> Pushing image to fly
    The push refers to repository [registry.fly.io/evcc-demo]
    fa96e5061e5a: Layer already exists
    89e30b2e6462: Layer already exists
    08a2ff92d5f1: Layer already exists
    ce6e84bf6850: Layer already exists
    f87ea68c3406: Layer already exists
    34d5ebaa5410: Layer already exists
    deployment-01GRNXTPR6ZPPY0K3MGBTVEVV6: digest: sha256:91b6350ee11b9d7b27b812c3dc065233bbc4eab06872d12837d2c224cce2d7c3 size: 1573
    --> Pushing image done
    ==> Creating release
    --> release v3 created

    --> You can detach the terminal anytime without stopping the deployment
    ==> Monitoring deployment
    Logs: https://fly.io/apps/evcc-demo/monitoring

    1 desired, 1 placed, 1 healthy, 0 unhealthy
    --> v3 deployed successfully

No this won’t work, since you are pushing from M1 (arm64) and deploying to an amd64 VM. The image architecture are different. It is trying to deploy an arm64 image on an amd64 VM.

I disagree. The build is multi-arch including linux/arm64. Imho this should work (and is actually how I’m doing that for other images).

10

Same thing on the remote builder (which one is that?) still fails:

flyctl deploy --remote-only --config packaging/fly.toml
==> Verifying app config
--> Verified app config
==> Building image
Searching for image 'evcc/evcc:latest' remotely...
Error failed to fetch an image or build from source: Could not find image "docker.io/evcc/evcc:latest"

Which builder is used here (that would need recreating)?
Why is the image on docker.io not found (it exists)?

Also seems you have additional logging for diagnosis that is not accessible to me, or I’m looking in the wrong place?

I’m really thankful for your help but I still don’t understand how the pieces connect.

11

To make sure that the host platform is not the root cause, I’ve also run the same thing in an amd64 docker container:

docker run -it --platform linux/amd64 -v $(pwd):/app golang

root@57936477ec2a:/go# curl -L https://fly.io/install.sh | sh
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  1475    0  1475    0     0   3954      0 --:--:-- --:--:-- --:--:--  4019
######################################################################## 100.0%
set channel to shell
flyctl was installed successfully to /root/.fly/bin/flyctl
Manually add the directory to your $HOME/.bash_profile (or similar)
  export FLYCTL_INSTALL="/root/.fly"
  export PATH="$FLYCTL_INSTALL/bin:$PATH"
Run '/root/.fly/bin/flyctl --help' to get started

root@57936477ec2a:/go# /root/.fly/bin/flyctl deploy --remote-only --config /app/packaging/fly.toml
Error No access token available. Please login with 'flyctl auth login'


root@57936477ec2a:/go# /root/.fly/bin/flyctl auth login
failed opening browser. Copy the url (https://fly.io/app/auth/cli/...) into a browser and continue
Opening https://fly.io/app/auth/cli/... ...

Waiting for session... Done
successfully logged in as ...

root@57936477ec2a:/go# /root/.fly/bin/flyctl deploy --remote-only --config /app/packaging/fly.toml
==> Verifying app config
--> Verified app config
==> Building image
Searching for image 'evcc/evcc:latest' remotely...
Error failed to fetch an image or build from source: Could not find image "docker.io/evcc/evcc:latest"
12

Your Fly org doesn’t have a builder? Strange. What’s the output of fly doctor? If you see anything failing then, try: fly wg websockets enable (ref) and then exec fly deploy --verbose --remote-only ....

You can also force create a (free) builder for your org: remote builder machine stuck in created state - #6 by ray-chen

Something to keep in mind: Such build errors also crop up due to slow / flaky internet connections: Deployments using remote builder failing - #4 by jasmine

13

Hi @andig ,

I understand that you are running flyctl deploy --local-only within a container. The container’s DNS or networking may be causing the connectivity issue to the docker registry.

Try running the flyctl commands outside the container.

14

@ignoramous fly doctor all happy:

fly doctor --config packaging/fly.toml
Testing authentication token... PASSED
Testing flyctl agent... PASSED
Testing local Docker instance... PASSED
Pinging WireGuard gateway (give us a sec)... PASSED

App specific checks for evcc-demo:
Checking that app has ip addresses allocated... PASSED
Checking A record for evcc-demo.fly.dev... PASSED
Checking AAAA record for evcc-demo.fly.dev... PASSED

Your Fly org doesn’t have a builder? Strange.

I’m still mentally stuck on what a “builder” is doing in this scenario at all. There is nothing to build. Why the builder? What’s it’s role?

I understand that you are running flyctl deploy --local-only within a container. The container’s DNS or networking may be causing the connectivity issue to the docker registry.

@vvishnu-indian no, I’ve run remote-only in the container to check if a host (container) with matching amd64 arch does not show the problem. It does.

15

I’d like to pop in and say that I’ve got a similar env (nomad/ord region) without a builder and I can’t pull from docker.io since yesterday.

Edit: I force created a builder by removing my build step and added a blank dockerfile, then reverted and tried to ship that same docker.io container, still not working.

1 Like
16

Hey @andig and @trevorm, it sounds like you may be running into the Buildkit issue described in this thread:

Images built using buildkit v 0.10.0 return a 404 error when we try to pull the image from docker.io, or from our registry. That post describes a work around to force buildkit to use an older version when building an image. Since you’re using a prebuilt public image that won’t necessarily help in this case.

As you’ve found out already, running fly deploy with the --local-only flag will work, because this uses your local docker daemon. If a local build isn’t an option for you, you can try downloading their public dockerfile and using fly launch to generate a fly.toml and build the image.

If you don’t have an active remote builder on your org we automatically create one during the fly deploy process. Since your deploy is failing before we create the builder, you won’t see it listed.

1 Like
17

Hi @andig ,

I was finally able to recreate your exact errors.

  1. Deleted my remote builder
  2. Added

[build]
image = “evcc/evcc:latest”

in my fly.toml

  1. Did a fly deploy

App is not running, deploy…
==> Building image
Searching for image ‘evcc/evcc:latest’ remotely…
Error failed to fetch an image or build from source: Could not find image “docker.io/evcc/evcc:latest”

It can be noted that fly is not launching a new remote builder in this case and is failing.

To fix this, I removed

[build]
image = “evcc/evcc:latest”

from my fly.toml and did a deploy and a new builder got created and the image from docker.io got pulled successfully.

==> Verifying app config
→ Verified app config
==> Building image
Remote builder fly-builder-old-cloud-5251 ready
==> Creating build context
→ Creating build context done
==> Building image with Docker
→ docker host: 20.10.12 linux x86_64
[+] Building 0.3s (0/1)
[+] Building 4.7s (6/6) FINISHED
=> [internal] load remote build context 0.0s
=> copy /context / 0.1s
=> [internal] load metadata for docker.io/evcc/evcc:latest 1.2s
=> [1/2] FROM docker.io/evcc/evcc:latest@sha256:92554c8e0f4ec89abf679062ac56941b13554ba03692a13b0040190686a11dfe 3.1s

Without a remote builder --remote-only flag during deployment is of no use. May be flyctl can provide a better error message.

→ Pushing image done
image: registry.fly.io/**************

The Dockerfile for my app has the following and the dockerfile builder is the only option that can work for your case.

FROM evcc/evcc:latest

It has also been noted that the flyctl image builder is only working for public images in DH and not for all.

This image works andig/evcc:latest

==> Building image
Searching for image ‘andig/evcc:latest’ remotely…
image found: img_xdkzvqmmwddp9nmw

But not with evcc/evcc:latest

Searching for image ‘evcc/evcc:latest’ remotely…
Error failed to fetch an image or build from source: Could not find image “docker.io/evcc/evcc:latest”

Looks like an issue with the evcc user on DH.

1 Like
18

Thank you @vvishnu-indian that is a perfect explanation. I can repro the builder getting created when remove the image line. Re-adding the line will not recreate a remote builder though and still fails. Seems we’ll need to use --local-only for this scenario.

A better error message would indeed help here.

Thanks for all your support!

2 Likes
19

That was a curved ball!

If you do have a better error message in mind, consider leaving a comment here: Confusing `fly deploy --remote-only` error with fly.toml `[build]` entry · Issue #1686 · superfly/flyctl · GitHub

1 Like
20

Hi @andig ,

I set LOG_LEVEL=debug and ran flyctl deploy to find why the evcc/evcc:latest is showing error where as the other images or not.

“errors”: [
{
“message”: “Could not find image "docker.io/evcc/evcc:latest"”,
“locations”: [
{
“line”: 1,
“column”: 74
}
],
“path”: [
“app”,
“image”
],
“extensions”: {
“code”: “NOT_FOUND”
}
}

“data”: {
“app”: {
“id”: “samplbes-db1”,
“image”: {
“id”: “img_xdkzvqmmwddp9nmw”,
“digest”: “sha256:e663f35d42ae89840c29ac040f3c96d6973b4f858ff96b0fc53f4f429b6d93d3”,
“ref”: “registry-1.docker.io/andig/evcc:latest”,
“compressedSize”: 20176252
}
}
}

This clearly shows that the API of fly is returning this error. Is evcc/evcc:latest a private registry?

You can still deploy to fly using dockerfile builder as it works with evcc/evcc:latest. Wondering how come that builder works if evcc/evcc:latest is a private one.

If we solve this your issue gets resolved.

1 Like