I’m pushing an image to registry.fly.io and then deploying it to 2 apps in parallel (GitHub Actions workflow).
Since I’ve added the second app I’m noticing that some builds are failing with a “Digest has already been taken” message:
Run flyctl deploy --config fly.sandbox.toml --image registry.fly.io/prereview:525c8cf73bb3a3d91648277db74addda2db88e55
==> Verifying app config
--> Verified app config
==> Building image
Searching for image 'registry.fly.io/prereview:525c8cf73bb3a3d91648277db74addda2db88e55' locally...
Searching for image 'registry.fly.io/prereview:525c8cf73bb3a3d91648277db74addda2db88e55' remotely...
Error failed to fetch an image or build from source: Validation failed: Digest has already been taken
Comparing the timestamps on the logs for the 2 jobs, the other one started to search for the remote image ~20ms earlier and succeeded. When I restart the failed job it deploys successfully.
I’m not sure where the error message is coming from: is it that Fly is already transforming the image into a VM and won’t do it a second time? (But when I restart the job the first deployment has already been completed.)
Is this still happening? It looks like the deployment is being sent back to the same app as the source app (prereview), which would lead to this error.
Not for any of today’s deployments, but it’s happened sporadically for the last few weeks. I’ve seen it happen to both apps (never both at the same time though).
Am I misusing the Fly registry, or should it be ok to do this?
Had a reoccurrence of this today.
Edit: Make that a few reoccurrences, to both deploys.
More than a few now (I’ve been merging a load of Dependabot PRs today).
I’m going to have to change the workflow so they run in serial. (Edit: Deploy the instances serially · PREreview/prereview.org@0b65a49 · GitHub)
Seeing this as well, deploying 5 apps in parallel on GitHub Actions.
This probably happens because the deployment tags are generated using the current timestamp. This should be fixed in a PR slated for the next release.
1 Like
@jsierles Unfortunately, I’ve just seen a repeat of the issue using the latest flyctl.
Here’s the relevant part of the raw GitHub Actions log:
2022-08-30T13:44:38.8569119Z ##[group]Run superfly/flyctl-actions/setup-flyctl@master
2022-08-30T13:44:38.8569405Z with:
2022-08-30T13:44:38.8569588Z version: latest
2022-08-30T13:44:38.8569790Z ##[endgroup]
2022-08-30T13:44:39.8757079Z [command]/usr/bin/tar xz --warning=no-unknown-keyword --overwrite -C /home/runner/work/_temp/9a7b2a59-f878-4472-9fee-e43bc99dfbd3 -f /home/runner/work/_temp/f85ecaf0-84f3-4402-8e36-435bbc57baeb
2022-08-30T13:44:40.3234998Z flyctl 0.0.383 is installed
2022-08-30T13:44:40.3471932Z ##[group]Run flyctl deploy --config fly.prod.toml --image registry.fly.io/prereview:884e78a6f898b2d4bf2b1b2be9adc4a800329af8
2022-08-30T13:44:40.3473029Z e[36;1mflyctl deploy --config fly.prod.toml --image registry.fly.io/prereview:884e78a6f898b2d4bf2b1b2be9adc4a800329af8e[0m
2022-08-30T13:44:40.3534406Z shell: /usr/bin/bash -e {0}
2022-08-30T13:44:40.3534643Z env:
2022-08-30T13:44:40.3535178Z FLY_API_TOKEN: ***
2022-08-30T13:44:40.3535398Z ##[endgroup]
2022-08-30T13:44:40.5246142Z ==> Verifying app config
2022-08-30T13:44:40.5247000Z e[38;5;252m--> Verified app confige[0m
2022-08-30T13:44:40.5247383Z ==> Building image
2022-08-30T13:44:40.5263583Z Searching for image 'registry.fly.io/prereview:884e78a6f898b2d4bf2b1b2be9adc4a800329af8' locally...
2022-08-30T13:44:40.5297722Z Searching for image 'registry.fly.io/prereview:884e78a6f898b2d4bf2b1b2be9adc4a800329af8' remotely...
2022-08-30T13:44:41.2630289Z Error failed to fetch an image or build from source: Validation failed: Digest has already been taken
It seems unlikely that this is a collision, but you can confirm this by introducing something like sleep $JOB_INDEX to introduce a small delay before the start of each job.
For comparison, logs of the environment that succeeded:
2022-08-30T13:44:40.7849284Z ==> Building image
2022-08-30T13:44:40.7850885Z Searching for image 'registry.fly.io/prereview:884e78a6f898b2d4bf2b1b2be9adc4a800329af8' locally...
2022-08-30T13:44:40.7852131Z Searching for image 'registry.fly.io/prereview:884e78a6f898b2d4bf2b1b2be9adc4a800329af8' remotely...
2022-08-30T13:44:41.2386498Z image found: img_e1zd4mx377yv02yw
Comparing the timestamps, the one that failed did start first…