I had to delete the builder 3 times and then eventually one worked. The IAD and DFW builders both kept failing to pull that layer. But a builder in SJC worked.
I wonder if there is any issue with a cache It would make sense for layers to be cached to avoid endlessly pulling the same ones. A quick Google for that “failed size validation” style error results in other people finding the same, seemingly randomly. Those responses suggest it is down to a prior failed pull. So perhaps that then persists. If the cache was per-region (don’t know how the innards work) that would follow, as yes, it would mean a subsequent builder in the same region would fetch the same failed cache … while one in a different region may see an empty cache and so pull it from the source. Total speculation though!
I think @greg’s theory is likely. We have VMs for our registry and our docker hub mirror in a bunch of regions. A bad layer was probably cached in IAD and DFW so moving to another region helped. I just wiped the cache in those two regions and I don’t see any errors so far, but let us know if you still have trouble.