Hi @empz,
We reviewed our internal build machine traces for some of the ones you provided and the slow down can occur when unpacking individual layers from images. What we’ve observed over the years is the time to pull and prepare an image is a combination of two factors: overall image size and total number of layers. From what I could tell, the image you’re using has good number of layers each of which has to be download and unpacked on top of another to produce the final filesystem we turn into a device for firecracker to use.
We continue to look for better ways to make this step faster and as of right now, the only suggestion we can provide is to adjust your image build step to use zstd. We’ve rolled out some changes to further instrument the image pull and prepare process which we hope will reveal more insight into when (and ideally why) some hosts occasionally take longer to pull and unpack layers.