“You are doing it wrong”, I would say. Even though image size has been increased from 2G->8G (ref Docker image size limit raised from 2GB to 8GB)
Do not load models on build stage, download those to persistent volume (as models probably do not change often) and use those from there.