Deployment stuck at "Pulling container image"

1 
2022-11-08T12:54:29Z runner[1d759bc8] sin [info]Starting instance
2022-11-08T12:54:30Z runner[1d759bc8] sin [info]Configuring virtual machine
2022-11-08T12:54:30Z runner[1d759bc8] sin [info]Pulling container image

Not sure why, but I started having problems deploying to this specific app starting around one and a half days ago. I have a few apps running (in the same region in Singapore), but only this one has been problematic. In the past 24 hours, I’ve tried deploying it many times, but it’s only succeeded getting past Pulling container image 2-3 times out of 20+ times I’ve tried. I’ve tried deleting the app and recreating it, still happening. Any ideas? I’ve also tried building it locally with --local-only, doesn’t work either.

If it helps, the logs from the builder are constantly printing:

2022-11-08T12:59:24.117 app[0e2861d5c73586] sin [info] time="2022-11-08T12:59:24.117088039Z" level=debug msg="checking docker activity"

2022-11-08T12:59:24.118 app[0e2861d5c73586] sin [info] time="2022-11-08T12:59:24.117465100Z" level=debug msg="Calling GET /v1.41/containers/json?filters=%7B%22status%22%3A%7B%22running%22%3Atrue%7D%7D&limit=0"

over and over.

2

If it helps, I tried creating a new app in San Jose and building it there instead – that worked. Not sure what’s up with the Singapore region, but heads up if it helps anyone at all. Lost a few hairs of my head trying to wrangle with this issue – with this and the past few outages at this point I’m seriously considering moving back to Heroku or something else. I was hoping deploying on Fly would have helped me worry less about devops while iterating quickly on a startup, but that dream is growing further and further away.

4

Hi Siaw, this might be something with our registry. Will you share some more information about which app you are deploying?

Also, try running the deployment with LOG_LEVEL=debug and let’s see if that gives us some additional info on what’s failing.

5

If someone happens to find this I found that deleting the fly-builder app and redeploying worked for me.

6

If someone happens to find this I found that deleting the fly-builder app and redeploying worked for me.

Thank you for this.