Deploy docker error. Client.Timeout exceeded while awaiting headers

I’ve had this app running for more than a year, now it suddenly can’t deploy. With the current FROM nginx:alpine I get no errors but nothing useful happens:

Image size: 23 MB
==> Creating release
Release v412 created

You can detach the terminal anytime without stopping the deployment
Monitoring Deployment
Error 1 error occurred:
* No deployment available to monitor


If I try to change the FROM in the Dockerfile, I get:

➜ flyio git ✗ flyctl deploy --no-cache --remote-only
Deploying --------
==> Validating app configuration
→ Validating app configuration done
Services
TCP 80/443 ⇢ 8080
Remote builder fly-builder-lively-bird-XXXX ready
==> Creating build context
→ Creating build context done
==> Building image with Docker
→ docker host: 20.10.10 linux x86_64
Sending build context to Docker daemon 56.57kB
Step 1/3 : FROM nginx:stable-alpine
Error error building: error rendering build status stream: Get “https://registry-1.docker.io/v2/”: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)


Is there some kind out outtage going on?

That “no deployment available to monitor” error probably means that your Docker image hasn’t changed since the previous deploy. Did you make a code change or something?


We’ve seen a lot of pull failures from Docker Hub lately. The error is saying that the Docker Daemon can’t connect to Docker’s public registry.

There are no outages on our end. This is pretty irritating, though. You may be able to workaround by running Docker locally. These errors seem somewhat localized – one region will see a burst of them, others won’t.

I ended up making a change to my Nginx config just to get the system to make a full deploy, since neither a restart or a deploy without changes did anything.

The Docker thing makes sense. The error read like a connection timeout.

Thanks.

I am again today in the same exact situation.

App is dead, no error messages I can find. Can you help me find an error message or track down what is going on here? This has worked as-is for more than a year before these issues.

I cannot re-deploy it with the same issues. Is there a flyctl option for just force this to deploy and not skip things if it seems to not have any changes?

This is a critical application that has now stopped working 3 days in a row after having no changes made to anything. That would be bad enough if I had to restart it by hand and this actually worked, but having to spend hours to fight through a deployment process that doesn’t work is creating a whole lot of downtime.

I think I misunderstood the problem you were having.

What’s happening when the app stops working? If you run fly status do you see any VMs in a failed state?

The app shows as dead with flyctl status or on the https://fly.io/apps/ listing.

Restarting it does nothing and deploying again has been a headache each time.

This is the only thing that appears like it could be useful from the logs:

2021-12-14T02:41:33.565 runner[3921b308] iad [info] Shutting down virtual machine
2021-12-14T02:41:33.630 app[3921b308] iad [info] Sending signal SIGINT to main child process w/ PID 510
2021-12-14T02:41:34.632 app[3921b308] iad [info] Main child exited normally with code: 0
2021-12-14T02:41:34.632 app[3921b308] iad [info] Starting clean up.
Error 1002: No suitable (healthy) instance found to handle request

Will you check your autoscale settings? Run fly autoscale show and see if the minimum count is set to 0. If it is, set the min to 1 with fly autoscale set min=1 max=10.

Those logs show it exiting normally, not crashing. The restart command restarts VMs in place. Deploys won’t do anything without a code change. If it’s scaled to 0, scaling it up with fly scale count X should make it run again.

That did seem to be set to min 0, and I set the minimum to 1 like you suggested. Hopefully this is the answer.

I reviewed all my other apps, and all seemed to have the 1 minimum setting. Is there any particular reason this would have been set to 0? It seems odd to be for that to suddenly be an issue.

Thanks for the help, I feel like this is probably the answer.