[URGENT] Contact Lost with instance

Questions / Help
1

Hello! My app experiencing downtime from around 30 minutes, I deployed it month ago and it was running smooth till now, but suddenly it stopped, and I got the error code:

runner[4550c243] fra [info]Shutting down virtual machine
app[4550c243] fra [info]Sending signal SIGINT to main child process w/ PID 521

I did not make any changes in my app and to troubleshoot I am trying to restart it several times and also tried redeploying it using deploy command. But logs are paused (not getting updated) and while deploying it giving me error code:

WARN Remote builder did not start in time. Check remote builder logs with `flyctl logs -a fly-builder-wispy-dust-2487
Error failed to fetch an image or build from source: error connecting to docker: remote builder app unavailable

While restarting it:

WARN failed querying for new release: Get "https://api.fly.io/app/flyctl_releases/windows/amd64/latest": dial tcp [2a09:8280:1:f28:246e:d6a:949:dbbf]:443: i/o timeout
2

May be the free Fly builder is gone. Either you can try to recreate the builder, or deploy with fly deploy --local-only <other-args> to see if it then recovers your app.

This is worrying, because ideally, Fly should have restarted the VM unless something scaled it down to zero, or got rid of it. If you’re on the launch plan, you should email support email specific to your org: I can’t deploy the application!!! - #3 by undefined

3

Thanks! I tried recreating the builder, but it still got paused in:

WARN Remote builder did not start in time. Check remote builder logs with `flyctl logs -a fly-builder-small-frost-9135`
Error failed to fetch an image or build from source: error connecting to docker: remote builder app unavailable
4

It sounds like Builder having serious problem:

2022-10-09T05:01:54Z app[9080173a767787] ams [info]time="2022-10-09T05:01:54.222976394Z" level=info msg="shutting down"
2022-10-09T05:01:54Z app[9080173a767787] ams [info]time="2022-10-09T05:01:54.225280605Z" level=info msg="gracefully stopped\n"
2022-10-09T05:01:54Z app[9080173a767787] ams [info]time="2022-10-09T05:01:54.225313226Z" level=info msg="disk space used: 0.11%"
2022-10-09T05:01:54Z app[9080173a767787] ams [info]time="2022-10-09T05:01:54.225328625Z" level=info msg="Waiting for dockerd to exit"
2022-10-09T05:01:54Z app[9080173a767787] ams [info]time="2022-10-09T05:01:54.225508482Z" level=info msg="Processing signal 'interrupt'"
2022-10-09T05:01:54Z app[9080173a767787] ams [info]time="2022-10-09T05:01:54.225830676Z" level=debug msg="daemon configured with a 15 seconds minimum shutdown timeout"
2022-10-09T05:01:54Z app[9080173a767787] ams [info]time="2022-10-09T05:01:54.225872034Z" level=debug msg="start clean shutdown of all containers with a 15 seconds timeout..."
2022-10-09T05:01:54Z app[9080173a767787] ams [info]time="2022-10-09T05:01:54.225978714Z" level=debug msg="found 0 orphan layers"
2022-10-09T05:01:54Z app[9080173a767787] ams [info]time="2022-10-09T05:01:54.226327748Z" level=debug msg="Unix socket /var/run/docker/libnetwork/c13a08614dc3.sock doesn't exist. cannot accept client connections"
2022-10-09T05:01:54Z app[9080173a767787] ams [info]time="2022-10-09T05:01:54.226389434Z" level=debug msg="Cleaning up old mountid : start."
2022-10-09T05:01:54Z app[9080173a767787] ams [info]time="2022-10-09T05:01:54.226532793Z" level=info msg="stopping event stream following graceful shutdown" error="<nil>" module=libcontainerd namespace=moby
2022-10-09T05:01:54Z app[9080173a767787] ams [info]time="2022-10-09T05:01:54.226690128Z" level=debug msg="Cleaning up old mountid : done."
2022-10-09T05:01:54Z app[9080173a767787] ams [info]time="2022-10-09T05:01:54.226774536Z" level=debug msg="unmounting daemon root" mountpoint=/data/docker
2022-10-09T05:01:54Z app[9080173a767787] ams [info]time="2022-10-09T05:01:54.226888149Z" level=debug msg="Clean shutdown succeeded"
2022-10-09T05:01:54Z app[9080173a767787] ams [info]time="2022-10-09T05:01:54.226905131Z" level=info msg="Daemon shutdown complete"
2022-10-09T05:01:54Z app[9080173a767787] ams [info]time="2022-10-09T05:01:54.226940287Z" level=info msg="stopping healthcheck following graceful shutdown" module=libcontainerd
2022-10-09T05:01:54Z app[9080173a767787] ams [info]time="2022-10-09T05:01:54.226945847Z" level=info msg="stopping event stream following graceful shutdown" error="context canceled" module=libcontainerd namespace=plugins.moby
2022-10-09T05:01:54Z app[9080173a767787] ams [info]time="2022-10-09T05:01:54.227122068Z" level=debug msg="received signal" signal=terminated
2022-10-09T05:01:54Z app[9080173a767787] ams [info]time="2022-10-09T05:01:54.227228507Z" level=debug msg="sd notification" error="<nil>" notified=false state="STOPPING=1"
2022-10-09T05:01:54Z app[9080173a767787] ams [info]time="2022-10-09T05:01:54.956733172Z" level=debug msg="checking docker activity"
2022-10-09T05:01:54Z app[9080173a767787] ams [info]time="2022-10-09T05:01:54.957092095Z" level=debug msg="Calling GET /v1.41/containers/json?filters=%7B%22status%22%3A%7B%22running%22%3Atrue%7D%7D&limit=0"
2022-10-09T05:01:55Z app[9080173a767787] ams [info]time="2022-10-09T05:01:55.227789004Z" level=warning msg="grpc: addrConn.createTransport failed to connect to {unix:///var/run/docker/containerd/containerd.sock  <nil> 0 <nil>}. Err :connection error: desc = \"transport: Error while dialing dial unix:///var/run/docker/containerd/containerd.sock: timeout\". Reconnecting..." module=grpc
2022-10-09T05:01:55Z app[9080173a767787] ams [info]time="2022-10-09T05:01:55.229980644Z" level=info msg="dockerd has exited"
2022-10-09T05:01:57Z runner[9080173a767787] ams [info]machine exited with exit code 0, not restarting
5

It looks like one host in the fra region reached a capacity limit and moved a couple apps to another host. This is usually quick enough, but maybe yours hit an issue on the restart. Can you share the name of the app having issues so we can look into it further?

7

App Name: drfilestreambot
URL: drfilestreambot.fly.io

8

Could you run fly scale show on the app and make sure count is greater than zero? (fly scale count 1 will set a single instance.).

9

@wjordan Thanks! now it is working as expected after restart. :heart:

1 Like
10

@wjordan Builders are still not working, can you look into it?

WARN Remote builder did not start in time. Check remote builder logs with `flyctl logs -a fly-builder-muddy-waterfall-1617`
Error failed to fetch an image or build from source: error connecting to docker: remote builder app unavailable
11

I checked it just now (Builder)! the output was same. It is not working still.

12

I am getting the same issue, check in [URGENT] Contact Lost with instance - Questions / Help - Fly.io

13

I’ve noticed one of our apps was scaled to zero. Scale the app up and it will work again.

1 Like