App Stops Again!!

1

So the app is connected, but is failing to connect to DB server. Our DB server is Hosted on Mongo Atlas (AWS) in SIN zone.

All of a sudden, app is failing to connect to DB and the connection is just timing out. We have not deployed anything new. It has just suddenly stopped.

2021-11-12T11:20:46.431 proxy[1560f0e8] maa [warn] Error 2004: App connection idle
2021-11-12T11:20:49.368 proxy[c15c7588] maa [warn] Error 2004: App connection idle
2021-11-12T11:20:51.792 proxy[c15c7588] maa [warn] Error 2004: App connection idle
2021-11-12T11:20:52.766 proxy[c15c7588] maa [warn] Error 2004: App connection idle
2021-11-12T11:20:52.797 proxy[c15c7588] maa [warn] Error 2004: App connection idle
2021-11-12T11:20:53.097 proxy[5cb292b7] maa [warn] Error 2004: App connection idle
2021-11-12T11:20:53.115 proxy[1560f0e8] maa [warn] Error 2004: App connection idle
2021-11-12T11:20:53.237 proxy[1560f0e8] maa [warn] Error 2004: App connection idle
2021-11-12T11:20:53.596 proxy[5cb292b7] maa [warn] Error 2004: App connection idle
2021-11-12T11:20:56.847 proxy[5cb292b7] maa [warn] Error 2004: App connection idle
2021-11-12T11:21:00.022 proxy[1560f0e8] maa [warn] Error 2004: App connection idle
2021-11-12T11:21:00.269 proxy[1560f0e8] maa [warn] Error 2004: App connection idle
2021-11-12T11:21:00.298 proxy[c15c7588] maa [warn] Error 2004: App connection idle
2021-11-12T11:21:00.422 proxy[3e895a3b] maa [warn] Error 2004: App connection idle
2021-11-12T11:21:00.980 proxy[1560f0e8] sin [warn] Error 2004: App connection idle
2021-11-12T11:21:01.452 proxy[5cb292b7] maa [warn] Error 2004: App connection idle
2021-11-12T11:21:02.852 proxy[3e895a3b] sin [warn] Error 2004: App connection idle
2021-11-12T11:21:02.852 proxy[1560f0e8] sin [warn] Error 2004: App connection idle
2021-11-12T11:21:02.945 proxy[3e895a3b] maa [warn] Error 2004: App connection idle
2021-11-12T11:21:03.789 proxy[5cb292b7] maa [warn] Error 2004: App connection idle
2021-11-12T11:21:05.010 proxy[1560f0e8] maa [warn] Error 2004: App connection idle

3

There’s a couple to things you can try to check the problem:

  • do a fly checks list to see the app’s health checks are failing because of the loss of DB connectivity — having this as a health check is generally good but in this case you might want the app running anyway so you can debug and it and serve again when connectivity comes back.
  • fly ssh console into the app to, and do check connectivity to the DB endpoints, using a combination of dig, traceroute to make sure DNS is working correctly and there is a network route available.
  • I’m assuming the DB is up and reachable from other places, but I usually double check that from a couple of other computers.
4

Also, the Error 2004: App connection idle means the connection between the application and the Fly proxy has been closed because there was no reading or writing activity on the connection for over 60 seconds — this marks the connection idle and closes it.

Are there web sockets running or long polling happening on this app? Not sending a ping for a long time might cause the connections to be cut.