Lots of connects/disconnects to other nodes with Phoenix running with libcluster

I am currently playing around with Phoenix and clustering.

It seems to work, but when watching the logs I see a lot of connects/disconnects:

2021-04-24T22:33:46.384Z 7a73f585 ord [info] [info] [libcluster:fly6pn] disconnected from :"phoenix-liveview-chat@xxxxxxxx:23bb:fa38:2"
2021-04-24T22:33:46.387Z 7a73f585 ord [info] [info] [libcluster:fly6pn] disconnected from :"phoenix-liveview-chat@xxxxxxxx:ead:e0bc:21a5:2"
2021-04-24T22:33:49.623Z 23bbfa38 fra [info] [info] [libcluster:fly6pn] disconnected from :"phoenix-liveview-chat@xxxxxxxx:ead:e0bc:21a5:2"
2021-04-24T22:33:49.625Z 23bbfa38 fra [info] [warn] [libcluster:fly6pn] unable to connect to :"phoenix-liveview-chat@xxxxxxxx:cad1:20dc:2"
2021-04-24T22:33:53.393Z 7a73f585 ord [info] [warn] [libcluster:fly6pn] unable to connect to :"phoenix-liveview-chat@xxxxxxxx:cad1:20dc:2"
2021-04-24T22:33:54.713Z 23bbfa38 fra [info] [warn] [libcluster:fly6pn] unable to connect to :"phoenix-liveview-chat@xxxxxxxx:cad1:20dc:2"
2021-04-24T22:33:59.720Z 23bbfa38 fra [info] [warn] [libcluster:fly6pn] unable to connect to :"phoenix-liveview-chat@xxxxxxxx:cad1:20dc:2"
2021-04-24T22:34:05.311Z 23bbfa38 fra [info] [info] [libcluster:fly6pn] connected to :"phoenix-liveview-chat@xxxxxxxx:81:7a73:f585:2"
2021-04-24T22:34:05.403Z 7a73f585 ord [info] [warn] [libcluster:fly6pn] unable to connect to :"phoenix-liveview-chat@xxxxxxxx:cad1:20dc:2"
2021-04-24T22:34:06.233Z 23bbfa38 fra [info] [info] [libcluster:fly6pn] connected to :"phoenix-liveview-chat@xxxxxxxx:ead:e0bc:21a5:2"
2021-04-24T22:34:10.721Z e0bc21a5 sin [info] [info] [libcluster:fly6pn] disconnected from :"phoenix-liveview-chat@xxxxxxxx:23bb:fa38:2"
2021-04-24T22:34:11.241Z 23bbfa38 fra [info] [info] [libcluster:fly6pn] disconnected from :"phoenix-liveview-chat@xxxxxxxx:81:7a73:f585:2"
2021-04-24T22:34:11.243Z 23bbfa38 fra [info] [info] [libcluster:fly6pn] disconnected from :"phoenix-liveview-chat@xxxxxxxx:ead:e0bc:21a5:2"
2021-04-24T22:34:11.246Z 23bbfa38 fra [info] [warn] [libcluster:fly6pn] unable to connect to :"phoenix-liveview-chat@xxxxxxxx:cad1:20dc:2"
2021-04-24T22:34:16.835Z 23bbfa38 fra [info] [info] [libcluster:fly6pn] connected to :"phoenix-liveview-chat@xxxxxxxx:81:7a73:f585:2"
2021-04-24T22:34:17.417Z 7a73f585 ord [info] [warn] [libcluster:fly6pn] unable to connect to :"phoenix-liveview-chat@xxxxxxxx:cad1:20dc:2"
2021-04-24T22:34:17.723Z e0bc21a5 sin [info] [warn] [libcluster:fly6pn] unable to connect to :"phoenix-liveview-chat@xxxxxxxx:cad1:20dc:2"
2021-04-24T22:34:17.757Z 23bbfa38 fra [info] [info] [libcluster:fly6pn] connected to :"phoenix-liveview-chat@xxxxxxxx:ead:e0bc:21a5:2"
2021-04-24T22:34:22.764Z 23bbfa38 fra [info] [info] [libcluster:fly6pn] disconnected from :"phoenix-liveview-chat@xxxxxxxx:81:7a73:f585:2"
2021-04-24T22:34:22.765Z 23bbfa38 fra [info] [info] [libcluster:fly6pn] disconnected from :"phoenix-liveview-chat@xxxxxxxx:ead:e0bc:21a5:2"
2021-04-24T22:34:22.768Z 23bbfa38 fra [info] [warn] [libcluster:fly6pn] unable to connect to :"phoenix-liveview-chat@xxxxxxxx:cad1:20dc:2"
2021-04-24T22:34:27.434Z 7a73f585 ord [info] [info] [libcluster:fly6pn] disconnected from :"phoenix-liveview-chat@xxxxxxxx:ead:e0bc:21a5:2"
2021-04-24T22:34:27.774Z 23bbfa38 fra [info] [warn] [libcluster:fly6pn] unable to connect to :"phoenix-liveview-chat@xxxxxxxx:cad1:20dc:2"
2021-04-24T22:34:32.783Z 23bbfa38 fra [info] [warn] [libcluster:fly6pn] unable to connect to :"phoenix-liveview-chat@xxxxxxxx:cad1:20dc:2"
2021-04-24T22:34:34.437Z 7a73f585 ord [info] [warn] [libcluster:fly6pn] unable to connect to :"phoenix-liveview-chat@xxxxxxxx:cad1:20dc:2"

Is there something I can do about it?

I basically copied most configuration from your GitHub Repo: GitHub - fly-apps/phoenix-liveview-cluster: LiveView in a global cluster.

There are two things that typically cause disconnects:

  1. Deploys get new IP addresses, and during the rolling deploy process they’ll connect to VMs that are about to vanish.
  2. The erlang cookie might be changing between deploys. I had to set a RELEASE_COOKIE secret in the liveview-counter to make it stick.

I now set the RELEASE_COOKIE as an environment variable and waited a few minutes to allow it to create the micro vms.

But I still see a lot of connected to followed by a disconnected from after a few seconds and then followed by an unable to connect to message. Then the cycle starts again :sweat_smile:

The app is still working, but usually, this should not happen.

With flyctl ips private I can see that the current IPs from the vms and can confirm those are being used to connect and disconnect.

Well that’s suspicious. Can you tell if it’s typically one region involved in the errors? Those logs make it look like singapore.

It was a problem in all regions, but I am afraid I already deleted the app and create a new one.

Seems to work fine now with the new app! :tada: No unexpected disconnects logged so far :+1: