Clustering incoming and out going deployments in one cluster

Phoenix
,
1

Hi!

I am trying out clustering with my phoenix app to get a synchronized in-memory store (using Groot). This was fairly easy to set up using dns_cluster following the Clustering Your Application guide. :smiley:

The project is (yet another) secrets sharing app: GitHub - ringvold/ots: 🔐 Share end-to-end encrypted secrets with others via a one-time URL. For anyone interessted in the actual code the syncing with goot version is in the full-phoenix branch.

I would like to enroll the new deployments into the cluster and keep both the new and the old for a little time so the new nodes can get the data in Groot before the old nodes are removed. I have not gotten this to work. It seems like the new and old nodes are clustered separately. :thinking:

Is this related to how the internal dns (<appname>.internal) works in Fly? Is what I’m trying to do possible? Should it not be done for some reason?

My use-case could be solved by a database but it would be really nice not require a database and now I am really curious about the clustering possibilities with Elixir/BEAM. :smile:

2

fly launch adds the following to your rel/env.sh.eex:

# configure node for distributed erlang with IPV6 support
export ERL_AFLAGS="-proto_dist inet6_tcp"
export ECTO_IPV6="true"
export DNS_CLUSTER_QUERY="${FLY_APP_NAME}.internal"
export RELEASE_DISTRIBUTION="name"
export RELEASE_NODE="${FLY_APP_NAME}-${FLY_IMAGE_REF##*-}@${FLY_PRIVATE_IP}"

The RELEASE_NODE intentionally derives its name from the FLY_IMAGE_REF to uniquely cluster per deploy, so you don’t have old code connecting to new code. If you expressly want the clustering, you can change it to RELEASE_NODE="${FLY_APP_NAME}@${FLY_PRIVATE_IP}", which will allow the nodes to cluster regardless of docker image. Note that you would need to allow your state handshake process whatever time it needs to complete, otherwise fly is going to teardown the existing machines as soon as it sees the new deployment is healthy. You could do this by having something in the app startup tree that does the handshake in a blocking way before you start the endpoint and pass health checks, but you’ll necessarily need something like that.

1 Like