Elixir SQLite LiteFS dns_cluster - cannot fetch cluster ID

Phoenix
, ,
1

A Phoenix with LiteFS enabled and dns_cluster. The deployment fails with:

cdg [info] level=INFO msg="host environment detected" type=fly.io

cdg [info] level=INFO msg="no backup client configured, skipping"

cdg [info] config file read from /etc/litefs.yml

cdg [info] LiteFS v0.5.8, commit=fd5808d0250cd17ec420225a010ec273971a471b

cdg [info] level=INFO msg="Using Consul to determine primary"

cdg [info] level=INFO msg="initializing consul: key=litefs/distributed-counters url=https://:6cb07397-b934-2bd8-1e53-d07a2b4e2e99@consul-fra-9.fly-shared.net/distributed-counters-ewpm905ok3317zoy/ hostname=3d8d336f002d08 advertise-url=http://3d8d336f002d08.vm.distributed-counters.internal:20202"

cdg [info] level=INFO msg="LiteFS mounted to: mydata"
cdg [info] level=INFO msg="http server listening on: http://localhost:20202"
cdg [info] level=INFO msg="waiting to connect to cluster"

cdg [info] level=INFO msg="cannot become primary, local node has no cluster ID and \"consul\" lease already initialized with cluster ID LFSCDFEFB1D466B64B24"

cdg [info] level=INFO msg="cannot find primary, retrying: no primary"

cdg [info] level=INFO msg="cannot fetch cluster ID from \"consul\" lease, retrying: Get \"http://127.0.0.1:8500/v1/kv/litefs/distributed-counters/clusterid\": dial tcp 127.0.0.1:8500: connect: connection refused"

My config:

fly volumes create mydata

#fly.toml
[env]
DATABASE_PATH = "/data/mydata/my_app_prod.db"
DNS_CLUSTER_QUERY = "my-app.internal"
PHX_HOST = "my_app.fly.dev"
PORT = "8080"
PRIMARY_REGION = "cdg"
RELEASE_COOKIE = "4vB0A5bYymxG_f4P-EOLFRJ7CTpH7GkLsXFwDHRcHSRFLFFxcGh1mg=="

[[mounts]]
source = "mydata"
destination = "/data"

#litefs.yml

fuse:
  dir: "/mydata"

data:
  dir: "/data/mydata"

exit-on-error: false

proxy:
  addr: ":8080"
  target: "localhost:8081"
  db: "my_app_prod.db"
  passthrough: 
    - "*.ico"
    - "*.png"

exec:
  - cmd: "/app/bin/server -addr :8081 -dsn /mydata/db"

lease:
  type: "consul"
  advertise-url: "http://${HOSTNAME}.vm.${FLY_APP_NAME}.internal:20202"
  candidate: ${FLY_REGION == PRIMARY_REGION}
  promote: true

  consul:
    url: "${FLY_CONSUL_URL}"
    key: "litefs/${FLY_APP_NAME}"

Thanks.

2

Hi @ndrean

This may happen if you re-create LiteFS cluster with the same Consul key.

There are some docs describing how to get out of this situation - Disaster Recovery from LiteFS Cloud · Fly Docs

3

Thanks. For some reason, I have type “replica”, no “primary” machine. I need to prune the app

4

[Partly solved]

Works now that you have a “primary” node. Lots of moving parts, so plenty of reasons to fail.

Then for some reason, if I deploy again, I have only replicas, no more primary machines.

A kinda first working solution if someone stumbles on this.

# stop when you have to deploy
fly launch
# remove release_command from fly.toml 
fly deploy
fly volumes create litefs --region cdg --size 1
fly consul attach
fly deploy

#Dockerfile - Debian based.
{RUNNER}
RUN apt-get install ca-certificates fuse3...
COPY --from=flyio/litefs:0.5 /usr/local/bin/litefs /usr/local/bin/litefs
COPY litefs.yml /etc/litefs.yml
COPY --from=builder --chown=nobody:root /app/_build/${MIX_ENV}/rel/liveview_counter ./
# USER nobody
ENV ECTO_IPV6 true
ENV ERL_AFLAGS "-proto_dist inet6_tcp"
ENTRYPOINT litefs mount

#Application.ex
def start(_type, _args) do
    MyApp.Release.migrate()
    children = [ {DNSCluster, query: System.get_env("DNS_CLUSTER_QUERY") || :ignore},...]

#litejs.yml
fuse:
  # This is the mount directory that applications will use to access their SQLite databases.
  dir: "/data/mydata"

data:
  # path to internal data storage
  dir: "/data/litefs"

proxy:
  addr: ":4000"
  target: "localhost:8080"
  #  SQLite database filename
  db: "my_app_prod.db"
  passthrough: 
    - "*.ico"
    - "*.png"

exec:
  - cmd: "/app/bin/server -addr :8080 -dsn /data/mydata" <- ??

lease:
  type: "consul"
  advertise-url: "http://${HOSTNAME}.vm.${FLY_APP_NAME}.internal:20202"
  candidate: ${FLY_REGION == PRIMARY_REGION}
  promote: true

  consul:
    url: "${FLY_CONSUL_URL}"
    key: "litefs/${FLY_APP_NAME}"

#fly.toml
primary_region="cdg"
#[deploy] release_command = "/app/bin/migrate"

[mounts]
source="mydata"
destination="/data"

[env]
PHX_HOST = "my-app.fly.dev"
DNS_CLUSTER_QUERY="my-app.internal"
DATABASE_PATH="/data/mydata/my_app_prod.db"
PORT="8080"
5

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.