What does success look like for LiteFS Cloud?

[taronull]

Problem

I managed deployment with Phoenix + LiteFS. But Cloud seems to be not picking it up?
The app and SQLite is working but Cloud says No databases.
What am I missing here?

Deployment log

2023-12-26T07:40:29.841 runner[d89d970c6e09d8] ams [info] Pulling container image registry.fly.io/fdr:deployment-01HJJH1T979Z6TVBHG03FPHEAY
2023-12-26T07:40:31.575 runner[d89d970c6e09d8] ams [info] Successfully prepared image registry.fly.io/fdr:deployment-01HJJH1T979Z6TVBHG03FPHEAY (1.734544429s)
2023-12-26T07:40:31.600 runner[d89d970c6e09d8] ams [info] Setting up volume 'data'
2023-12-26T07:40:31.600 runner[d89d970c6e09d8] ams [info] Opening encrypted volume
2023-12-26T07:40:32.148 runner[d89d970c6e09d8] ams [info] Configuring firecracker
2023-12-26T07:43:21.654 proxy[d89d970c6e09d8] ams [info] Starting machine
2023-12-26T07:43:21.891 app[d89d970c6e09d8] ams [info] [ 0.053687] Spectre V2 : WARNING: Unprivileged eBPF is enabled with eIBRS on, data leaks possible via Spectre v2 BHB attacks!
2023-12-26T07:43:21.949 app[d89d970c6e09d8] ams [info] [ 0.074574] PCI: Fatal: No config space access function found
2023-12-26T07:43:22.241 app[d89d970c6e09d8] ams [info] INFO Starting init (commit: 15238e9)...
2023-12-26T07:43:22.256 app[d89d970c6e09d8] ams [info] INFO Mounting /dev/vdb at /data w/ uid: 0, gid: 0 and chmod 0755
2023-12-26T07:43:22.258 app[d89d970c6e09d8] ams [info] INFO Resized /data to 3204448256 bytes
2023-12-26T07:43:22.259 app[d89d970c6e09d8] ams [info] INFO Preparing to run: `/bin/sh -c litefs mount` as root
2023-12-26T07:43:22.265 app[d89d970c6e09d8] ams [info] INFO [fly api proxy] listening at /.fly/api
2023-12-26T07:43:22.270 app[d89d970c6e09d8] ams [info] 2023/12/26 07:43:22 listening on [fdaa:0:ebfe:a7b:242:1105:8b2c:2]:22 (DNS: [fdaa::3]:53)
2023-12-26T07:43:22.313 app[d89d970c6e09d8] ams [info] config file read from /etc/litefs.yml
2023-12-26T07:43:22.313 app[d89d970c6e09d8] ams [info] LiteFS v0.5.10, commit=1dc7fcfa7360313fe573c2ff7ad01ef58c05c146
2023-12-26T07:43:22.313 app[d89d970c6e09d8] ams [info] level=INFO msg="host environment detected" type=fly.io
2023-12-26T07:43:22.314 app[d89d970c6e09d8] ams [info] level=INFO msg="litefs cloud backup client configured: https://litefs.fly.io"
2023-12-26T07:43:22.314 app[d89d970c6e09d8] ams [info] level=INFO msg="Using Consul to determine primary"
2023-12-26T07:43:22.400 app[d89d970c6e09d8] ams [info] level=INFO msg="initializing consul: key=litefs/fdr url=https://:533a0abf-a5a2-4652-70c0-921f42d485da@consul-fra-9.fly-shared.net/fdr-jlyv9rlod6y98xrg/ hostname=d89d970c6e09d8 advertise-url=http://d89d970c6e09d8.vm.fdr.internal:20202"
2023-12-26T07:43:22.402 app[d89d970c6e09d8] ams [info] level=INFO msg="using existing cluster id: \"LFSC7A05EE99BC7104E9\""
2023-12-26T07:43:22.407 app[d89d970c6e09d8] ams [info] level=INFO msg="LiteFS mounted to: /litefs"
2023-12-26T07:43:22.407 app[d89d970c6e09d8] ams [info] level=INFO msg="http server listening on: http://localhost:20202"
2023-12-26T07:43:22.407 app[d89d970c6e09d8] ams [info] level=INFO msg="waiting to connect to cluster"
2023-12-26T07:43:22.466 app[d89d970c6e09d8] ams [info] level=INFO msg="52667911B2324512: primary lease acquired, advertising as http://d89d970c6e09d8.vm.fdr.internal:20202"
2023-12-26T07:43:22.475 app[d89d970c6e09d8] ams [info] level=INFO msg="connected to cluster, ready"
2023-12-26T07:43:22.475 app[d89d970c6e09d8] ams [info] level=INFO msg="node is a candidate, automatically promoting to primary"
2023-12-26T07:43:22.476 app[d89d970c6e09d8] ams [info] level=INFO msg="node is already primary, skipping promotion"
2023-12-26T07:43:22.476 app[d89d970c6e09d8] ams [info] level=INFO msg="proxy server listening on: http://localhost:8080"
2023-12-26T07:43:22.476 app[d89d970c6e09d8] ams [info] level=INFO msg="starting background subprocess: /app/bin/server []"
2023-12-26T07:43:22.480 app[d89d970c6e09d8] ams [info] level=INFO msg="begin primary backup stream: url=https://litefs.fly.io"
2023-12-26T07:43:22.480 app[d89d970c6e09d8] ams [info] waiting for signal or subprocess to exit
2023-12-26T07:43:22.545 proxy[d89d970c6e09d8] ams [info] machine started in 890.753634ms
2023-12-26T07:43:23.478 app[d89d970c6e09d8] ams [info] level=INFO msg="begin streaming backup" full-sync-interval=10s
2023-12-26T07:43:25.825 app[d89d970c6e09d8] ams [info] 07:43:25.825 [info] Running FederWeb.Endpoint with Bandit 1.1.2 at :::4000 (http)
2023-12-26T07:43:25.827 app[d89d970c6e09d8] ams [info] 07:43:25.827 [info] Access FederWeb.Endpoint at https://fdr.fly.dev
2023-12-26T07:43:26.268 app[d89d970c6e09d8] ams [info] WARN Reaped child process with pid: 381 and signal: SIGUSR1, core dumped? false
2023-12-26T07:43:26.777 proxy[d89d970c6e09d8] ams [info] machine became reachable in 4.232040573s

Dockerfile

# Start a builder.
FROM elixir:alpine AS builder

# Work in `/app` directory.
WORKDIR /app

# Set environment variable for production.
ENV MIX_ENV="prod"

# Get basic tools.
RUN apk add --no-cache build-base git

# Get package managers.
RUN mix local.hex --force && \
    mix local.rebar --force

# Copy configurations.
RUN mkdir config

COPY config/config.exs config/$MIX_ENV.exs config/

# Get dependencies.
COPY mix.exs mix.lock ./
RUN mix deps.get --only $MIX_ENV

# Compile the dependencies.
RUN mix deps.compile

# Copy the app and digest its assets.
COPY lib lib
COPY priv priv
COPY assets assets
RUN mix assets.deploy

# Compile the app.
RUN mix compile

# Setup runtime.
COPY config/runtime.exs config/
COPY rel rel
RUN mix release

# Start a runner.
FROM elixir:alpine AS runner

# Work in `/app` directory.
WORKDIR /app

# Set environment variable for production.
ENV MIX_ENV="prod"

# Get basic tools.
RUN apk add --no-cache libstdc++ ncurses-libs openssl 

# Copy the app.
COPY --from=builder /app/_build/${MIX_ENV}/rel/feder ./

# Install LiteFS dependencies.
RUN apk add ca-certificates fuse3 sqlite

# Copy in the LiteFS binary and config.
COPY --from=flyio/litefs:0.5 /usr/local/bin/litefs /usr/local/bin/litefs
COPY litefs.yml /etc/litefs.yml

# Run LiteFS. Server gets started by /litefs.yml.
ENTRYPOINT litefs mount

fly.toml

# fly.toml app configuration file generated for fdr on 2023-12-26T13:41:17+09:00
#
# See https://fly.io/docs/reference/configuration/ for information about how to use this file.
#

app = "fdr"
primary_region = "ams"
kill_signal = "SIGTERM"

[build]

[deploy]

[env]
  PHX_HOST = "fdr.fly.dev"
  PORT = "4000"
  DATABASE_PATH = "/data/feder.db"

[mounts]
  source = "data"
  destination = "/data"

[http_service]
  internal_port = 4000
  force_https = true
  auto_stop_machines = true
  auto_start_machines = true
  min_machines_running = 0
  processes = ["app"]
  [http_service.concurrency]
    type = "connections"
    hard_limit = 1000
    soft_limit = 1000

[[services]]
  internal_port = 8080
  protocol = "tcp"

[[vm]]
  cpu_kind = "shared"
  cpus = 1
  memory_mb = 1024

litefs.yml

# The fuse section describes settings for the FUSE file system. This file system
# is used as a thin layer between the SQLite client in your application and the
# storage on disk. It intercepts disk writes to determine transaction boundaries
# so that those transactions can be saved and shipped to replicas.
fuse:
  dir: "/litefs"

# The data section describes settings for the internal LiteFS storage. We'll 
# mount a volume to the data directory so it can be persisted across restarts.
# However, this data should not be accessed directly by the user application.
data:
  dir: "/data"

# This flag ensure that LiteFS continues to run if there is an issue on starup.
# It makes it easy to ssh in and debug any issues you might be having rather
# than continually restarting on initialization failure.
exit-on-error: false

# This section defines settings for the option HTTP proxy.
# This proxy can handle primary forwarding & replica consistency
# for applications that use a single SQLite database.
proxy:
  addr: ":8080"
  target: "localhost:4000"
  db: "feder.db"
  passthrough: 
    - "*.ico"
    - "*.png"

# This section defines a list of commands to run after LiteFS has connected
# and sync'd with the cluster. You can run multiple commands but LiteFS expects
# the last command to be long-running (e.g. an application server). When the
# last command exits, LiteFS is shut down.
exec:
  - cmd: "/app/bin/server"

# The lease section specifies how the cluster will be managed. We're using the
# "consul" lease type so that our application can dynamically change the primary.
#
# These environment variables will be available in your Fly.io application.
lease:
  type: "consul"
  advertise-url: "http://${HOSTNAME}.vm.${FLY_APP_NAME}.internal:20202"
  candidate: ${FLY_REGION == PRIMARY_REGION}
  promote: true

  consul:
    url: "${FLY_CONSUL_URL}"
    key: "litefs/${FLY_APP_NAME}"

[lubien]

First: thanks for the well structured post!

I believe you might been having the same issue I used to have when I first met litefs: storing the DB in the wrong place.

Can you check if your app creates/reads-from a DB under /data/something.db? It should be /litefs/something.db

Edit: found my example nodejs app

[taronull]

Thanks!
OK, there’s a litefs directory with nothing in it.
What would that mean?

d89d970c6e09d8:/# pwd
/
d89d970c6e09d8:/# ls
app     data    etc     home    litefs  mnt     proc    run     srv     tmp     var
bin     dev     help    lib     media   opt     root    sbin    sys     usr
d89d970c6e09d8:/# ls litefs
d89d970c6e09d8:/# ls data
clusterid     dbs           feder.db      feder.db-shm  feder.db-wal  lost+found
d89d970c6e09d8:/#

[lubien]

Its okay for it to be empty but youd need to move your DB there. Quoting litefs.yml:

This file system is used as a thin layer between the SQLite client in your application and the
storage on disk. It intercepts disk writes to determine transaction boundaries
so that those transactions can be saved and shipped to replicas.

Basically, in the south park-esque kind of explanation:

1. Put feder.db* files on /litefs and read/write from it
2. ??? ← [at]benbjohnson and litefs team magic
3. Profit

[taronull]

It works!

My mistake was that I made a persistent volume for db file from sqlite.
I did that because I used to do that.
Now I see I need to make LiteFS persistent.

from
  app -> data(persistent volume)
to
  app -> data -> LiteFS(persistent volume)

Changes

# fly.toml

[mounts]
- source = "data"
- destination = "/data"
+ source = "litefs"
+ destination = "/var/lib/litefs"

litefs.yml was little confusing too.
So, fuse.dir is where SQLite stores .db files.
data.dir is where LiteFS lives and should be a persistent volume.

# litefs.yml

fuse:
- dir: "/litefs"
+ dir: "/data"

data:
- dir: "/data"
+ dir: "/var/lib/litefs"

I’m getting logs from failing consul lease

2023-12-27T15:16:00.123 app[e28673df76e348] ams [info] level=INFO msg="cannot fetch cluster ID from \"consul\" lease, retrying: Get \"http://127.0.0.1:8500/v1/kv/litefs/fdr/clusterid\": dial tcp 127.0.0.1:8500: connect: connection refused"
2023-12-27T15:16:01.123 app[e28673df76e348] ams [info] level=INFO msg="cannot fetch cluster ID from \"consul\" lease, retrying: Get \"http://127.0.0.1:8500/v1/kv/litefs/fdr/clusterid\": dial tcp 127.0.0.1:8500: connect: connection refused"
2023-12-27T15:16:02.123 app[e28673df76e348] ams [info] level=INFO msg="cannot fetch cluster ID from \"consul\" lease, retrying: Get \"http://127.0.0.1:8500/v1/kv/litefs/fdr/clusterid\": dial tcp 127.0.0.1:8500: connect: connection refused"

But everything seems to be working so I’m happy

[lubien]

Im glad you got it running! Go LiteFS