LiteFS replicas do not replicate the primary

Questions / Help
1

Hello,

My machines are

Machines
PROCESS	ID            	VERSION	REGION	STATE  	ROLE   	CHECKS	LAST UPDATED
app    	3287ee30bdd4d8	29     	arn   	stopped	primary	     	2025-11-22T22:00:27Z	
app    	3d8d1170a4d458	29     	arn   	stopped	replica	     	2025-11-22T22:00:37Z	
app    	e2863e26a33238	29     	arn   	stopped	replica	     	2025-11-22T22:07:44Z

If I start all machines, to see the database checksums, I get:

[touny@touny-lenovo snippets]$ fly ssh console -s -C "cat /litefs/db.sqlite-pos"
? Select VM: arn: 3287ee30bdd4d8 fdaa:d:7960:a7b:5e2:bf73:194e:2 shy-star-7613
Connecting to fdaa:d:7960:a7b:5e2:bf73:194e:2... complete
0000000000000002/a3a1744061b6cd3b

[touny@touny-lenovo snippets]$ fly ssh console -s -C "cat /litefs/db.sqlite-pos"
? Select VM: arn: 3d8d1170a4d458 fdaa:d:7960:a7b:5d6:24f0:2b5:2 aged-violet-322
Connecting to fdaa:d:7960:a7b:5d6:24f0:2b5:2... complete
cat: can't open '/litefs/db.sqlite-pos': No such file or directory
Error: ssh shell: Process exited with status 1

[touny@touny-lenovo snippets]$ fly ssh console -s -C "cat /litefs/db.sqlite-pos"
? Select VM: arn: e2863e26a33238 fdaa:d:7960:a7b:5bc:f663:c6d8:2 solitary-sky-4523
Connecting to fdaa:d:7960:a7b:5bc:f663:c6d8:2... complete
cat: can't open '/litefs/db.sqlite-pos': No such file or directory
Error: ssh shell: Process exited with status 1

I am using static leasing. my litefs.yml is

litefs.yml 
# Database
# # # # #

# directory where the application accesses the database
fuse:
  dir: "/litefs"

# a directory for LiteFS internal data
# must be mounted to a persistent volume
data:
  dir: "/var/lib/litefs"


# Lease protocol
# # # # #

lease:
  # Consul protocol: https://consul.io/
  type: "static"

  # nodes which could possibly become primary, are nodes in the primary region
  candidate: ${FLY_ALLOC_ID == "3287ee30bdd4d8"}

  # API URL other nodes will use to connect to this node
  advertise-url: "http://${FLY_ALLOC_ID}.vm.${FLY_APP_NAME}.internal:20202"

  consul:
    # URL of the Consul cluster
    url: "${FLY_CONSUL_URL}"

    # A unique key shared by all nodes
    key: "${FLY_APP_NAME}/primary"


# Proxy — redirecting requests
# # # # #

proxy:
  # address for the proxy to listen on
  # matches fly
  # if building a local docker container, matches it
  addr: ":8080"

  # port the application is listening on
  # matches the application itself
  target: "localhost:8081"

  # file name of SQLite database
  db: "db.sqlite"


# Application Server
# # # # #

exec:
  # migrations on candidate nodes
  # not sure why valid
  - cmd: "${SCH_PATH}/migrate.sh"
    if-candidate: true

  # web app
  - cmd: "/bin/sh -c /bin/app"

If I stop the primary machine and start a replica machine, a request to my app SEEMS to be redirected to the primary, as I notice the primary is immediately started after the request.

Thank you in advance.

2

Hi, again… In this one, you’re mixing static leasing with Consul-based leasing. For static leasing, advertise-url should always be the address of the primary node.

It’s generally a good idea to start the logs running (fly logs) in the background in a separate terminal before doing experiments, since LiteFS is fairly verbose and can give a lot of useful information about what is is trying to do at any given point in time.

This didn’t happen in my own recent experiment. Instead, the primary stayed off after the request, while the replica continued to complain in the logs about not being able to reach it.

The sequence that you saw was probably just a coincidence, in light of that. My guess is that the replica was reporting itself unhealthy, due to not being able to contact the primary, and the Fly Proxy eventually tried starting a different Machine.

Like we were discussing earlier, scaling all three to 0 doesn’t seem like a viable setup overall to me…

Since the replicas are not actually synching while they’re asleep, you’d have the worst of both worlds: the same risk of permanent data loss as a single-volume app, :dragon:, but then also these extra delays and 502 glitches from the Fly Proxy bouncing around, at random, trying to find a Machine that will reply.

3

I am grateful for your persistent kindful support.

You are right. I fixed my litefs.yml file following your comment:

litefs.yml 
# Database
# # # # #

# directory where the application accesses the database
fuse:
  dir: "/litefs"

# a directory for LiteFS internal data
# must be mounted to a persistent volume
data:
  dir: "/var/lib/litefs"


# Lease protocol
# # # # #

lease:
  # Consul protocol: https://consul.io/
  type: "static"

  # nodes which could possibly become primary, are nodes in the primary region
  candidate: ${FLY_ALLOC_ID == "e2863e26a33238"}

  # API URL other nodes will use to connect to this node
  advertise-url: "http://e2863e26a33238.vm.${FLY_APP_NAME}.internal:20202"


# Proxy — redirecting requests
# # # # #

proxy:
  # address for the proxy to listen on
  # matches fly
  # if building a local docker container, matches it
  addr: ":8080"

  # port the application is listening on
  # matches the application itself
  target: "localhost:8081"

  # file name of SQLite database
  db: "db.sqlite"


# Application Server
# # # # #

exec:
  # migrations on candidate nodes
  # not sure why valid
  - cmd: "${SCH_PATH}/migrate.sh"
    if-candidate: true

  # web app
  - cmd: "/bin/sh -c /bin/app"

If all machines start, replication succeeds

[touny@touny-lenovo snippets]$ fly ssh console -s -C "cat /litefs/db.sqlite-pos"
? Select VM: arn: 3d8d1170a4d458 fdaa:d:7960:a7b:5d6:24f0:2b5:2 aged-violet-322
Connecting to fdaa:d:7960:a7b:5d6:24f0:2b5:2... complete
0000000000000002/a3a1744061b6cd3b

[touny@touny-lenovo snippets]$ fly ssh console -s -C "cat /litefs/db.sqlite-pos"
? Select VM: arn: 3d8d1170a4d458 fdaa:d:7960:a7b:5d6:24f0:2b5:2 aged-violet-322
Connecting to fdaa:d:7960:a7b:5d6:24f0:2b5:2... complete
0000000000000002/a3a1744061b6cd3b

[touny@touny-lenovo snippets]$ fly ssh console -s -C "cat /litefs/db.sqlite-pos"
? Select VM: arn: 3287ee30bdd4d8 fdaa:d:7960:a7b:5e2:bf73:194e:2 shy-star-7613
Connecting to fdaa:d:7960:a7b:5e2:bf73:194e:2... complete
0000000000000002/a3a1744061b6cd3b

If the primary is stopped, a replica starting does not lead to starting the primary, and the following error shows in logs:

2025-11-23T06:58:04Z app[3287ee50ad0438] arn [info]level=INFO msg="AA7F0E23241C980D: existing primary found (), connecting as replica to \"http://3287ee30bdd4d8.vm.snippet-damp-sea-6799.internal:20202\""
2025-11-23T06:58:04Z app[3287ee50ad0438] arn [info]level=INFO msg="AA7F0E23241C980D: disconnected from primary with error, retrying: connect to primary: Post \"http://3287ee30bdd4d8.vm.snippet-damp-sea-6799.internal:20202/stream\": dial tcp: lookup 3287ee30bdd4d8.vm.snippet-damp-sea-6799.internal on [fdaa::3]:53: no such host ('http://3287ee30bdd4d8.vm.snippet-damp-sea-6799.internal:20202')"

The error above is consistent with your guess.

You are correct. Even in static leasing, a running replica is supposed to find a running primary machine.

My app is currently in very early stage. It may be used like for 2 minutes every 3 days. a single volume app with manual back-up is not bad.

However, I still see potential in litefs with scaling to 0.

Static Leasing

  • Every x minutes, replica machines, start up the primary, and sync their local databases.
  • If a replica received a POST request, it starts up the primary machine, then forwards the request.

Consul Leasing

  • If all machines are stopped, the primary machine, which is the last running machine, is given priority to handle the request.
  • For fault-tolerance, if all machines are stopped, a single request starts-up two machines.

I am happy to learn fly-replay and invest time to implement these two mechanisms. @mayailurus, do you have any recommendations?

1 Like
4

This is a worthy goal! The forum has a lot of people who are interested in Fly-Replay, in general, so I’d imagine there’d be a fair amount of help, suggestions, opinions, etc., on hand…

In my own view, the document from the earlier thread makes a good introduction to the conceptually safest approach: a dedicated router app. In this case, it would be written to ensure that the primary is always started first and stopped last (along with any additional invariants).

Static leasing seems the easiest to begin with, since that would provide more safeguards against accidentally waking things in the wrong order—and the like. [However, it would inherently have worse uptime, since the underlying physical host machine that the primary is on will fail someday, and then you would need to manually construct a replacement.]

This way, you externally create a sufficient simulation of the situation that static-leasing LiteFS is internally designed for: an always-on primary with occasionally-on replicas. (In reality, the primary is off for most of the week, but that happens only when the replicas are shutdown, too. None of those Machines can tell the difference!)

The idea of having the replicas themselves wake up the primary is intriguing but would require a great deal more care, I’d say. At that point, you would be at the threshold of designing your own distributed database, which is challenging.

You might also find the VFS variant of LiteStream that was announced a couple months back interesting. That’s still under construction, last I heard, and I don’t know whether that one would scale completely to zero, either, but it might be worth looking into.

Anyway, others here may well have their own opinions on the various tradeoffs and difficulty estimates…

1 Like
5

I am grateful for your generous time and detailed responses. For sure, I’ll follow your guidelines and advises.
If you clicked on my picture, visiting my profile, you’ll see my website, which contains private messaging channels. Feel free to privately send me.

6

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.