Hi there, I’m not super familiar with dev-ops stuff, but I wanted to use LiteFS, could someone give some examples of how you would go about deploying with LiteFS + Safety precautions you would want to setup in a production environment to have safe migrations/deployments with it? there’s so many different pieces I’ve been reading about consul, fly, fuse, etc and none of it I am familiar with, so any explainations or resources that I could read would be super helpful.
Thank you!
If you’re not super comfortable with devops and database administration, you might want to wait for managed LiteFS which (my guess is) should be out any month now.
could someone give some examples of how you would go about deploying with LiteFS
Here’s the official LiteFS guide: LiteFS - Distributed SQLite · Fly Docs
If you follow the docs and have specific questions not yet answered on these forums, feel free to ask.
consul, fly, fuse, etc and none of it I am familiar with, so any explainations or resources that I could read
A series of blogposts about sqlite and LiteFS by its lead developer might be what you’re looking for: fly.io/blog, mirror.
Safety precautions you would want to setup in a production environment
Per docs, LiteFS is beta software and not recommended by its developer for production use.
re beta software: on his twitter he mentioned that the only reason its marked as not safe for production use is due to not having a simple way to do backups, I guess what I’m wondering is there must be a way to do backups, but how would you go about it in its current state? Kent C Dodds is already recommending it in GitHub - epicweb-dev/epic-stack: This is a Remix Stack with the foundational things setup and configured for you to hit the ground running on your next EPIC idea. but there is still very little documentation (at least from what I have found) around the typical lifecycles you would want to care about when it comes to managing LiteFS.
I’m not interested in a managed solution as I’d like to learn and get my hands dirty with understanding how to do this kind of stuff, so again, any resources would be greatly appreciated! 
You can do a simple cron-based backup by using the litefs export command and then you can upload that SQLite file to S3. The tricky part is that you may have multiple nodes that could be candidates to be primary.
You have a few options:
Upload a backup from each one. S3 backups are pretty cheap and typically people run two candidate nodes.
Upload backups from only one node. Even if it’s not the primary, it’s probably still very up-to-date.
If you need to recover, you can use the litefs import command.
We have a streaming backup service coming soon (similar to Litestream but built for LiteFS). It’ll handle things like switching primaries automatically and it’ll provide point-in-time recovery so you can revert back to an old state, if needed.
We are ramping up our LiteFS docs so please let me know if there’s any specific information you want to see covered!
First of all, thank you for the detailed response!
Re docs: would love to see some general guidelines of how to do things. E.g. how would a typical production setup look like? What should I care about? Why do I care about X, what is consol, how is it used? why do I care?
a guide to all of these new tools would be super helpful, I’m not stupid, just uneducated about the tools, so if you can have guides that informs me about them and allows me to unblock myself by sharing the information I’m lacking I think that would be a really great doc experience
That’s great feedback. Thank you!
I’ll get some docs updated with that info but I’ll try to give a quick explanation to your questions here. Let me know if my answers spur more questions and we can update the docs to reflect those too.
Our recommendation looks like this:
2 candidate nodes in your primary region. This lets you handoff “primary” status between these nodes during a deploy or if one goes down. It also keeps your primary in the same physical location so your write latency shouldn’t change just because your primary does.
Use volumes with each of your instances. I’d recommend having volumes at least 2x the size of your database since LiteFS stores transaction data temporarily so it can replicate it. Volumes also make it so your instance is ready faster on startup and doesn’t need to copy a new snapshot from your primary node.
Add nodes in additional regions to reduce latency for read requests. You probably don’t need instances in every region. For example, if you only had instances in the US and you added a replica in Asia then you’d automatically cut your round trip time for read requests for Asia users by ~250ms.
Right now, LiteFS is optimized for read-heavy workloads so if you have a lot of writes then it might be worth benchmarking that. Most of that overhead for writes comes from FUSE. We’ll be releasing a SQLite VFS implementation in the near future which will avoid that FUSE overhead entirely.
LiteFS uses asynchronous replication too so there’s a small window where SQLite can confirm a write and then you suddenly lose that primary before it’s able to replicate that change to the other candidate. Replication is really fast so that window is typically only a few milliseconds (or less) with two candidates in the same region.
Consul provides a distributed lease for LiteFS. That’s what ensures that at any given time there’s only one primary node. It also allows us to change the primary node safely and automatically when the current primary goes down.
The alternative is to use a "static" lease type in LiteFS. This is where you only have a single instance that is ever the primary. In this case, if it goes down then you’ll lose write availability until you reconfigure your cluster to point to a new primary node.
Thank you so much Ben!
in order to prep my project for enabling things like backups, is there something I can do in the interm now and be prepared for the solution you guys are working to release for a easy migration? or is that still TBD?
also, I plan on building a store so the only writes would be adding products and orders, would liteFS be sutable for a use case like this?
It should just be a few lines of config in your litefs.yml file. LiteFS will handle everything else. 
It mostly depends on the number of write transactions per second. Most online stores are read-heavy so that’s a good fit for LiteFS. Unless you’re expecting to get a sustained 100+ products or orders per second, then you should be fine. We have improvements coming for write transaction throughput in the future.
Thanks this was super helpful!
I’m getting an error during deploy mount helper error: fusermount3: failed to open /dev/fuse: Permission denied I think due to having my user set to node do you happen to know if I can add the node user to a group that fuse uses? or do I need to give it a root user?
Run everything as root. It is okay to do so on Fly since each container is further jailed in a firecracker VM.
This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.