There is a new addition to LiteFS Cloud we’ve been working on - LiteVFS. LiteVFS is a dynamically loadable extension for SQLite that provides a SQLite Virtual File System (VFS) for on-demand paging from LiteFS Cloud.
https://github.com/superfly/litevfs
Intro
LiteFS is great, but it may not be suitable for dynamic or ephemeral environments, like Machines with auto-start/auto-stop, Machines without volumes or AWS Lambda. LiteVFS is designed specifically with such use cases in mind, uses LiteFS Cloud for persistence and locally available space for cache.
A database opened via LiteVFS will be fetched from LiteFS Cloud and cached locally. Subsequent reads that need the same data pages are served from local cache and don’t need any network access.
Only the pages requested by SQLite (with some additional prefetching) are fetched from LiteFS Cloud, so, if the query needs only a small subset of data, this should work well even on large databases.
LiteVFS periodically queries LiteFS Cloud to maintain the same database state. Pages changed on LiteFS Cloud side are dropped from the local cache and refetched when needed. On average, you can expect ~1 second delay between LiteFS Cloud state and local LiteVFS state.
How to use it
The extension can be loaded from any language with SQLite bindings or from SQLite shell:
$ export LITEFS_CLOUD_TOKEN="<your token here>"
$ sqlite
sqlite> .load target/release/liblitevfs.so
sqlite> .open file:demo.db?vfs=litevfs
sqlite> .tables
data
sqlite>
I’ve made a couple of demo apps to show how to use it from Go and Node: https://github.com/fly-apps/litevfs-demo
Writing via LiteVFS
Each LiteVFS node can write data, but it needs to hold a write lease. Writing data via LiteVFS results in several network requests to LiteFS Cloud: acquiring a lease, sending changes, releasing the lease. A lease can be acquired by issuing a litevfs_acquire_lease pragma statement:
sqlite> pragma litevfs_acquire_lease;
sqlite> insert into data (data) values (123);
sqlite> pragma litevfs_release_lease;
Using together with LiteFS
LiteVFS can be used to read data from LiteFS Cluster backed up to LiteFS Cloud. At the moment, it’s not safe to write data to the same clusters via LiteVFS, but it can a good way to quickly bring up additional read-only replicas on machines without volumes.
Limitations
For now, databases in WAL journal mode can be opened only in read-only mode. The same applies to the databases with auto_vacuum enabled.
Manually issued VACUUM statements may also fail if they end up reducing database file size.