There is a new addition to LiteFS Cloud we’ve been working on - LiteVFS. LiteVFS is a dynamically loadable extension for SQLite that provides a SQLite Virtual File System (VFS) for on-demand paging from LiteFS Cloud.
LiteFS is great, but it may not be suitable for dynamic or ephemeral environments, like Machines with auto-start/auto-stop, Machines without volumes or AWS Lambda. LiteVFS is designed specifically with such use cases in mind, uses LiteFS Cloud for persistence and locally available space for cache.
A database opened via LiteVFS will be fetched from LiteFS Cloud and cached locally. Subsequent reads that need the same data pages are served from local cache and don’t need any network access.
Only the pages requested by SQLite (with some additional prefetching) are fetched from LiteFS Cloud, so, if the query needs only a small subset of data, this should work well even on large databases.
LiteVFS periodically queries LiteFS Cloud to maintain the same database state. Pages changed on LiteFS Cloud side are dropped from the local cache and refetched when needed. On average, you can expect ~1 second delay between LiteFS Cloud state and local LiteVFS state.
The extension can be loaded from any language with SQLite bindings or from SQLite shell:
$ export LITEFS_CLOUD_TOKEN="<your token here>" $ sqlite sqlite> .load target/release/liblitevfs.so sqlite> .open file:demo.db?vfs=litevfs sqlite> .tables data sqlite>
I’ve made a couple of demo apps to show how to use it from Go and Node: https://github.com/fly-apps/litevfs-demo
Each LiteVFS node can write data, but it needs to hold a write lease. Writing data via LiteVFS results in several network requests to LiteFS Cloud: acquiring a lease, sending changes, releasing the lease. A lease can be acquired by issuing a
litevfs_acquire_lease pragma statement:
sqlite> pragma litevfs_acquire_lease; sqlite> insert into data (data) values (123); sqlite> pragma litevfs_release_lease;
LiteVFS can be used to read data from LiteFS Cluster backed up to LiteFS Cloud. At the moment, it’s not safe to write data to the same clusters via LiteVFS, but it can a good way to quickly bring up additional read-only replicas on machines without volumes.
For now, databases in WAL journal mode can be opened only in read-only mode. The same applies to the databases with auto_vacuum enabled.
VACUUM statements may also fail if they end up reducing database file size.