LumoSQL

LumoSQL is a modification (not a fork) of the SQLite embedded data storage library, the most-deployed software. We are currently in Phase II of the project.

If you are reading this on GitHub you are looking at a read-only mirror. The master is always available at lumosql.org. LumoSQL adds security, privacy, performance and measurement features to SQLite.

Benchmarking

SQLite can test and compare results consistently across many kinds of system and configurations using the Not-forking tool. Example:

Example statistical result from LumoSQL benchmarking

Pluggable backends

LumoSQL can swap back end key-value store engines in and out of SQLite. LMDB is the most famous (but not the only) example of an alternative key-value store, and LumoSQL can combine dozes of versions of LMDB and SQLite source code like this:

Diagram of Not-forking used to create two example binaries

In LumoSQL 0.4 there are three LumoSQL backends:

the default SQLite Btree storage system
LMDB
the ancient Berkley Database

We are looking at some interesting new development in key-value storage to add and benchmark.

Encryption and corruption detection, optionally per-row

LumoSQL adds modern encryption to SQLite, including Attribute-Based Encryption (ABE). This can be done on a per-row basis, and also includes per-row checksums so that any error can be noticed quickly and located down to the individual row. Per-row checksums also make some search and comparison operations much faster.

Organised and Supported

LumoSQL is distributed under very liberal MIT licence terms.

LumoSQL is supported by the NLNet Foundation.

LumoSQL runs on x86, ARM-32 and RISC-V architectures, and many Linux and BSD OSs.

Design, Not-Forking and Participating
LumoSQL, and SQLite's Billions of Users
Limitations of LumoSQL
Build Environment and Dependencies
Using the Build and Benchmark System
A Brief History of LumoSQL

Design, Not-Forking and Participating

If you are reading this on Github, then you are looking at a mirror. LumoSQL is is maintained using the Fossil repository. If you want to participate in LumoSQL there is a forum, and if you have code contributions you can ask for access to the respository.

LumoSQL has multiple upstreams, but does not fork any of them despite needing modifications. The novel Not-forking tool semi-automatically tracks upstream changes and is a requirement for building LumoSQL. Between not-forking and the LumoSQL Build and Benchmark System, LumoSQL is as much about combining and configuring upstreams as it is about creating original database software. By maintaining Not-forking outside LumoSQL, we hope other projects will find it useful.

The LumoSQL and SQLite projects are cooperating, so any merge friction is expected to become less over time, and key to that is the approach of not forking.

LumoSQL, and SQLite's Billions of Users

LumoSQL exists to demonstrate changes to SQLite that might be useful, but which SQLite probably cannot consider for many years because of SQLite's unique position of being used by a majority of the world's population.

SQLite is used by thousands of software projects, just three being Google's Android, Mozilla's Firefox and Apple's iOS which between them have billions of users. That is a main reason why SQLite is so careful and conservative with all changes.

On the other hand, many of these same users need SQLite to have new features which do not fit with the SQLite project's cautious approach, and LumoSQL is a demonstration of some of these improvements.

The LumoSQL documentation project reviews dozens of relevant codebases. SQLite has become ubiquitous over two decades, which means there is a great deal of preparation needed when considering architectural changes.

Limitations of LumoSQL

As of LumoSQL 0.4, there are many obvious limitations, including:

The tests used in benchmarking mostly come from an ancient version of SQLite's speedtest.tcl modified many times, to which DATASIZE and DEBUG have been added. Experts in SQLite and LMDB database testing should review the files in not-fork.d/sqlite3/benchmark/*test. There are 9 tools named *speed* in the SQLite source, and any/all of them should be added here.
Neither LMDB nor BDB backends ship with latest SQLite builds. Now all the LumoSQL infrastructure exists, that is a smaller, more maintainable and repeatable task. But it is not done yet. There are some generic problems to be solved in the process, such as the optimal way to address keysize disparities between a KVP store provider and SQLite's internal large keysize.
If we import more of the speed tests from SQLite identified above, then we will have a problem with several LMDB and at least two BDB instances, where the SQLite tests will fail. In most cases this is about the LMDB port needing to be more complete but in some it is about relevance, where some SQLite tests will not apply. In addition some backends will always need to have additional tests (for example, BDB has more extensive user management than SQLite).

Build Environment and Dependencies

Most developers already have the required minimum of git and core unix-style development tools. SQLite has very few dependencies (mostly Tcl), and LumoSQL adds one Perl-based processing tool.

LumoSQL is mirrored to Github and application developers can use git with Github in the usual way. LumoSQL developers working on the LumoSQL library internals choose to use Fossil source code manager instead of git, and if you're planning to develop LumoSQL internals then you need Fossil.

There are many reasons why people choose Fossil. For LumoSQL one of them is that SQLite and Fossil are symbiotic projects, each written in the other.

Debian or Ubuntu-derived Operating Systems

Uncomment existing deb-src line in /etc/apt/sources.list, for example for Ubuntu 20.04.2 a valid line is:

deb-src http://gb.archive.ubuntu.com/ubuntu focal main restricted

Then run

sudo apt update                              # this fetches the deb-src updates
sudo apt full-upgrade                        # this gets the latest OS updates
sudo apt install git build-essential tclx
sudo apt build-dep sqlite3

The exact commands above have been tested on a pristine install of Ubuntu 20.04.2 LTS, as installed from ISO or one of the operating systems shipped with Windows Services for Linux.

Fedora-derived Operating Systems

On any reasonably recent Fedora-derived Linux distribution, including Red Hat:

```sh sudo dnf install --assumeyes \ git make gcc ncurses-devel readline-devel glibc-devel autoconf tcl-devel tclx-devel ```

Common to all Linux Operating Systems

Once you have done the setup specific to your operating system in the previous steps, the following should work on reaonably recent Debian and Fedora-related operating systems, and Gentoo.

Other required tools can be installed from your operating system's standard packages. Here are the tool dependencies:

Mandatory: the not-forking tool, which is a Perl script that needs to be downloaded and installed manually. The instructions for not-forking are on its website.
Recommended: Fossil. As described above, you don't necessarily need Fossil. But Fossil is very easy to install: if you can't get version 2.13 or later from your distrbution then it is easy to build from source. (Note! Ubuntu 20.04, Debian Buster and Gentoo do not include a sufficiently modern Fossil, while NetBSD and Ubuntu 20.10 do.) Since you now have a development environment anyway you can build Fossil trunk according to the official instructions or this simpler version (tested on Ubuntu 20.04 LTS):
- wget -O- https://fossil-scm.org/home/tarball/trunk/Fossil-trunk.tar.gz | tar -zxf -
- sudo apt install libssl-dev
- cd Fossil-trunk ; ./configure ; make
- sudo make install
For completeness (although every modern Linux/Unix includes these), to build and benchmark any of the Oracle Berkeley DB targets, you need either "curl" or "wget", and also "file", "gzip" and GNU "tar". Just about any version of these will be sufficient, even on Windows.
If you are running inside a fresh Docker or similar container system, Fossil may be confused about the user id. One solution is to

Lumosql

Install / Use

README