Sonic

Sonic is a fast, lightweight and schema-less search backend. It ingests search texts and identifier tuples that can then be queried against in a microsecond's time.

Sonic can be used as a simple alternative to super-heavy and full-featured search backends such as Elasticsearch in some use-cases. It is capable of normalizing natural language search queries, auto-completing a search query and providing the most relevant results for a query. Sonic is an identifier index, rather than a document index; when queried, it returns IDs that can then be used to refer to the matched documents in an external database.

A strong attention to performance and code cleanliness has been given when designing Sonic. It aims at being crash-free, super-fast and puts minimum strain on server resources (our measurements have shown that Sonic - when under load - responds to search queries in the μs range, eats ~30MB RAM and has a low CPU footprint; see our benchmarks).

Tested at Rust version: rustc 1.74.1 (a28077b28 2023-12-04)

🇫🇷 Crafted in Nantes, France.

:newspaper: The Sonic project was initially announced in a post on my personal journal.

« Sonic » is the mascot of the Sonic project. I drew it to look like a psychedelic hipster hedgehog.

Who uses it?

<table> <tr> <td align="center"><a href="https://crisp.chat/"><img src="https://valeriansaliou.github.io/sonic/images/logo-crisp.png" width="64" /></a></td> <td align="center"><a href="https://scrumpy.io/"><img src="https://valeriansaliou.github.io/sonic/images/logo-scrumpy.png" width="64" /></a></td> </tr> <tr> <td align="center">Crisp</td> <td align="center">Scrumpy</td> </tr> </table>

👋 You use Sonic and you want to be listed there? Contact me.

Demo

Sonic is integrated in all Crisp search products on the Crisp platform. It is used to index half a billion objects on a $5/mth 1-vCPU SSD cloud server (as of 2019). Crisp users use it to search in their messages, conversations, contacts, helpdesk articles and more.

You can test Sonic live on: Crisp Helpdesk, and get an idea of the speed and relevance of Sonic search results. You can also test search suggestions from there: start typing at least 2 characters for a word, and get suggested a full word (press the tab key to expand suggestion). Both search and suggestions are powered by Sonic.

Sonic fuzzy search in helpdesk articles at its best. Lookup for any word or group of terms, get results instantly.

Features

• Search terms are stored in collections, organized in buckets; you may use a single bucket, or a bucket per user on your platform if you need to search in separate indexes.
• Search results return object identifiers, that can be resolved from an external database if you need to enrich the search results. This makes Sonic a simple word index, that points to identifier results. Sonic doesn't store any direct textual data in its index, but it still holds a word graph for auto-completion and typo corrections.
• Search query typos are corrected if there are not enough exact-match results for a given word in a search query, Sonic tries to correct the word and tries against alternate words. You're allowed to make mistakes when searching.
• Insert and remove items in the index; index-altering operations are light and can be committed to the server while it is running. A background tasker handles the job of consolidating the index so that the entries you have pushed or popped are quickly made available for search.
• Auto-complete any word in real-time via the suggest operation. This helps build a snappy word suggestion feature in your end-user search interface.
• Full Unicode compatibility on 80+ most spoken languages in the world. Sonic removes useless stop words from any text (eg. 'the' in English), after guessing the text language. This ensures any searched or ingested text is clean before it hits the index; see languages.
• Simple protocol (Sonic Channel), that let you search your index, manage data ingestion (push in the index, pop from the index, flush a collection, flush a bucket, etc.) and perform administrative actions. Sonic Channel was designed to be lightweight on resources and simple to integrate with; read protocol specification.
• Easy-to-use libraries, that let you connect to Sonic from your apps; see libraries.

How to use it?

Installation

Sonic is built in Rust. To install it, either download a version from the Sonic releases page, use cargo install or pull the source code from master.

👉 Each release binary comes with an .asc signature file, which can be verified using @valeriansaliou GPG public key: :key:valeriansaliou.gpg.pub.asc.

👉 Install from packages:

Sonic provides pre-built packages for Debian-based systems (Debian, Ubuntu, etc.).

Important: Sonic only provides 64 bits packages targeting Debian 12 for now (codename: bookworm). You might still be able to use them on other Debian versions, as well as Ubuntu (although they rely on a specific glibc version that might not be available on older or newer systems).

First, add the Sonic APT repository (eg. for Debian bookworm):

echo "deb [signed-by=/usr/share/keyrings/valeriansaliou_sonic.gpg] https://packagecloud.io/valeriansaliou/sonic/debian/ bookworm main" > /etc/apt/sources.list.d/valeriansaliou_sonic.list

curl -fsSL https://packagecloud.io/valeriansaliou/sonic/gpgkey | gpg --dearmor -o /usr/share/keyrings/valeriansaliou_sonic.gpg

apt-get update

Then, install the Sonic package:

apt-get install sonic

Then, edit the pre-filled Sonic configuration file:

nano /etc/sonic.cfg

Finally, restart Sonic:

service sonic restart

👉 Install from source:

If you pulled the source code from Git, you can build it using cargo:

cargo build --release

You can find the built binaries in the ./target/release directory.

Install build-essential, clang, libclang-dev, libc6-dev, g++ and llvm-dev to be able to compile the required RocksDB dependency.

Note that the following optional features can be enabled upon building Sonic: allocator-jemalloc, tokenizer-chinese and tokenizer-japanese (some might be already enabled by default).

👉 Install from Cargo:

You can install Sonic directly with cargo install:

cargo install sonic-server

Ensure that your $PATH is properly configured to source the Crates binaries, and then run Sonic using the sonic command.

Install build-essential, clang, libclang-dev, libc6-dev, g++ and llvm-dev to be able to compile the required RocksDB dependency.

👉 Install from Docker Hub:

You might find it convenient to run Sonic via Docker. You can find the pre-built Sonic image on Docker Hub as valeriansaliou/sonic.

First, pull the valeriansaliou/sonic image:

docker pull valeriansaliou/sonic:v1.4.9

Then, seed it a configuration file and run it (replace /path/to/your/sonic/config.cfg with the path to your configuration file):

docker run -p 1491:1491 -v /path/to/your/sonic/config.cfg:/etc/sonic.cfg -v /path/to/your/sonic/store/:/var/lib/sonic/store/ valeriansaliou/sonic:v1.4.9

In the configuration file, ensure that:

• channel.inet is set to 0.0.0.0:1491 (this lets Sonic be reached from outside the container)
• store.kv.path is set to /var/lib/sonic/store/kv/ (this lets the external KV store directory be reached by Sonic)
• store.fst.path is set to /var/lib/sonic/store/fst/ (this lets the external FST store directory be reached by Sonic)

Sonic will be reachable from tcp://localhost:1491.

👉 Install from another source (non-official):

Other installation sources are available:

• Homebrew (macOS): brew install sonic (see formula)

Note that those sources are non-official, meaning that they are not owned nor maintained by the Sonic project owners. The latest Sonic version available on those sources might be outdated, in comparison to the latest version available through the Sonic project.

Configuration

Use the sample config.cfg configuration file and adjust it to your own environment.

_If you are looking to fine-tune your configuration, you may read our [detailed configurati