🥤 RAGLite

RAGLite is a Python toolkit for Retrieval-Augmented Generation (RAG) with DuckDB or PostgreSQL.

Features

Configurable

• 🧠 Choose any LLM provider with LiteLLM, including local llama-cpp-python models
• 💾 Choose either DuckDB or PostgreSQL as a keyword & vector search database
• 🥇 Choose any reranker with rerankers, including multilingual FlashRank as the default

Fast and permissive

• ❤️ Only lightweight and permissive open source dependencies (e.g., no PyTorch or LangChain)
• 🚀 Acceleration with Metal on macOS, and CUDA on Linux and Windows

Unhobbled

• 📖 PDF to Markdown conversion on top of pdftext and pypdfium2
• 🧬 Multi-vector chunk embedding with late chunking and contextual chunk headings
• ✏️ Optimal sentence splitting with wtpsplit-lite by solving a binary integer programming problem
• ✂️ Optimal semantic chunking by solving a binary integer programming problem
• 🔍 Hybrid search with the database's native keyword & vector search (FTS+VSS; tsvector+pgvector)
• 💭 Adaptive retrieval where the LLM decides whether to and what to retrieve based on the query
• 💰 Improved cost and latency with a prompt caching-aware message array structure
• 🍰 Improved output quality with Anthropic's long-context prompt format
• 🌀 Optimal closed-form linear query adapter by solving an orthogonal Procrustes problem

Extensible

• 🔌 A built-in Model Context Protocol (MCP) server that any MCP client like Claude desktop can connect with
• 💬 Optional customizable ChatGPT-like frontend for web, Slack, and Teams with Chainlit
• ✍️ Optional conversion of any input document to Markdown with Pandoc
• 🔎 Optional high-quality document processing with Mistral OCR for PDFs, images, DOCX, and PPTX with automatic image descriptions
• ✅ Optional evaluation of retrieval and generation performance with Ragas

Installing

[!TIP] 🚀 If you want to use local models, it is recommended to install an accelerated llama-cpp-python precompiled binary with:

# Configure which llama-cpp-python precompiled binary to install (⚠️ not every combination is available):
LLAMA_CPP_PYTHON_VERSION=0.3.9
PYTHON_VERSION=310|311|312
ACCELERATOR=metal|cu121|cu122|cu123|cu124
PLATFORM=macosx_11_0_arm64|linux_x86_64|win_amd64

# Install llama-cpp-python:
pip install "https://github.com/abetlen/llama-cpp-python/releases/download/v$LLAMA_CPP_PYTHON_VERSION-$ACCELERATOR/llama_cpp_python-$LLAMA_CPP_PYTHON_VERSION-cp$PYTHON_VERSION-cp$PYTHON_VERSION-$PLATFORM.whl"

Install RAGLite with:

pip install raglite

To add support for a customizable ChatGPT-like frontend, use the chainlit extra:

pip install raglite[chainlit]

To add support for filetypes other than PDF, use the pandoc extra:

pip install raglite[pandoc]

To add support for high-quality document processing with Mistral OCR, install mistralai:

pip install mistralai

To add support for evaluation, use the ragas extra:

pip install raglite[ragas]

Using

Overview

1. Configuring RAGLite
2. Inserting documents
3. Retrieval-Augmented Generation (RAG)
4. Computing and using an optimal query adapter
5. Evaluation of retrieval and generation
6. Running a Model Context Protocol (MCP) server
7. Serving a customizable ChatGPT-like frontend

1. Configuring RAGLite

[!TIP] 🧠 RAGLite extends LiteLLM with support for llama.cpp models using llama-cpp-python. To select a llama.cpp model (e.g., from Unsloth's collection), use a model identifier of the form "llama-cpp-python/<hugging_face_repo_id>/<filename>@<n_ctx>", where n_ctx is an optional parameter that specifies the context size of the model.

[!TIP] 💾 You can create a PostgreSQL database in a few clicks at neon.tech.

First, configure RAGLite with your preferred DuckDB or PostgreSQL database and any LLM supported by LiteLLM:

from raglite import RAGLiteConfig

# Example 'remote' config with a PostgreSQL database and an OpenAI LLM:
my_config = RAGLiteConfig(
    db_url="postgresql://my_username:my_password@my_host:5432/my_database",
    llm="gpt-4o-mini",  # Or any LLM supported by LiteLLM
    embedder="text-embedding-3-large",  # Or any embedder supported by LiteLLM
)

# Example 'local' config with a DuckDB database and a llama.cpp LLM:
my_config = RAGLiteConfig(
    db_url="duckdb:///raglite.db",
    llm="llama-cpp-python/unsloth/Qwen3-8B-GGUF/*Q4_K_M.gguf@8192",
    embedder="llama-cpp-python/lm-kit/bge-m3-gguf/*F16.gguf@512", # More than 512 tokens degrades bge-m3's performance
)

You can also configure any reranker supported by rerankers:

from rerankers import Reranker

# Example remote API-based reranker:
my_config = RAGLiteConfig(
    db_url="postgresql://my_username:my_password@my_host:5432/my_database"
    reranker=Reranker("rerank-v3.5", model_type="cohere", api_key=COHERE_API_KEY, verbose=0)  # Multilingual
)

# Example local cross-encoder reranker per language (this is the default):
my_config = RAGLiteConfig(
    db_url="duckdb:///raglite.db",
    reranker={
        "en": Reranker("ms-marco-MiniLM-L-12-v2", model_type="flashrank", verbose=0),  # English
        "other": Reranker("ms-marco-MultiBERT-L-12", model_type="flashrank", verbose=0),  # Other languages
    }
)

Self-query is also supported, allowing the LLM to automatically generate and apply metadata filters to refine search results based on the user's input. To enable self-query, set self_query=True in your RAGLiteConfig:

my_config = RAGLiteConfig(
    db_url="duckdb:///raglite.db",
    llm="gpt-4o-mini",
    embedder="text-embedding-3-large",
    self_query=True,  # Enable self-query
)

2. Inserting documents

[!TIP] ✍️ To insert documents other than PDF, install the pandoc extra with pip install raglite[pandoc].

[!TIP] 🔎 For higher-quality document processing with automatic image descriptions, install mistralai and configure it as follows:

from raglite import RAGLiteConfig, MistralOCRConfig

my_config = RAGLiteConfig(
    document_processor=MistralOCRConfig(
        include_image_descriptions=True,  # Describe images, charts, and diagrams as text
        image_types=frozenset({"chart", "diagram", "photo", "table", "logo", "icon"}),  # Custom image categories
        exclude_image_types=frozenset({"logo", "icon"}),  # Filter out specific types from the output
    ),
)

The image_types parameter defines the categories that Mistral classifies each image into — you can use the defaults or provide your own domain-specific types. Use exclude_image_types to filter out any classified types that are not useful for retrieval.

Next, insert some documents into the database. RAGLite will take care of the conversion to Markdown, optimal level 4 semantic chunking, and [multi-vecto