<div align="center"> <img src="assets/logo_claw.png" alt="LLMRouter Logo" width="200"> </div> <h1 align="center">🚀 LLMRouter: An Open-Source Library for LLM Routing</h1> <div align="center"> <p> <a href="https://www.python.org/downloads/release/python-3109/"><img src="https://img.shields.io/badge/PYTHON-3.10-3776AB?style=for-the-badge&logo=python&logoColor=white" alt="Python"></a> <a href="https://github.com/ulab-uiuc/LLMRouter/pulls"><img src="https://img.shields.io/badge/PRS-WELCOME-orange?style=for-the-badge" alt="PRs"></a> <a href="https://join.slack.com/t/llmrouteropen-ri04588/shared_invite/zt-3mkx82cut-A25v5yR52xVKi7_jm_YK_w"><img src="https://img.shields.io/badge/SLACK-JOIN%20US-4A154B?style=for-the-badge&logo=slack&logoColor=white" alt="Slack"></a> <a href="https://github.com/ulab-uiuc/LLMRouter/issues/136"><img src="https://img.shields.io/badge/💬WeChat-Group-07c160?style=for-the-badge&logo=wechat&logoColor=white&labelColor=1a1a2e"></a> <a href="https://ulab-uiuc.github.io/LLMRouter/" style="text-decoration:none;"><img src="https://img.shields.io/badge/DOCS-ONLINE-0A9EDC?style=for-the-badge&logo=readthedocs&logoColor=white" alt="Docs"></a> <a href="https://x.com/youjiaxuan/status/2005877938554589370" style="text-decoration:none;"><img src="https://img.shields.io/badge/TWITTER-ANNOUNCEMENTS-1DA1F2?style=for-the-badge&logo=x&logoColor=white" alt="Twitter"></a> <a href="LICENSE"><img src="https://img.shields.io/badge/LICENSE-MIT-2EA44F?style=for-the-badge" alt="License"></a> </p> </div>

✨ Introduction

LLMRouter is an intelligent routing system designed to optimize LLM inference by dynamically selecting the most suitable model for each query. To achieve intelligent routing, it defines:

🚀 Smart Routing: Automatically routes queries to the optimal LLM based on task complexity, cost, and performance requirements.
📊 Multiple Router Models: Support for over 16 routing models, organized into four major categories—single-round routers, multi-round routers, agentic routers, and personalized routers—covering a wide range of strategies such as KNN, SVM, MLP, Matrix Factorization, Elo Rating, graph-based routing, BERT-based routing, hybrid probabilistic methods, transformed-score routers, and more.
🛠️ Unified CLI: Complete command-line interface for training, inference, and interactive chat with Gradio-based UI.
📈 Data Generation Pipeline: Complete pipeline for generating training data from 11 benchmark datasets with automatic API calling and evaluation.

📰 News

🖥️ [2026-02]: ComfyUI Interface - We've released the visual interface for LLMRouter! Now you can visually construct data generation and routing pipelines, drag-and-drop nodes to train routers, and monitor performance in real-time. See ComfyUI Interface for details.
🔗 [2026-02]: OpenClaw Router - OpenAI-compatible server with OpenClaw integration! We've also released llmrouter-lib v0.3.1. Deploy LLMRouter as a production API server that works seamlessly with Slack, Discord, and other messaging platforms via OpenClaw. Features include multimodal understanding (image/audio/video), retrieval-augmented routing memory, streaming support, and all 16+ LLMRouter routing strategies. See OpenClaw Router Integration. For deployment with social platforms like Slack, refer to the Getting Started Guide for step-by-step setup instructions.
⭐ [2026-01]: LLMRouter just crossed 1K GitHub stars! We've also released llmrouter-lib v0.2.0. Updates include service-specific dict configs (OpenAI, Anthropic, etc.) and multimodal routing (Video/Image + Text) on Geometry3K, MathVista, and Charades-Ego—all in the first unified open-source LLM routing library with 16+ routers, a unified CLI, Gradio UI, and 11 datasets. Install via pip install llmrouter-lib. More updates soon! 🚀
🚀 [2025-12]: LLMRouter is officially released - ship smarter 🧠, cost-aware 💸 LLM routing with 16+ routers 🧭, a unified llmrouter CLI 🛠️, and a plugin workflow for custom routers 🧩.

🧭 Supported Routers

Single-Round Routers

| Router | Training | Inference | Description | Tutorial | |--------|:--------:|:---------:|-------------|:--------:| | knnrouter | ✅ | ✅ | K-Nearest Neighbors based routing | 📖 | | svmrouter | ✅ | ✅ | Support Vector Machine based routing | 📖 | | mlprouter | ✅ | ✅ | Multi-Layer Perceptron based routing | 📖 | | mfrouter | ✅ | ✅ | Matrix Factorization based routing | 📖 | | elorouter | ✅ | ✅ | Elo Rating based routing | 📖 | | routerdc | ✅ | ✅ | Dual Contrastive learning based routing | 📖 | | automix | ✅ | ✅ | Automatic model mixing | 📖 | | hybrid_llm | ✅ | ✅ | Hybrid LLM routing strategy | 📖 | | graphrouter | ✅ | ✅ | Graph-based routing | 📖 | | causallm_router | ✅ | ✅ | Causal Language Model router | 📖 | | smallest_llm | N/A | ✅ | Always routes to smallest model | 📖 | | largest_llm | N/A | ✅ | Always routes to largest model | 📖 |

Multi-Round Routers

| Router | Training | Inference | Description | Tutorial | |--------|:--------:|:---------:|-------------|:--------:| | router_r1 | LINK | ✅ | Pre-trained Router-R1 model for multi-turn conversations | 📖 |

Personalized Routers

| Router | Training | Inference | Description | Tutorial | |--------|:--------:|:---------:|-------------|:--------:| | gmtrouter | ✅ | ✅ | Graph-based personalized router with user preference learning | 📖 | | personalizedrouter | ✅ | ✅ | GNN-based personalized router with user features | 📖 |

Agentic Routers

| Router | Training | Inference | Description | Tutorial | |--------|:--------:|:---------:|-------------|:--------:| | knnmultiroundrouter | ✅ | ✅ | KNN-based agentic router for complex tasks | 📖 | | llmmultiroundrouter | N/A | ✅ | LLM-based agentic router for complex tasks | 📖 |

🚀 Get Started

Installation

Install from source

Clone the repository and install in editable mode using a virtual environment (e.g., with anaconda3):

# Clone the repository
git clone https://github.com/ulab-uiuc/LLMRouter.git
cd LLMRouter

# Create and activate virtual environment
conda create -n llmrouter python=3.10
conda activate llmrouter

# Install the package (base installation)
pip install -e .

# Optional: Install with RouterR1 support (requires GPU)
# RouterR1 is tested with vllm==0.6.3 (torch==2.4.0); the extra pins these versions.
pip install -e ".[router-r1]"

# Optional: Install all optional dependencies
pip install -e ".[all]"

Install from PyPI

pip install llmrouter-lib

🔑 Setting Up API Keys

LLMRouter requires API keys to make LLM API calls for inference, chat, and data generation. Set the API_KEYS environment variable using one of the following formats:

💡 Free NVIDIA API Keys: The NVIDIA endpoints currently used in LLMRouter have freely available API keys. To get started, visit https://build.nvidia.com/ to create an account, then you can generate your API keys at no cost.

Service-Specific Dict Format (recommended for multiple providers)

Use this format when you have models from different service providers (e.g., NVIDIA, OpenAI, Anthropic) and want to use different API keys for each provider:

export API_KEYS='{"NVIDIA": "nvidia-key-1,nvidia-key-2", "OpenAI": ["openai-key-1", "openai-key-2"], "Anthropic": "anthropic-key-1"}'

Dict Format Details:

Keys: Service provider names (must match the service field in your LLM candidate JSON)
Values: Can be:
- Comma-separated string: "key1,key2,key3"
- JSON array: ["key1", "key2", "key3"]
- Single string: "key1"
Service Matching: The system automatically matches the service field from your LLM candidate JSON to select the appropriate API keys
Round-Robin: Each service maintains its own round-robin counter for load balancing
Error Handling: If a service is not found in the dict, a clear error message will be raised with available services listed

Example LLM Candidate JSON with service field:

{
  "qwen2.5-7b-instruct": {
    "service": "NVIDIA",
    "model": "qwen/qwen2.5-7b-instruct",
    "api_endpoint": "https://integrate.api.nvidia.com/v1"
  },
  "gpt-4": {
    "service": "Open

LLMRouter

Install / Use

README