llmrix-router

<h1 align="center">LLMRix Model Router</h1> <p align="center"> <strong>A production-oriented multi-model routing and orchestration framework for Java.</strong> </p> <p align="center"> <em>OpenAI · DeepSeek · OpenRouter · Semantic routing · Contextual bandits · Fugu orchestration</em> </p> <p align="center"> <a href="https://github.com/llmrix/llmrix-router/stargazers"><img src="https://img.shields.io/github/stars/llmrix/llmrix-router?style=for-the-badge&logo=github&color=ffca28" alt="GitHub Stars"></a> <a href="https://github.com/llmrix/llmrix-router/network/members"><img src="https://img.shields.io/github/forks/llmrix/llmrix-router?style=for-the-badge&logo=github&color=8bc34a" alt="GitHub Forks"></a> <a href="./LICENSE"><img src="https://img.shields.io/badge/License-MIT-blue?style=for-the-badge" alt="License"></a> <a href="#requirements"><img src="https://img.shields.io/badge/Java-17%2B-orange?style=for-the-badge&logo=openjdk&logoColor=white" alt="Java Version"></a> <a href="docs/server.md"><img src="https://img.shields.io/badge/Spring%20Boot-3.x-6db33f?style=for-the-badge&logo=springboot&logoColor=white" alt="Spring Boot"></a> <a href="https://central.sonatype.com/artifact/com.llmrix.model/llmrix-model-router-core"><img src="https://img.shields.io/maven-central/v/com.llmrix.model/llmrix-model-router-core?style=for-the-badge&label=Maven%20Central&color=blue" alt="Maven Central"></a> </p> <p align="center"> <a href="#why-llmrix">Why LLMRix</a> • <a href="#architecture">Architecture</a> • <a href="#modules">Modules</a> • <a href="#quick-start">Quick Start</a> • <a href="#http-protocol">HTTP Protocol</a> • <a href="docs/server.md">Server</a> • <a href="docs/client.md">Client</a> </p>

LLMRix exposes multiple model providers through stable provider-neutral ModelClient and modality-specific model interfaces. It selects an eligible model using declared operations, features, input modalities, quality, cost, latency, quota, and health signals, then applies bounded retries and cooldown without leaking routing complexity into application code.

Use it as an embedded Java SDK, a Spring Boot starter, or an OpenAI-compatible routing service.

Project status: General Availability (1.0.2). The Java API and configuration model are production-ready, with Semantic Versioning strictly enforced. Published to Maven Central.

Why LLMRix

• Three first-class providers: OpenAI, DeepSeek, and OpenRouter over one validated OpenAI protocol transport.
• Policy separated from execution: strategies rank model targets; the executor owns timeout, retry, quota, and cooldown correctness.
• Streaming-safe candidate switching: the router can try another configured model before output begins and never replays after output begins.
• Local or distributed state: zero-infrastructure local mode and Redis-backed health, leases, RPM, and TPM for multi-instance deployments.
• OpenAI-compatible edge: Chat, Responses, Embeddings, Rerank, Audio, Images, Videos, Models, and SSE endpoints.
• Framework-neutral client: Orion provides a small Java client plus optional Spring Boot auto-configuration.
• Observable by design: lifecycle events, Micrometer metrics, Spring Observations, request IDs, and health indicators.
• Composable advanced routing: semantic routing, contextual bandits, online shadow traffic, evaluation, and Fugu-style iterative orchestration.

Architecture

Clients enter through embedded Java, Spring Boot, or OpenAI-compatible HTTP APIs. The provider-neutral core filters and ranks model targets, executes calls with reliability controls, shares runtime state, and emits telemetry. The framework owns routing semantics and request correctness. Infrastructure remains responsible for TLS, WAF, load balancing, Redis HA, secret management, telemetry storage, and container orchestration.

Modules

| Artifact | Responsibility | |---|---| | llmrix-model-open | Shared model contracts, common model exceptions/authentication SPI, and reusable OpenAI-compatible transport/adapters. | | llmrix-model-router-core | Runtime facade and Builder, model targets, strategies, execution, state SPI, provider SPI, quota, health, and events. | | llmrix-model-router-integrations | Default OpenAI/DeepSeek/OpenRouter registrations, Redis, Bucket4j, ONNX, evaluation, shadow, and Fugu adapters. | | llmrix-model-router-spring-starter | Router properties, auto-configuration, OpenAI-compatible HTTP/SSE endpoints, HTTP authentication, request IDs, Actuator, Micrometer/Observation, and configuration metadata. | | llmrix-model-orion | Lightweight framework-neutral Java client for the routing server. | | llmrix-model-orion-spring-starter | Orion auto-configuration and Micrometer integration. | | llmrix-model-examples | Maven aggregator for executable examples and module-scoped tests. Not a production dependency. | | llmrix-model-router-core-examples | Core routing examples and tests. | | llmrix-model-router-integrations-examples | Provider and infrastructure integration examples and tests. | | llmrix-model-router-spring-starter-examples | Spring Boot starter, HTTP protocol, and observability tests. | | llmrix-model-router-server-examples | Runnable standalone Spring Boot server example and launch smoke test. | | llmrix-model-client-examples | Orion client and client starter tests. |

Requirements

• Java 17 or later; Java 21 is recommended.
• Spring Boot 3.x when using either starter.
• Redis is optional and required only for shared multi-instance runtime state.

Quick Start

Maven

<dependency>
  <groupId>com.llmrix.model</groupId>
  <artifactId>llmrix-model-router-core</artifactId>
  <version>1.0.2</version>
</dependency>
<dependency>
  <groupId>com.llmrix.model</groupId>
  <artifactId>llmrix-model-router-integrations</artifactId>
  <version>1.0.2</version>
</dependency>

Programmatic configuration

The same router can be built without Spring or YAML. The runtime Builder lives in Core; the integrations artifact registers the built-in OpenAI-compatible providers through the Core SPI. Integrations own provider credentials and may define multiple models:

try (LlmRouter router = LlmRouter.builder()
        .integration("openai", integration -> integration
            .apiKey(System.getenv("OPENAI_API_KEY"))
            .model("gpt-4.1-mini", model -> model
                .operations(ModelOperation.CHAT).features(ModelFeature.TOOLS)))
        .integration("deepseek", integration -> integration
            .apiKey(System.getenv("DEEPSEEK_API_KEY"))
            .model("deepseek-chat", model -> model
                .operations(ModelOperation.CHAT).features(ModelFeature.TOOLS).traits(ModelTrait.CODE)))
        .route("general", route -> route
            .strategy("balanced")
            .quota(600L, 100_000L) // shared route RPM and TPM
            .models("openai/gpt-4.1-mini", "deepseek/deepseek-chat"))
        .build()) {
    ChatResponse response = router.chat("Review this Java code");
}

Route quotas are optional and apply to all targets in the route. The two-argument form is quota(requestsPerMinute, tokensPerMinute). When an authenticated request contains RoutingHints.AUTH_QUOTA_KEY, each key receives an independent quota partition; otherwise the route uses a shared partition. Target-level limits(...) remain independent provider-model limits.

Policy-based routing

RoutedChatModel model = RoutedChatModel.builder()
    .target("reasoning", reasoningModel, target -> target
        .operations(ModelOperation.CHAT).features(ModelFeature.TOOLS).traits(ModelTrait.REASONING)
        .inputCostPerMillion(1.25)
        .outputCostPerMillion(10.00))
    .target("fast", fastModel, target -> target
        .operations(ModelOperation.CHAT).traits(ModelTrait.CODE)
        .inputCostPerMillion(0.27)
        .outputCostPerMillion(1.10))
    .strategy(Strategies.balanced())
    .timeout(Duration.ofSeconds(30))
    .maxRetries(1)
    .build();

ChatResponse response = model.chat(ChatRequest.builder()
    .userMessage("Find the race condition")
    .routingHints(RoutingHints.builder()
        .require(ModelTrait.CODE)
        .maxCostUsd(0.05)
        .build())
    .build());

Applications can call synchronously, asynchronously, or as a Flow.Publisher<ChatChunk>. Text, images, input audio, tools, structured response formats, usage, finish reasons, and common generation options are represented by provider-neutral Core types.

Routing Model

Every request follows one deterministic execution pipeline:

1. Validate the request and normalize routing hints.
2. Remove targets that violate capability, model, context, cost, quota, concurrency, or health constraints.
3. Rank eligible targets with the configured strategy.
4. Acquire runtime quota and concurrency leases.
5. Execute with a bounded per-attempt and total timeout.
6. Retry only retryable failures and only within the configured budget.
7. Mark failures, apply cooldown, and move to the next eligible model in the route pool.
8. Settle token usage, release leases, and publish lifecycle observations.

Built-in strategies include priority, round-robin, weighted random, balanced scoring, semantic scoring, and contextual bandit selection. Custom policies implement RoutingStrategy; custom runtime persistence implements RouterStateStore or BanditStateStore.

HTTP Protocol

The Spring Boot starter exposes an OpenAI-compatible HTTP API. The model field in every request identifies a configured Router route name (such as general, vision, or multimodal) rather than an upstream provider model ID.

Enable the HTTP API

llmrix:
  model:
    router:
      http:
        enabled: true
        auth:
          mode: api-key
          bootstrap-key: ${LLMRIX_MODEL_ROUTER_API_KEY}

export BASE_URL=http://127.0.0.1:8080
export API_KEY=your-llmrix-http-key

Endpoint Catalog

| Endpoint | Description | |---|---| | POST /v1/chat/completions | Synchronous and SSE streaming chat completions. | | POST /v1/responses | Core Responses API subset, with JSON and SSE streaming responses. | | POST /v1/embeddings | Text or token-array embeddings with float and base64 encoding. | | POST /v1/rerank | Query/document reranking with relevance scores. | | POST /v1/audio/transcriptions | Multipart audio transcription. | | POST /v1/audio/translations | Multipart audio translation. | | POST /v1/audio/speech | Text-to-speech with a binary audio response. | | POST /v1/images/generations | Image generation. | | POST /v1/images/edits | Multipart image editing. | | POST /v1/videos | Create a video generation task. | | GET /v1/videos/{video_id} | Retrieve video task status. | | GET /v1/videos/{video_id}/content | Download completed video content. | | DELETE /v1/videos/{video_id} | Delete a video task. | | POST /v1/videos/{video_id}/remix | Create a remix task. | | GET /v1/models | Available chat route identifiers. Operation-only routes are selected by their endpoint. |

The server example includes embeddings and rerank routes backed by free OpenRouter models. The request model is the Router route name, not the upstream model ID:

curl --location "${BASE_URL}/v1/embeddings" \
  --header "Authorization: Bearer ${API_KEY}" \
  --header 'Content-Type: application/json' \
  --data '{"model":"embeddings","input":"Text to embed"}'

curl --location "${BASE_URL}/v1/rerank" \
  --header "Authorization: Bearer ${API_KEY}" \
  --header 'Content-Type: application/json' \
  --data '{"model":"rerank","query":"refund policy","documents":["Refunds are available within 30 days.","C