llm-council

<p align="center"> <a href="https://amiable-dev.github.io/llm-council"> <img src="https://raw.githubusercontent.com/amiable-dev/llm-council/master/docs/img/logo.svg" alt="LLM Council Logo" width="200"> </a> </p> <h1 align="center">LLM Council Core</h1> <p align="center"> <a href="https://github.com/amiable-dev/llm-council/actions/workflows/ci.yml"><img src="https://github.com/amiable-dev/llm-council/actions/workflows/ci.yml/badge.svg" alt="CI"></a> <a href="https://github.com/amiable-dev/llm-council/actions/workflows/security.yml"><img src="https://github.com/amiable-dev/llm-council/actions/workflows/security.yml/badge.svg" alt="Security Scanning"></a> <a href="https://scorecard.dev/viewer/?uri=github.com/amiable-dev/llm-council"><img src="https://api.scorecard.dev/projects/github.com/amiable-dev/llm-council/badge" alt="OpenSSF Scorecard"></a> <a href="https://www.bestpractices.dev/projects/14204"><img src="https://www.bestpractices.dev/projects/14204/badge" alt="OpenSSF Best Practices"></a> <a href="https://pypi.org/project/llm-council-core/"><img src="https://img.shields.io/pypi/v/llm-council-core.svg" alt="PyPI version"></a> <a href="https://www.python.org/downloads/"><img src="https://img.shields.io/badge/python-3.11+-blue.svg" alt="Python 3.11+"></a> <a href="https://opensource.org/licenses/MIT"><img src="https://img.shields.io/badge/License-MIT-yellow.svg" alt="License: MIT"></a> <a href="https://llm-council.dev"><img src="https://img.shields.io/badge/docs-llm--council.dev-blue" alt="Documentation"></a> <a href="https://discord.gg/y467DGHF"><img src="https://img.shields.io/badge/Discord-Join%20Chat-7289da?logo=discord&logoColor=white" alt="Discord"></a> <a href="https://github.com/users/amiable-dev/projects/1"><img src="https://img.shields.io/badge/roadmap-project%20board-blue?logo=github" alt="Roadmap"></a> </p> <p align="center"> <em>A multi-LLM deliberation system where multiple LLMs collaboratively answer questions through peer review and synthesis. Available as a Python library, MCP server, or HTTP API.</em> </p>

What is This?

Instead of asking a single LLM for answers, this MCP server:

1. Stage 1: Sends your question to multiple LLMs in parallel (GPT, Claude, Gemini, Grok, etc.)
2. Stage 2: Each LLM reviews and ranks the other responses (anonymized to prevent bias)
3. Stage 3: A Chairman LLM synthesizes all responses into a final, high-quality answer

Quick Deploy

Deploy your own LLM Council instance:

| Platform | Deploy | Best For | |----------|--------|----------| | Railway | | Production, webhooks | | Render | | Evaluation, free tier |

Required Environment Variables:

• OPENROUTER_API_KEY - Your OpenRouter API key
• LLM_COUNCIL_API_TOKEN - A secure token for API authentication (generate with openssl rand -hex 16)

Note: Railway is recommended for n8n integration (no cold-start). Render Free tier spins down after 15 minutes which may cause webhook timeouts.

For detailed deployment instructions, see the Deployment Guide.

Credits & Attribution

This project is a derivative work based on the original llm-council by Andrej Karpathy.

Karpathy's original README stated:

"I'm not going to support it in any way, it's provided here as is for other people's inspiration and I don't intend to improve it. Code is ephemeral now and libraries are over, ask your LLM to change it in whatever way you like."

...the irony of producing a derivative work that packages the core concept for broader use via the Model Context Protocol!

Installation

# Recommended (isolated tool install; puts `llm-council` on PATH):
uv tool install "llm-council-core[mcp,secure]"

# Or with pip:
pip install "llm-council-core[mcp,secure]"

For core library only (no MCP server):

pip install llm-council-core

Why [mcp,secure] and not just [mcp]? The secure extra provides keychain support. Without it, llm-council setup-key is unavailable AND a key already stored in your keychain is silently ignored — the server starts but council_health_check reports api_key_configured: false. Include both extras unless you only ever use environment variables.

Setup

1. Choose Your Gateway

The council supports three gateway options for accessing LLMs:

| Gateway | Best For | API Keys Needed | |---------|----------|-----------------| | OpenRouter (default) | Easiest setup, 100+ models via single key | OPENROUTER_API_KEY | | Requesty | BYOK mode, analytics, cost tracking | REQUESTY_API_KEY + provider keys | | Direct | Maximum control, direct provider APIs | Provider keys (Anthropic, OpenAI, Google) |

Quick Start (OpenRouter):

# Sign up at openrouter.ai and get your API key
export OPENROUTER_API_KEY="sk-or-v1-..."

Direct Provider Access:

# Use your existing provider API keys directly
export ANTHROPIC_API_KEY="sk-ant-..."
export OPENAI_API_KEY="sk-..."
export GOOGLE_API_KEY="..."
export LLM_COUNCIL_DEFAULT_GATEWAY=direct

Requesty with BYOK:

export REQUESTY_API_KEY="..."
export ANTHROPIC_API_KEY="sk-ant-..."  # Your own key, routed through Requesty
export LLM_COUNCIL_DEFAULT_GATEWAY=requesty

2. Store Your API Keys Securely

Choose one of these options (in order of recommendation):

Option A: System Keychain (Most Secure)

Store keys encrypted in your OS keychain:

# Install with keychain support
pip install "llm-council-core[mcp,secure]"

# Store key securely (prompts for key, no echo)
llm-council setup-key

# For CI/CD automation, pipe from stdin:
echo "$OPENROUTER_API_KEY" | llm-council setup-key --stdin

Option B: Environment Variables

Set in your shell profile (~/.zshrc, ~/.bashrc):

# OpenRouter (default gateway)
export OPENROUTER_API_KEY="sk-or-v1-..."

# Or use direct provider APIs
export ANTHROPIC_API_KEY="sk-ant-..."
export OPENAI_API_KEY="sk-..."
export GOOGLE_API_KEY="..."

Option C: Environment File

Create a .env file (ensure it's in .gitignore):

# For OpenRouter
echo "OPENROUTER_API_KEY=sk-or-v1-..." > .env

# Or for direct APIs
cat > .env << 'EOF'
ANTHROPIC_API_KEY=sk-ant-...
OPENAI_API_KEY=sk-...
GOOGLE_API_KEY=...
LLM_COUNCIL_DEFAULT_GATEWAY=direct
EOF

Security Note: Never put API keys in command-line arguments or JSON config files that might be committed to version control.

3. Customize Models (Optional)

You can customize which models participate in the council using three methods (in priority order):

Option 1: Environment Variables (Recommended)

# Comma-separated list of council models
export LLM_COUNCIL_MODELS="openai/gpt-4,anthropic/claude-3-opus,google/gemini-pro"

# Chairman model (synthesizes final response)
export LLM_COUNCIL_CHAIRMAN="anthropic/claude-3-opus"

# Skip chairman synthesis, return the top-ranked Stage 1 response directly
# (cheaper/faster, but NEVER enable this for council-verify/council-gate --
# no verdict is computed; see docs/guides/verify.md)
export LLM_COUNCIL_CHAIRMAN_DISABLED=false

Option 2: YAML Configuration (Recommended)

Create llm_council.yaml in your project root or ~/.config/llm-council/llm_council.yaml:

council:
  # Tier configuration (ADR-022)
  tiers:
    default: high
    pools:
      quick:
        models:
          - openai/gpt-4o-mini
          - anthropic/claude-3-5-haiku-20241022
        timeout_seconds: 30
      balanced:
        models:
          - openai/gpt-4o
          - anthropic/claude-3-5-sonnet-20241022
        timeout_seconds: 90
      high:
        models:
          - openai/gpt-4o
          - anthropic/claude-opus-4-7
          - google/gemini-3-pro
        timeout_seconds: 180

  # Triage configuration (ADR-020)
  triage:
    enabled: false
    wildcard:
      enabled: true
    prompt_optimization:
      enabled: true

  # Gateway configuration (ADR-023, ADR-025a)
  gateways:
    default: openrouter
    fallback:
      enabled: true
      chain: [openrouter, ollama]  # Can use Ollama as fallback

    # Provider-specific configuration
    providers:
      ollama:
        enabled: true
        base_url: http://localhost:11434
        timeout_seconds: 120.0
        hardware_profile: recommended  # minimum|recommended|professional|enterprise

      openrouter:
        enabled: true
        base_url: https://openrouter.ai/api/v1/chat/completions

      requesty:
        enabled: true
        base_url: https://router.requesty.ai/v1/chat/completions
        api_key: ${REQUESTY_API_KEY}

    # Per-gateway model-id translation (e.g. Requesty rejects OpenRouter's
    # ":free" suffix and uses provider-prefixed names for some models)
    model_name_map:
      requesty:
        "some/model:free": "some/model"

  # Webhook notifications (ADR-025a)
  webhooks:
    enabled: false  # Opt-in
    timeout_seconds: 5.0
    max_retries: 3
    https_only: true
    default_events:
      - council.complete
      - council.error

  observability:
    log_escalations: true

Priority: YAML config > Environment variables > Defaults

Option 3: JSON Configuration (Legacy)

Create ~/.config/llm-council/config.json:

{
  "council_models": [
    "openai/gpt-4-turbo",
    "anthropic/claude-3-opus",
    "google/gemini-pro",
    "meta-llama/llama-3-70b-instruct"
  ],
  "chairman_model": "anthropic/claude-3-opus",
  "synthesis_mode": "consensus",
  "exclude_self_votes": true,
  "style_normalization": false,
  "max_reviewers": null
}

Option 4: Use Defaults

If you don't configure anything, these defaults are used:

• Council: GPT-5.1, Gemini 3 Pro, Claude Sonnet 4.5, Grok 4
• Chairman: Gemini 3 Pro
• Mode: consensus
• Self-vote exclusion: enabled

Finding Models:

• OpenRouter: openrouter.ai/models
• Anthropic: docs.anthropic.com/models
• OpenAI: platform.openai.com/docs/models
• Google: ai.google.dev/gemini-api/docs/models

Usage

With Claude Code

# First, store your API key securely (one-time setup)
llm-council setup-key

# Then add the MCP server (key is read from keychain or environment)
claude mcp add --transport stdio llm-council --scope user -- llm-council

Then in Claude Code:

Consult the LLM council about best practices for error handling

With Claude Desktop

First ensure your API key is available (via keychain, environment variable, or .env file).

Add to ~/Library/Application Support/Claude/claude_desktop_config.json:

{
  "mcpServers": {
    "llm-council": {
      "command": "llm-council"
    }
  }
}

Note: No env block needed—the key is resolved from your system keychain or environment automatically.

Client timeouts (high/reasoning tiers): deliberation can exceed MCP clients' default transport timeout (~60s). Set MCP_TIMEOUT (milliseconds) in your client config to at least the tier budget (high ≈ 180000, reasoning ≈ 600000) or the client cuts the connection mid-deliberation.

With Other MCP Clients

Any MCP client can use the server by running:

llm-council

Quality Benchmark (llm-council bench)

A golden dataset (bench/dataset/v1/, 20 items across coding/reasoning/factual/judgment) guards council quality against drift (ADR-048). Every run costs real API spend — it runs nightly/on-demand, never per-PR: