LLM — Unified AI Chat Completions with Streaming

MCP tool providing unified LLM access: local models, OpenRouter, and direct API connections — all with native tool-calling and streaming support.

One tool. Every LLM. Local or cloud. Tool-calling built in. Run local models offline with full privacy, or connect to OpenRouter/OpenAI/Anthropic for frontier capabilities — same interface, same streaming, same tool-calling support.

---

Benefits

1. One Interface, Every Provider

Not locked to one backend. Local models (transformers, Ollama, llama.cpp), OpenRouter (100+ cloud models), and direct API connections (OpenAI, Anthropic, Google) all through the same operations. Switch providers by changing one parameter.

2. Streaming First

Not request-response — real-time tokens. Native SSE streaming delivers tokens as they generate, to Android apps, Chrome extensions, and desktop UIs. The tool returns a stream_id immediately; the server pushes deltas over SSE until completion.

3. Autonomous Tool-Calling

Not just text generation — AI agents. Models (local via Ollama, or cloud) can call MCP tools automatically during generation. Query databases, control browsers, show UI popups — all within a single completion request.

4. Multimodal Support

Not text-only — images too. Vision-language models (VL) accept images alongside text. Send a screenshot and ask what's on screen. Works with both local VL models and cloud multimodal APIs.

5. Complete Privacy When You Need It

Not cloud-dependent. Local models run on your hardware. Data never leaves your machine. Perfect for sensitive work, proprietary code, or air-gapped environments. Use cloud when you need frontier capabilities; use local when you need privacy.

---

Why This Tool Changes LLM Access

Cloud APIs are powerful but limited. OpenAI, Anthropic, Google — great models, but you pay per token, data leaves your network, and you're dependent on their availability.

Existing local LLM tools are basic. Ollama, LM Studio, llama.cpp — good for chat, but limited tool-calling, no unified MCP integration, no streaming to multiple clients.

This tool bridges both worlds. One unified interface for every LLM backend, with streaming, tool-calling, and multimodal support regardless of whether the model runs locally or in the cloud. Named endpoints in settings let you define backends once and reference them by name.

Tool-calling is cloud-only. OpenAI's function calling, Anthropic's tool use — powerful features locked to cloud APIs. Local models can't compete.

This tool solves all of that.

Run models locally with:

• Full tool-calling support (OpenAI-compatible format)
• Automatic MCP tool integration
• Hardware optimization (CUDA/CPU, quantization)
• OpenRouter-compatible interface
• Zero cloud dependencies

Your AI can now be an autonomous agent, calling tools and making decisions, all running on your hardware.

---

Real-World Story: The Air-Gapped Environment

The Problem:

A defense contractor needed AI assistance for code analysis. Security requirements:

• No internet connection allowed
• No data can leave the facility
• No cloud API access
• Must work on classified networks

Standard solution: None. All major AI assistants require internet.

With This Tool:

# 1. Check hardware capabilities
hardware = hardware_info()
# Shows: CUDA available, 24GB VRAM, RTX 4090

# 2. List installed models
models = list_installed_models()
# Shows: Qwen2.5-7B-Instruct (7GB), Llama-3.1-8B (8GB)

# 3. Run local AI with tool-calling
response = chat_completion(
    model="Qwen/Qwen2.5-7B-Instruct",
    messages=[
        {"role": "user", "content": "Analyze the security of auth.py"}
    ],
    tools=[
        {
            "type": "function",
            "function": {
                "name": "read_file",
                "description": "Read file contents",
                "parameters": {
                    "type": "object",
                    "properties": {
                        "path": {"type": "string"}
                    }
                }
            }
        }
    ],
    auto_execute_tools=True  # Autonomous mode
)

# Model reads file, analyzes code, returns findings
# All local, no internet, no cloud APIs

Result:

• Code analysis working in air-gapped environment
• AI can read files, analyze patterns, suggest improvements
• Zero data exfiltration risk
• Unlimited usage, no costs
• Full audit trail of all operations

The kicker: Same setup now powers their entire development workflow. Code review, documentation generation, test creation — all local, all private, all autonomous.

---

The Complete Feature Set

Hardware Detection

Check Capabilities:

# Get hardware info
info = hardware_info()

# Returns:
# {
#   "cuda_available": true,
#   "cuda_version": "12.1",
#   "gpu_name": "NVIDIA RTX 4090",
#   "gpu_memory_gb": 24,
#   "cpu_cores": 16,
#   "ram_gb": 64,
#   "recommended_models": ["7B", "13B", "30B"]
# }

Why hardware detection matters: Automatically selects appropriate models and quantization based on your hardware. No manual configuration needed.

Model Management

List Installed Models:

# See what's already downloaded
models = list_installed_models()

# Returns list of cached models:
# [
#   {
#     "id": "Qwen/Qwen2.5-7B-Instruct",
#     "size_gb": 7.2,
#     "type": "text-generation",
#     "supports_tools": true,
#     "cache_path": "~/.cache/huggingface/..."
#   }
# ]

Model Information:

# Get details about a specific model
info = model_info(model="Qwen/Qwen2.5-7B-Instruct")

# Returns:
# {
#   "name": "Qwen2.5-7B-Instruct",
#   "size_gb": 7.2,
#   "context_length": 32768,
#   "supports_tools": true,
#   "supports_vision": false,
#   "quantization_options": ["4bit", "8bit", "none"],
#   "recommended_vram_gb": 8
# }

Why model management matters: Know what's installed, what's possible, what will fit in your VRAM. Make informed decisions about which models to use.

Chat Completions

Basic Chat:

# Simple text generation
response = chat_completion(
    model="Qwen/Qwen2.5-7B-Instruct",
    messages=[
        {"role": "user", "content": "Explain quantum computing"}
    ]
)

Multi-Turn Conversations:

# Maintain conversation history
response = chat_completion(
    model="Qwen/Qwen2.5-7B-Instruct",
    messages=[
        {"role": "system", "content": "You are a helpful coding assistant"},
        {"role": "user", "content": "Write a Python function to sort a list"},
        {"role": "assistant", "content": "Here's a sorting function..."},
        {"role": "user", "content": "Now optimize it for large lists"}
    ]
)

Temperature and Sampling:

# Control randomness and creativity
response = chat_completion(
    model="Qwen/Qwen2.5-7B-Instruct",
    messages=[{"role": "user", "content": "Write a creative story"}],
    temperature=0.9,  # More creative
    top_p=0.95,
    max_tokens=1000
)

Tool-Calling (Autonomous Agents):

# AI can call MCP tools automatically
response = chat_completion(
    model="Qwen/Qwen2.5-7B-Instruct",
    messages=[
        {"role": "user", "content": "Show me all browser tabs and save to database"}
    ],
    tools=[
        {
            "type": "function",
            "function": {
                "name": "browser_list_tabs",
                "description": "Get all open browser tabs",
                "parameters": {"type": "object", "properties": {}}
            }
        },
        {
            "type": "function",
            "function": {
                "name": "sqlite_execute",
                "description": "Execute SQL query",
                "parameters": {
                    "type": "object",
                    "properties": {
                        "sql": {"type": "string"},
                        "database": {"type": "string"}
                    }
                }
            }
        }
    ],
    auto_execute_tools=True  # Autonomous mode
)

# Model will:
# 1. Call browser_list_tabs
# 2. Process results
# 3. Call sqlite_execute to save data
# 4. Return summary

Why tool-calling matters: Local AI becomes an autonomous agent. Not just answering questions — taking actions, calling tools, completing complex workflows.

Hardware Optimization

Automatic Quantization:

# Let tool decide best quantization
response = chat_completion(
    model="Qwen/Qwen2.5-7B-Instruct",
    messages=[{"role": "user", "content": "Hello"}],
    quantization="auto"  # Chooses 4bit/8bit/none based on VRAM
)

Device Selection:

# Control where model runs
response = chat_completion(
    model="Qwen/Qwen2.5-7B-Instruct",
    messages=[{"role": "user", "content": "Hello"}],
    device="cuda"  # or "cpu" or "auto"
)

Why optimization matters: Run larger models on limited hardware. 7B model in 4-bit quantization fits in 4GB VRAM. 13B model fits in 8GB. Automatic selection ensures best performance for your hardware.

---

Usage Examples

Hardware Info

{
  "input": {
    "operation": "hardware_info",
    "tool_unlock_token": "YOUR_TOKEN"
  }
}

List Installed Models

{
  "input": {
    "operation": "list_installed_models",
    "tool_unlock_token": "YOUR_TOKEN"
  }
}

Model Info

{
  "input": {
    "operation": "model_info",
    "model": "Qwen/Qwen2.5-7B-Instruct",
    "tool_unlock_token": "YOUR_TOKEN"
  }
}

Chat Completion

{
  "input": {
    "operation": "chat_completion",
    "model": "Qwen/Qwen2.5-7B-Instruct",
    "messages": [
      {"role": "user", "content": "Explain quantum computing"}
    ],
    "temperature": 0.7,
    "max_tokens": 1000,
    "tool_unlock_token": "YOUR_TOKEN"
  }
}

Chat with Tool-Calling

{
  "input": {
    "operation": "chat_completion",
    "model": "Qwen/Qwen2.5-7B-Instruct",
    "messages": [
      {"role": "user", "content": "Get browser tabs and save to database"}
    ],
    "tools": [
      {
        "type": "function",
        "function": {
          "name": "browser_list_tabs",
          "description": "Get all open browser tabs",
          "parameters": {"type": "object", "properties": {}}
        }
      }
    ],
    "auto_execute_tools": true,
    "tool_unlock_token": "YOUR_TOKEN"
  }
}

---

Technical Architecture

Supported Models

Text Generation:

• Small (0.5-3B): Qwen2.5-0.5B, Phi-3-mini, Gemma-2-2B
• Medium (7-13B): Llama-3.1-8B, Qwen2.5-7B, Mistral-7B
• Large (30B+): Llama-3.1-70B, Qwen2.5-32B

Tool-Calling Support:

• Qwen2.5-Instruct family (recommended)
• Llama-3.x Instruct
• Any model with function-calling training

Multi-Modal (Future):

• LLaVA (vision + text)
• Qwen2-VL (vision + text)
• KOSMOS-2 (vision + text)

Hardware Requirements

Minimum (CPU-only):

• 8GB RAM
• 4-core CPU
• Small models only (0.5-3B)

Recommended (GPU):

• 8GB+ VRAM (NVIDIA GPU)
• CUDA 11.8+
• Medium models (7-13B) with quantization

Optimal (High-end GPU):

• 24GB+ VRAM
• CUDA 12.1+
• Large models (30B+) or unquantized medium models

Quantization

4-bit (BitsAndBytes):

• ~4GB VRAM for 7B model
• Minimal quality loss
• 4x memory reduction

8-bit:

• ~8GB VRAM for 7B model
• Negligible quality loss
• 2x memory reduction

None (FP16/BF16):

• ~14GB VRAM for 7B model
• Full quality
• Maximum memory usage

Model Storage

Cache Location:

• Linux/Mac: ~/.cache/huggingface/hub/
• Windows: %USERPROFILE%\.cache\huggingface\hub\
• Override: HF_HOME environment variable

**Stor