Codebase Contextifier 9000

A Docker-based Model Context Protocol (MCP) server for semantic code search with AST-aware chunking, relationship tracking via Neo4j graph database, local LLM support, and incremental indexing.

Documentation

• 📚 Quick Start Guide - Get running in 5 minutes
• 🔧 Multi-Project Setup - Index multiple projects with shared backend
• ⚙️ Background Jobs - Job-based indexing for large codebases
• 👁️ File Watcher Guide - Real-time monitoring and auto-indexing
• 🔬 Research & Methodology - Deep dive into semantic code search
• 📖 Full Documentation - Complete docs directory

Table of Contents

• Features
• Architecture
  • Key Architectural Features
• Quick Start
  • Prerequisites
  • Two Deployment Options
  • Claude Desktop Configuration
  • Usage
• MCP Tools
  • Indexing Tools: index_repository, get_job_status, list_indexing_jobs, cancel_indexing_job
  • Search Tools: search_code, get_symbols
  • Graph Query Tools: find_usages, find_dependencies, query_graph
  • Dependency Tools: detect_dependencies, index_dependencies, list_indexed_dependencies
  • Status Tools: get_indexing_status, clear_index, get_watcher_status, health_check
• Supported Languages
• Configuration
  • Environment Variables
  • Recommended Embedding Models
• Performance
• Troubleshooting
• Development
• Architecture Details
  • AST-Aware Chunking
  • Incremental Indexing
  • Content-Addressable Storage
• Roadmap
• Research & References
• License
• Contributing
• Support

Features

• AST-Aware Chunking: Uses tree-sitter to respect function and class boundaries, maintaining semantic integrity
• Relationship Tracking: Neo4j graph database tracks function calls, imports, inheritance, and dependencies across your codebase
• External Dependency Mapping: Automatically creates placeholder nodes for external functions (WordPress, npm packages, etc.)
• Job-Based Indexing: Background indexing with progress tracking for large codebases
• On-Demand Container Spawning: Index any repository on your system without manual mounting
• Multi-Repository Search: Index and search across multiple projects with a shared backend
• Real-Time Updates: File system watcher automatically re-indexes changed files (optional)
• Local-First: All processing happens locally using Ollama for embeddings (no data leaves your machine)
• Polyglot Support: Supports 10+ programming languages including TypeScript, Python, PHP, Go, Rust, Java, C++, and more
• Incremental Indexing: Merkle tree-based change detection with 80%+ cache hit rates
• Production-Grade: Uses Qdrant vector database for sub-10ms search latency and Neo4j for relationship queries
• Dependency Knowledge Base: Special collection for indexing WordPress plugins, Composer packages, and npm modules
• Flexible Deployment: Per-project or centralized server deployment options
• MCP Integration: Works with Claude Desktop, Cursor, VS Code, and other MCP-compatible tools

Architecture

┌─────────────────────────────────────────────────────────────────┐
│  MCP Client (Claude Code, Claude Desktop, Cursor, etc.)         │
└──────────────────────────────┬──────────────────────────────────┘
                               │ MCP Protocol (stdio)
                               │
┌──────────────────────────────▼──────────────────────────────────┐
│  MCP Server Container (codebase-mcp-server)                     │
│  ┌──────────────────────────────────────────────────────────┐  │
│  │  FastMCP Server - Exposes MCP Tools:                      │  │
│  │  • index_repository (spawns indexer containers)           │  │
│  │  • search_code (semantic search across all repos)         │  │
│  │  • find_usages, find_dependencies (graph queries)         │  │
│  │  • detect_dependencies, index_dependencies                │  │
│  │  • get_job_status, list_indexing_jobs, cancel_job        │  │
│  │  • get_symbols, get_indexing_status, health_check        │  │
│  └──────────────┬───────────────────────────────────────────┘  │
│                 │                                                │
│                 │ Spawns via Docker Socket                       │
│                 ▼                                                │
│  ┌─────────────────────────────────────────────────────┐       │
│  │  On-Demand Indexer Containers (ephemeral)           │       │
│  │  • Mounts any host directory                        │       │
│  │  • AST-aware chunking with tree-sitter              │       │
│  │  • Extracts relationships (CALLS, IMPORTS, etc.)    │       │
│  │  • Generates embeddings via Ollama                  │       │
│  │  • Updates shared Qdrant & Neo4j databases          │       │
│  │  • Reports progress back to MCP server              │       │
│  └──────────────────────┬──────────────────────────────┘       │
└─────────────────────────┼──────────────────────────────────────┘
                          │
          ┌───────────────┴───────────────────┐
          │                                   │
   ┌──────▼──────┐              ┌────────────▼────────┐
   │   Qdrant    │              │      Neo4j          │
   │  Container  │              │    Container        │
   │  (Vectors)  │              │  (Relationships)    │
   └──────┬──────┘              └──────┬──────────────┘
          │                            │
   ┌──────▼────────────────────────────▼───────────┐
   │  Persistent Docker Volumes:                   │
   │  • qdrant_data (vector DB)                    │
   │  • neo4j_data (graph DB)                      │
   │  • index_data (merkle trees)                  │
   │  • cache_data (embeddings cache)              │
   └───────────────────────────────────────────────┘

   ┌────────────────────────┐
   │  Ollama (Host)         │
   │  Embedding Model       │
   └────────────────────────┘

Key Architectural Features

• Dual Database Architecture: Qdrant for semantic vector search, Neo4j for relationship graph queries
• Container Orchestration: MCP server spawns lightweight indexer containers on-demand via Docker socket
• Multi-Repository Support: Each repository gets its own merkle tree state, but shares the vector & graph databases
• Shared Backend: All projects use the same Qdrant & Neo4j instances, enabling cross-repository search and relationship tracking
• Job-Based Processing: Background jobs with progress tracking for large codebases
• Content-Addressable Caching: Embeddings are cached by content hash, shared across all repositories
• Relationship Extraction: AST-based extraction of CALLS, IMPORTS, EXTENDS, and IMPLEMENTS relationships
• External Dependency Tracking: Automatic creation of placeholder nodes for unresolved function calls

Quick Start

See QUICK_START.md for detailed setup instructions.

Prerequisites

1. Docker Desktop (or Docker + Docker Compose)
2. Ollama running locally with an embedding model:

   # Install Ollama: https://ollama.ai
   
   # Recommended: Google's Gemma embedding model (best quality)
   ollama pull embeddinggemma:latest
   
   # Alternative: Nomic Embed (faster, smaller)
   ollama pull nomic-embed-text
   

Two Deployment Options

Option A: Centralized Server (Recommended)

Best for: Indexing from the MCP server, querying across all repositories

# 1. Start the backend
cd codebase-contextifier-9000
docker-compose up -d

# 2. Configure Claude Desktop (see below)

# 3. Index any repository
# In Claude: "Index the repository at /Users/me/projects/my-app"

Option B: Per-Project Setup

Best for: Each project manages its own indexing

# 1. Start shared backend (once)
cd codebase-contextifier-9000
docker-compose up -d

# 2. Copy .mcp.json to each project
cp .mcp.json.template ~/projects/my-app/.mcp.json

# 3. Open project in Claude Code
cd ~/projects/my-app
claude-code .

See MULTI_PROJECT_SETUP.md for details.

Claude Desktop Configuration

For Centralized Server (Option A):

Add to ~/Library/Application Support/Claude/claude_desktop_config.json (macOS) or %APPDATA%\Claude\claude_desktop_config.json (Windows):

{
  "mcpServers": {
    "codebase-contextifier": {
      "command": "docker",
      "args": [
        "exec",
        "-i",
        "codebase-mcp-server",
        "python",
        "-m",
        "src.server"
      ]
    }
  }
}

For Per-Project Setup (Option B):

Just copy .mcp.json.template to your project directory - no manual configuration needed!

Usage

Once configured, you can use these tools in Claude Desktop or Claude Code:

Index any repository on your system:

Claude, index the repository at /Users/me/projects/my-app

The system spawns a container, indexes the repository in the background, and reports progress.

Monitor indexing progress:

Claude, show me the status of job abc123

Search for code across all indexed repositories:

Claude, search for "authentication logic" in the codebase

Search with filters:

Claude, search for "error handling" filtering by language=python and repo_name=my-api

Extract symbols from a file:

Claude, get all functions from /workspace/src/utils.py

Find all usages of a function (graph query):

Claude, find all places where authenticate_user is called

Find dependencies of a function (graph query):

Claude, show me all functions that processPayment depends on

Detect and index external dependencies:

Claude, detect available WordPress plugins in this project
Claude, index the woocommerce plugin into the knowledge base

Check system status:

Claude, show me the indexing status and list all jobs

MCP Tools

Indexing Tools

index_repository

Index a repository from any directory on your host machine by spawning a lightweight indexer container.

Parameters:

• host_path (string, required): Absolute path on host machine to repository (e.g., /Users/me/projects/my-app)
• repo_name (string, optional): Unique identifier for this repository (defaults to directory name)
• incremental (bool): Use incremental indexing to only re-index changed files (default: true)
• exclude_patterns (string, optional): Comma-separated glob patterns to exclude (e.g., "node_modules/*,dist/*")

Returns:

{
  "success": true,
  "job_id": "abc123def456",
  "repo_name": "my-app",
  "status": "queued",
  "message": "Background indexing started for 'my-app'"
}

Example:

# Index a WordPress site, excluding plugins and uploads
await index_repository(
    host_path="/Users/me/sites/my-wordpress",
    repo_name="my-wordpress",
    exclude_patterns="wp-content/plugins/*,wp-content/uploads/*,wp-includes/*"
)

get_job_status

Get the status and progress of an indexing job.

Parameters:

• job_id (string, required): Job identifier returned from index_repository

Returns:

{
  "success": true,
  "job_id": "abc123def456",
  "repo_name": "my-app",
  "repo_path": "/Users/me/projects/my-app",
  "status": "running",
  "created_at"