doc-search-mcp

A local MCP server that indexes PDFs, EPUBs, HTML and text files and makes them searchable via MCP tools. Designed for LLM coding agents that need to query large local document collections without uploading them to the cloud.

Features

• Indexes PDFs (with TOC detection), EPUBs, HTML, Markdown, and plain text
• Keyword search via FTS5/BM25 and semantic search via vector embeddings
• Hybrid search merging both with Reciprocal Rank Fusion
• Documents organized into named categories
• Same content at multiple paths shares one index record (deduplication by checksum)
• Async indexing, one job per file, with progress tracking via job IDs
• Re-running an index is cheap and idempotent: unchanged files cost one stat
• Startup health checks with per-file change detection
• Accessible over a local network via SSE transport

Install

git clone git@github.com:nburns/doc-search-mcp.git
cd doc-search-mcp
uv sync
uv run doc-search-mcp

Requires Python 3.10+. All dependencies are installed by default — no extras needed.

Connect to Claude Code

Start the server, then register it:

claude mcp add --transport sse --scope user doc-search http://localhost:8080/sse

Or, to share it with everyone working in a repo, commit a .mcp.json at that repo's root:

{
  "mcpServers": {
    "doc-search": {
      "type": "sse",
      "url": "http://localhost:8080/sse"
    }
  }
}

Use the machine's hostname or IP instead of localhost if connecting from another machine.

In a new Claude Code session, run /mcp to confirm the server is connected and its tools are listed.

No auth or TLS. Only expose on trusted local networks.

Configuration

Config file lives at ~/.doc-search/config.toml. All settings are optional — defaults work out of the box.

[server]
transport = "sse"           # sse | stdio
host = "0.0.0.0"
port = 8080

[storage]
backend = "sqlite"          # sqlite is the only implementation
db_path = "~/.doc-search/index.db"

[embeddings]
backend = "auto"            # auto | fastembed | sentence-transformers | ollama | none
model = "nomic-ai/nomic-embed-text-v1.5"
ollama_url = "http://localhost:11434"
# cache_dir = "~/.doc-search/models"   # unset = fastembed's temp dir, re-downloaded on reboot
enable_cpu_mem_arena = false           # true trades ~30GB memory spikes for throughput

[search]
default_mode = "auto"       # auto | keyword | semantic | hybrid
default_limit = 10
rerank = false

[chunking]
target_tokens = 400
overlap_tokens = 50
max_toc_depth = 2
max_file_size_mb = 500

[performance]
extraction_workers = 0      # 0 = auto (CPU count)
embedding_batch_size = 32
embedding_queue_size = 256
max_concurrent_jobs = 3
embedding_max_padded_tokens = 16384    # batch size x longest chunk; this is what bounds peak memory
extraction_timeout = 1800   # seconds per file; covers large OCR jobs

[startup]
check_on_startup = true     # read-only integrity check, runs after the port opens

[profiling]
enabled = true              # one memory sample per interval to the log below
interval_s = 60
path = "~/.doc-search/mem-profile.jsonl"

Environment variable overrides:

| Env var | Setting | |---------|---------| | DOC_SEARCH_DB | storage.db_path | | DOC_SEARCH_BACKEND | storage.backend | | DOC_SEARCH_POSTGRES_URL | storage.postgres_url (accepted, but no Postgres backend exists yet) | | DOC_SEARCH_EMBEDDER | embeddings.backend | | DOC_SEARCH_OLLAMA_URL | embeddings.ollama_url | | DOC_SEARCH_EMBED_CACHE | embeddings.cache_dir | | DOC_SEARCH_CPU_MEM_ARENA | embeddings.enable_cpu_mem_arena | | DOC_SEARCH_DEFAULT_MODE | search.default_mode | | DOC_SEARCH_PORT | server.port | | DOC_SEARCH_MEMPROFILE | profiling.enabled | | DOC_SEARCH_MEMPROFILE_INTERVAL | profiling.interval_s | | DOC_SEARCH_MEMPROFILE_PATH | profiling.path |

Embeddings

Backend auto-detection order:

1. DOC_SEARCH_EMBEDDER env var if set
2. CUDA available → sentence-transformers
3. Default → fastembed (CPU, no PyTorch required)
4. ollama — opt-in via config
5. none — keyword-only search, no vectors

Set backend = "none" to skip embedding entirely and use keyword search only. This makes indexing much faster and is a good default for technical documentation with precise terminology.

MCP Tools

Indexing

| Tool | Args | Description | |------|------|-------------| | index_path | path, category | Scan a file or directory and queue one job per file needing work. Returns the scan summary and its job IDs. Safe to re-run — unchanged files are skipped, so this is also how you re-index after edits. | | remove_document | path, category | Remove a path from the index. Also how you force a rebuild of one file. |

Search

| Tool | Args | Description | |------|------|-------------| | search | query, category, mode, limit, file_type, path_prefix | Search documents. | | search_in_document | query, path, category, limit | Search within one file. | | get_chunks | ids, category | Retrieve chunks by ID to expand context around a result. |

Documents

| Tool | Args | Description | |------|------|-------------| | list_documents | category | All indexed docs with metadata. | | list_categories | — | Categories with doc/chunk counts and last indexed timestamp. | | get_stats | category | DB size, document count, chunk count. |

Jobs

| Tool | Args | Description | |------|------|-------------| | get_job_status | job_id | Progress report for one file's job. The 8-character prefix is accepted anywhere a job_id is. | | list_jobs | status, path_prefix, limit | Live jobs merged with DB history. Filter by path_prefix to follow a single import. | | cancel_job | job_id | Cancel one queued or running job. There is no bulk cancel — restart the server to stop a large import. | | resume_job | job_id | Re-run a failed or cancelled job. Files already indexed are skipped. |

Maintenance

| Tool | Args | Description | |------|------|-------------| | check_index | category | Full health report — changed, missing, and unindexed files. | | acknowledge_warnings | ids | Suppress known warnings until next startup. | | get_config | — | Current config and active backend. Read-only; edit config.toml and restart to change it. | | inspect_pdf | path | Per-page extraction stats for one PDF: which pages carry text, character/word counts, TOC structure. Use to check coverage before or after indexing. |

Supported Formats

| Format | Notes | |--------|-------| | PDF | TOC-aware chunking; falls back to per-page if no TOC. Image-only PDFs are automatically OCR'd via tesseract if available. | | EPUB | Per-chapter extraction | | HTML | .html/.htm; one section per <h1>-<h6>. Scripts, styles and site furniture (nav, header, footer, aside, form) are dropped. Both suffixes store file_type = html | | Web archive | .webarchive (Safari "Save As → Web Archive"). The saved page is extracted exactly as HTML above; embedded images, stylesheets, scripts and iframes are skipped. Stores file_type = webarchive, and titles the document from the page's own <title> plus its originating host | | Markdown | Heading-aware splitting on ## | | Plain text | Paragraph block splitting |

OCR for image-only PDFs

If a PDF has no text layer, the extractor automatically falls back to OCR using tesseract. Install the system binary to enable it:

sudo apt install tesseract-ocr        # Debian/Ubuntu
brew install tesseract                 # macOS

Without tesseract, image-only PDFs fail with a clear error rather than silently producing an empty index entry.

Running it as a service

deploy/install.sh installs three system units (not user units — a user manager is torn down at logout, which took the server offline for 24 hours once):

| Unit | Role | |------|------| | doc-search-mcp.service | The server, running as the doc-search system account | | doc-search-sweep.timer | Fires every 30 minutes, Persistent=true | | doc-search-sweep.service | Oneshot: calls index_path over the watched tree, exits once jobs are queued |

To watch another tree, add an ExecStart= line to the sweep unit.

CLI helpers

mcp_call.py calls a single tool against a running server (DOC_SEARCH_URL, DOC_SEARCH_TIMEOUT override the target). The Makefile wraps the common ones:

make import SRC=~/books CATEGORY=reading   # index a tree (SRC=, never PATH= — Make inherits PATH)
make list CATEGORY=reading                 # list indexed documents
make jobs STATUS=running                   # list jobs; SRC=<prefix> filters to one import
make status JOB=1a2b3c4d                   # progress for one job (8-char prefix is fine)
make sweep                                 # run the incremental sweep now
make memprofile N=20                       # summarise the last N memory samples
make help                                  # all targets

Dev

uv run pytest                                # tests
uv run ruff check src                        # lint
uv run ruff format src                       # format
uv run --with mypy mypy src/doc_search_mcp/  # type check
uv run pyright                               # second type check

Both type checkers are clean; keep them that way.

Run the server from a checkout:

uv run doc-search-mcp              # production mode
uv run doc-search-mcp --reload     # auto-reload on source changes
uv run doc-search-mcp --port 9090  # override port

AGENTS.md covers the architecture: the job model, the shared embedding pipeline, and the memory findings behind the current defaults.