MCP Test Harness

<p align="center"> <a href="#mcp-test-harness"><strong>README</strong></a> · <a href="https://vaquarkhan.github.io/mcp-test-harness/guide/"><strong>Documentation</strong></a> · <a href="https://vaquarkhan.github.io/mcp-test-harness/examples.html"><strong>Examples</strong></a> · <a href="https://vaquarkhan.github.io/mcp-test-harness/features-deck.html"><strong>Features</strong></a> · <a href="https://vaquarkhan.github.io/mcp-test-harness/integrations.html"><strong>Integrations</strong></a> · <a href="https://vaquarkhan.github.io/mcp-test-harness/guide/multi-language.html"><strong>Multi-language</strong></a> · <a href="https://vaquarkhan.github.io/mcp-test-harness/reports/sample_mcp_test_report.html"><strong>Reports</strong></a> · <a href="LICENSE"><strong>License</strong></a> · <a href="SECURITY.md"><strong>Security</strong></a> </p> <p align="center"> <img src="docs/images/hero-banner.png" alt="MCP Test Harness - pytest-style testing for MCP servers" width="100%" /> </p>

Current release: 5.3.1 · Docker 5.3.1 · CHANGELOG

| Go to | Link | |-------|------| | Documentation handbook | https://vaquarkhan.github.io/mcp-test-harness/guide/handbook.html | | Docs tabs (Load · Chaos · Resiliency · Security · Reports) | https://vaquarkhan.github.io/mcp-test-harness/guide/ | | llms.txt (agent-readable) | https://vaquarkhan.github.io/mcp-test-harness/llms.txt | | Docs hub | https://vaquarkhan.github.io/mcp-test-harness/guide/ | | Examples / demos | https://vaquarkhan.github.io/mcp-test-harness/examples.html | | Sample scan report | https://vaquarkhan.github.io/mcp-test-harness/reports/sample_mcp_test_report.html | | Features deck | https://vaquarkhan.github.io/mcp-test-harness/features-deck.html | | Integrations & download totals | https://vaquarkhan.github.io/mcp-test-harness/integrations.html · JSON | | Multi-language suite | https://vaquarkhan.github.io/mcp-test-harness/guide/multi-language.html · suite repo · downloads | | MCP-Bastion (sister product — runtime security) | https://github.com/vaquarkhan/MCP-Bastion · handbook |

Latest Release - install from PyPI (pip install mcp-test-harness) or use the OCI image on GitHub Container Registry: ghcr.io/vaquarkhan/mcp-test-harness. Tags: latest and 5.3.1 (runtime, mcp-test entrypoint), dev and 5.3.1-dev (pytest + dev extras). Browse tags on GHCR · docker run --rm ghcr.io/vaquarkhan/mcp-test-harness:latest --version · docs/DOCKER.md · docs/RELEASING.md

Author: Vaquar Khan · License: MIT (NOTICE) · Cite: CITATION.cff · Sponsors: SPONSORS.md

---

For C-suite & directors

What it is: a CI gate for MCP servers - the connectors that let AI agents call tools, data, and APIs. Teams write deterministic tests once; every pull request proves the server still works before it ships.

Why it matters

| Business need | What the harness delivers | |---------------|---------------------------| | Ship AI features without silent breakage | Automated pass/fail on every PR - no manual Inspector click-through | | Audit & governance evidence | JUnit / SARIF / HTML reports, conformance badges, security payload packs | | Lower cost of quality | One tool for functional, regression, performance, and resiliency - not three vendors | | Trust the gate | 1080+ self-tests, 100% library coverage, and e2e dogfood in CI |

Not an LLM eval product. It does not score model answers. It proves your MCP server is correct, fast enough, and hardened enough for production. Pair with runtime security (MCP-Bastion) and IDE config scanning (mcp-shark). Positioning: docs/POSITIONING.md · comparison: docs/COMPARISON.md.

---

For architects & tech leads

Role in the stack: deterministic, code-first test automation between local MCP Inspector exploration and production. Fits stdio / SSE / HTTP transports; plugs into GitHub Actions (Marketplace Action), JUnit consumers, and Code Scanning (SARIF).

Architecture (one run)

mcp-test CLI → config → discover test_*.py → schedule (optional parallel)
  → real MCP session (lifecycle + transport) → assertions → reports

| Concern | Approach | |---------|----------| | Correctness | Protocol-aware fixtures (mcp_server), schema checks, snapshots | | Performance | assert_latency / assert_throughput / assert_load_phases / assert_stateless_throughput, baselines, SLO params | | Security / resiliency | Payload packs, chaos faults, experiment catalog + scorecard (RFC-005) | | Conformance signal | Levels + README badge via mcp-test try / conformance (RFC-002); stateless via mcp-test conformance stateless (RFC-006) | | Adoption speed | init, generate, record (RFC-001) - live session → suite | | Extensibility | Plugins (assertions, fixtures, reporters, transports) |

Deep dives: docs/ARCHITECTURE.md · docs/DECISIONS.md · docs/ENTERPRISE_GOVERNANCE.md · RFCs under docs/design/.

---

For developers

Fastest path

pip install mcp-test-harness
mcp-test init --server-command "python your_server.py"
# edit tests/test_*.py - then:
mcp-test --config mcp-test.yaml

Zero-config probe: mcp-test try --server-command "…". CI: Marketplace Action mcp-test-harness. Docs hub: docs/QUICK_START.md → docs/DEVELOPER_GUIDE.md → docs/CI_AND_REPORTS.md.

MCP Test Harness is a pytest-style framework for MCP servers: the mcp-test CLI discovers, runs, and reports tests automatically - replacing much of the manual validation you might do in the MCP Inspector. Repository: github.com/vaquarkhan/mcp-test-harness.

Documentation: Docs hub · Handbook (published) · QUICK_START · DEVELOPER_GUIDE · CI & reports · performance · comparison · discovery checklist. Agent map: llms.txt · llms-full.txt · skill skills/mcp-test-harness · user-repo template docs/templates/AGENTS.md. Companion runtime: Bastion handbook. Community: Issues for bugs; PRs for docs and examples.

---

What ships end-to-end

Everything below is implemented, tested, and documented in this repo (1080+ tests, 100% lib coverage gate, e2e dogfood in CI). Version pins live only in PyPI, CHANGELOG.md, and install/Docker tags - not in this map.

| Area | Feature | Where to start | |------|---------|----------------| | CLI | mcp-test - discover, run, filter (-k / -m), parallel, watch, reports; subcommands init, try, record, generate, experiment, conformance | CLI Reference · Quick Start | | Automation | Scaffold (init), zero-config probe (try), live record / generate, --watch re-runs, config-driven CI gates | Quick Start · Core Features | | Docker | OCI image on GHCR (ghcr.io/vaquarkhan/mcp-test-harness) - runtime + :dev; local Dockerfile | Docker · docs/DOCKER.md | | Hooks | pre-commit hook (mcp-test-try) + example config; plugin / discovery hooks for custom automation | examples/pre-commit-config.yaml · Plugins | | Diagnostics | MCP trace - per-test JSON-RPC timeline in HTML/JSON reports; stdio pollution hints | example_mcp_trace.md | | Resiliency | Chaos testing - @marker(tags=["chaos"], chaos_faults=[...]) (delay, 503, truncate, schema drift) | example_chaos_testing.md | | Resiliency | Experiment catalog - mcp-test experiment run --suite core with guardrails and scorecard (RFC-005) | docs/design/RFC-005-resiliency-experiments.md | | Conformance | Levels + badge - mcp-test try / conformance badge (RFC-002); stateless SEP-2575 via mcp-test conformance stateless (RFC-006); README shields.io seal | docs/design/RFC-002-conformance-levels.md · RFC-006 | | Productivity | mcp-test record (RFC-001) + generate - live calls → suite + snapshots; schema drafts | RFC-001 · example_generate_scaffold.md | | Marketplace | GitHub Action with conformance PR outputs | Marketplace · example_github_actions.md | | Platform QA | Tool **coverage m