agent-harness
Test and evaluation harness for AI agents — scenario suites, deterministic replay, regression diffing, cost and latency budgets
Install / Use
npx skills add borghei/Claude-Skills --skill agent-harnessInstalls into whichever agent you are using.
SKILL.md
Installable skill definition
Quality Score
Category
AI & Machine LearningSupported Platforms
Our assessment of agent-harness
agent-harness scores 91/100 on our quality scale, 337th of 950 AI & Machine Learning skills we index (top 36%).
Its SKILL.md is 12 KB long, well organised into 19 sections with 3 code examples: a thorough specification that gives an agent plenty to work with.
It has 817 GitHub stars, a meaningful sign that others use it.
Maintenance, license and trust
- The repository was last updated about 2 months ago, so agent-harness is actively maintained.
- No license is declared. By default that means all rights are reserved: you can read it, but reusing or redistributing it is not clearly permitted. Ask the author before building on it commercially.
- Its trust signals score 88/100, with 1 caution from licensing, adoption, age or documentation. These come from repository metadata, not a code audit — read the skill file before letting an agent act on it.
Safety scan
No issues foundOur scan of the whole file found no instruction hijacking, hidden characters, credential access, data exfiltration or destructive commands. An AI review of the same text found nothing harmful.
AI review by kimi-k2.7-code on 2026-10-07. Automated pattern scan on 2026-10-07. It catches known dangerous patterns, not every risk — read a skill before letting an agent act on it.
agent-harness compared with similar skills
All 4 of these similar skills score higher than agent-harness; compare them before choosing.
| Skill | Score | Stars | Updated | Format |
|---|---|---|---|---|
| agent-harness (this skill)by borghei | 91 | 817 | 57d ago | SKILL.md |
| claude-memby thedotmack | 100 | 117.9k | today | CLAUDE.md |
| Understand-Anythingby Egonex-AI | 100 | 93.0k | 2d ago | CLAUDE.md |
| headroomby headroomlabs-ai | 100 | 79.9k | today | CLAUDE.md |
| CowAgentby zhayujie | 100 | 48.0k | today | CLAUDE.md |
Frequently asked questions
- How do I install agent-harness?
- Run
npx skills add borghei/Claude-Skills --skill agent-harness. The install tabs above show the steps for each supported agent. - Which AI agents does agent-harness work with?
- It is written for Universal, as a SKILL.md file. Other agents that read the same format can often use it too.
- Is agent-harness safe to use?
- Our scan of the whole file found no instruction hijacking, hidden characters, credential access, data exfiltration or destructive commands. An AI review of the same text found nothing harmful. It declares no license and scores 88/100 on trust signals. Skills are instructions an agent will follow, so read the file before installing it and do not approve commands you do not understand.
- Is agent-harness still maintained?
- The repository was last updated about 2 months ago, so agent-harness is actively maintained.
Skill content
View source on GitHubname: agent-harness description: > Test and evaluation harness for AI agents — scenario suites, deterministic replay, regression diffing, cost and latency budgets. Use when agent quality is vibe-checked, before shipping a prompt or model change, or when evals drift. license: MIT + Commons Clause metadata: version: 1.0.0 author: borghei category: engineering domain: agent-evaluation updated: 2026-07-21 tags: [agent-eval, regression-testing, harness, replay, llm-testing]
Agent Harness
Most agents ship on vibes: someone tries eight prompts, the output looks good, it goes to production, and the next prompt tweak silently breaks a refusal nobody re-tested. This skill builds the harness around an agent so its behaviour becomes measurable — scenario suites with structural assertions, deterministic replay of recorded tool calls, paired regression diffing across prompt and model changes, and per-scenario cost and latency budgets. The tools here score an agent; they never invoke one, so they run offline on every commit.
When to use this skill
- An agent is going to production and the only quality evidence is manual spot-checking
- A prompt, tool schema, or model version is changing and you need to know what broke
- Two model or configuration options need a defensible comparison, not a demo
- An incident happened and you need the behaviour encoded as a permanent regression test
- Agent cost or latency is climbing across releases and nobody can point to when
- An existing eval suite reports a healthy pass rate that nobody trusts
Inputs the skill expects
- The agent's tool inventory — names, arguments, and which tools are irreversible
- Recorded transcripts per scenario: tool calls, final output, turns, latency, cost, error state
- The behavioural rules the agent must hold (refusals, escalation triggers, policy boundaries)
- Known failure history — past incidents, customer complaints, internal bug reports
- Current cost and latency expectations per interaction
- The release gate that consumes the result (CI job, review checklist, launch review)
Clarify First
Before building the harness, confirm these inputs. If any is unknown or vague, ASK — do not assume:
- [ ] Which agent actions are irreversible — determines which scenarios need
tool_not_calledassertions atcriticalseverity, and what the release gate blocks on - [ ] Whether transcripts are already recorded — decides whether workflow 1 starts from replay or from an instrumentation task first
- [ ] What the suite gates — a CI blocking check, a nightly report, or a one-off comparison; changes suite size, runtime budget, and severity strictness
- [ ] The known failure modes — past incidents seed the adversarial and refusal buckets, which is where regressions actually hide
Stop rule: ask only the 2-3 that most change the output. If the user says "just draft it," proceed and list your assumptions at the top of the artifact.
Workflows
Workflow 1 — Stand up a scenario suite and score a run
- Enumerate the agent's irreversible actions; each one gets a refusal scenario.
- Draft 20-30 scenarios across all six buckets (happy, boundary, refusal,
adversarial, failure-recovery, ambiguity) using
assets/scenario_authoring_checklist.md. Structural assertions first — tool called / not called / order / arguments — text assertions only on domain tokens. - Declare suite-wide
defaultsfor latency, cost, and turn ceilings so every scenario is budgeted without repeating yourself. - Record one transcript per scenario, scrubbing PII at record time, and stamp
the run with
modelandprompt_sha. - Score the run and read critical failures before the pass rate.
python3 engineering/agent-harness/scripts/scenario_runner.py \
--suite engineering/agent-harness/assets/sample_suite.json \
--transcripts engineering/agent-harness/assets/sample_transcripts_baseline.json \
--strict-critical
Workflow 2 — Gate a prompt or model change on a paired regression diff
- Score the baseline and the candidate with the same suite file, saving both as JSON reports.
- Diff them. Read regressions and budget drift before the aggregate rate.
- Triage every regression: intended trade, real defect, or flaky scenario (re-run the flipped scenario five times to tell the last two apart).
- Record the decision in
assets/eval_report_template.mdand promote the accepted candidate report to the new baseline.
python3 engineering/agent-harness/scripts/scenario_runner.py \
--suite engineering/agent-harness/assets/sample_suite.json \
--transcripts engineering/agent-harness/assets/sample_transcripts_candidate.json \
--format json > /tmp/candidate.report.json
python3 engineering/agent-harness/scripts/eval_diff.py \
--baseline engineering/agent-harness/assets/sample_baseline_report.json \
--candidate /tmp/candidate.report.json \
--fail-on-regression --drift-threshold 0.15
The shipped sample data demonstrates the core lesson: both runs score 83.3%, and the candidate contains a critical prompt-injection regression. A gate on pass rate ships it; the paired diff catches it.
Workflow 3 — Establish cost and latency budgets, then track drift
- Take the last release's accepted run as the reference.
- Set per-scenario latency at p95 × 1.3, cost at median × 1.5, and the turn
ceiling at observed max + 2. Put them in the suite
defaults, overriding only where a scenario is legitimately expensive. - Score the current run; budget breaches surface as
minorassertions, so they report without blocking. - Diff against the reference with a tight drift threshold to catch the slow bleed that stays inside budget.
python3 engineering/agent-harness/scripts/eval_diff.py \
--baseline engineering/agent-harness/assets/sample_baseline_report.json \
--candidate engineering/agent-harness/assets/sample_candidate_report.json \
--drift-threshold 0.10 --format json
Decision frameworks
Which assertion type to reach for
| Need | Use | Durability |
|------|-----|------------|
| The agent must take an action | tool_called, tool_call_order | [PROVEN] Exact; survives rewording |
| The agent must NOT take an action | tool_not_called | [PROVEN] The single highest-value assertion in any agent suite |
| The action must use the right data | tool_arg_equals | [PROVEN] Catches the right tool with wrong arguments |
| Structured output correctness | json_field_equals | [PROVEN] Exact when the agent has a JSON mode |
| A required domain fact appears | output_contains on an ID, number, or policy name | [RECOMMENDED] Stable if you never quote sentences |
| A forbidden phrase must not appear | output_not_contains | [RECOMMENDED] Good for injection and leak checks |
| Tone, helpfulness, faithfulness | Model-graded rubric (outside this harness) | [EXPERIMENTAL] Noisy and drifts with the judge; calibrate against human labels first, and never gate on it alone |
Severity, and what each one gates
| Severity | Covers | Gate |
|----------|--------|------|
| critical | Safety, money movement, data loss, refusals that must hold | Blocks on a single failure (--strict-critical) |
| major | Task correctness — the user did not get what they asked for | Blocks below the pass-rate floor (--fail-under) |
| minor | Budgets, verbosity, style | Reported; never blocks |
Can I trust this diff?
| Discordant scenarios (flipped either way) | Read it as |
|-------------------------------------------|------------|
| 0 | No behavioural change detected at this suite's resolution |
| 1-5 | Read the individual scenarios; the p-value has no power here |
| 6-24 | Exact McNemar p is meaningful; eval_diff.py reports it |
| 25+ | Both the p-value and the aggregate rate movement are informative |
A single critical regression is actionable at n = 1. Significance testing is
for aggregate movement, never for safety failures.
Anti-Patterns
Gating on the aggregate pass rate
Mistake: The release check is "pass rate ≥ 90%," and everything else is advisory. Why it happens: One number is easy to put in a dashboard and easy to explain to leadership, and it genuinely looks like the summary statistic. Instead: Gate on critical-severity failures and on the paired per-scenario diff. The pass rate is the last number you read, always with its confidence interval — at 30 scenarios that interval is ±13 points, which cannot resolve the regressions you care about. The sample data here shows two runs at an identical 83.3% where one refunds money on an injected instruction.
Asserting on sentences instead of structure
Mistake: output_contains: "I've issued your refund of $49.00 and it should arrive in 3-5 business days".
Why it happens: It is the fastest thing to do — copy the good output into the assertion and move on.
Instead: Assert on the tool call (issue_refund with order_id=A-10041) and on a domain token in the text ("refund", the order ID). Structural assertions do not break when the model rewords, so the suite keeps signal across model upgrades instead of generating a wall of false failures that trains the team to ignore it.
Only testing what the agent should do
Mistake: Every scenario is a happy path; the suite has no tool_not_called assertions.
Why it happens: Suites get written from the product spec, and specs describe intended behaviour, not forbidden behaviour.
Instead: For every irreversible action the agent can take, write a scenario where taking it is wrong. Refusal and adversarial scenarios are where prompt changes actually regress, because a change that makes an agent more capable usually makes it more eager. Target roughly 35% of the suite across refusal and adversarial buckets.
Tuning the prompt until the suite goes green
Mistake: Iterating on the prompt with the full suite visible until every scenario passes. Why it happens: It feels like the tight feedback loop that good engineering is supposed to have. Instead: Hold out 20% of scenarios and never look at them while iterating; run them only at the gate. Thirty scenarios is a small enough surface to overfit in an afternoon, producing an agent that passes the suite and fails users.
Chasing regressions without a noise floor
Mistake: Four scenarios flip after a prompt edit, so the team spends two days finding the cause. Why it happens: Nobody ever ran the identical configuration twice, so run-to-run variance is unmeasured and every flip looks causal. Instead: Before trusting any diff, score the same configuration twice and diff it against itself. That flip count is your noise floor. Then reduce it — temperature 0 where the product allows, replayed tool results rather than live backends, and re-runs of flipped scenarios to separate flaky from real.
Files
| File | Purpose |
|------|---------|
| scripts/scenario_runner.py | Runs a JSON scenario suite against recorded transcripts; reports pass/fail per assertion with severity, budget checks, and CI exit codes |
| scripts/eval_diff.py | Diffs two runs into regressed/fixed/stable, with Wilson intervals, exact McNemar on discordant pairs, and cost/latency drift |
| references/scenario-and-fixture-design.md | The six scenario buckets, replay modes, fixture recording rules, assertion tiers, suite sizing |
| references/eval-methodology-and-budgets.md | Scoring layers, small-sample statistics, budget setting, CI wiring, methodology anti-patterns |
| assets/sample_suite.json | Six-scenario support-agent suite covering all assertion types |
| assets/sample_transcripts_baseline.json | Recorded baseline run |
| assets/sample_transcripts_candidate.json | Recorded candidate run containing a critical regression at an unchanged pass rate |
| assets/sample_baseline_report.json | Scored baseline report — input for eval_diff.py |
|
Truncated for display — read the full file on GitHub.
Related Skills
claude-mem
117.9kPersistent Context Across Sessions for Every Agent – Captures everything your agent does during sessions, compresses it with AI, and injects relevant context back into future sessions. Works with Claude Code, OpenClaw, Codex, Gemini, Hermes, Copilot, OpenCode + More
Understand-Anything
93.0kGraphs that teach > graphs that impress. Turn any code into an interactive knowledge graph you can explore, search, and ask questions about. Works with Claude Code, Codex, Cursor, Copilot, Gemini CLI, and more.
headroom
79.9kCompress tool outputs, logs, files, and RAG chunks before they reach the LLM. 20% fewer tokens for coding agents, 60-95% fewer tokens for JSON, same answers. Library, proxy, MCP server.
CowAgent
48.0kOpen-source personal AI assistant & Agent Harness. Plans tasks, runs tools and skills, self-evolves with memory and knowledge. Multi-agent, multi-model, multi-channel. Lightweight, extensible, one-line install.
Languages
Trust signals
From repository metadata: license, adoption, age and documentation. Not a code audit — see the Safety scan above for what the skill file itself contains.
