nw-agent-evals
Lightweight eval method for testing nWave AGENTS and SKILLS (LLM behavior) as a lean alternative to heavy BDD/ATD. An eval = one prompt -> one captured run (trace + artifacts) -> a small set of checks -> a comparable score over time.
Install / Use
npx skills add nWave-ai/nWave --skill nw-agent-evalsInstalls into whichever agent you are using.
SKILL.md
Installable skill definition
Quality Score
Category
AI & Machine LearningSupported Platforms
Our assessment of nw-agent-evals
nw-agent-evals scores 89/100 on our quality scale, 426th of 947 AI & Machine Learning skills we index (top 45%).
Its SKILL.md is 12 KB long, well organised into 12 sections with 3 code examples: a thorough specification that gives an agent plenty to work with.
It has 617 GitHub stars, a meaningful sign that others use it.
Maintenance, license and trust
- The repository was last updated about 2 months ago, so nw-agent-evals is actively maintained.
- It is released under the MIT license, a permissive license that allows use, modification and commercial use with attribution.
- Its trust signals score 100/100, with no cautions. These come from repository metadata, not a code audit — read the skill file before letting an agent act on it.
Safety scan
No issues foundOur scan of the whole file found no instruction hijacking, hidden characters, credential access, data exfiltration or destructive commands. An AI review of the same text found nothing harmful.
AI review by kimi-k2.7-code on 2026-11-10. Automated pattern scan on 2026-11-10. It catches known dangerous patterns, not every risk — read a skill before letting an agent act on it.
nw-agent-evals compared with similar skills
All 4 of these similar skills score higher than nw-agent-evals; compare them before choosing.
| Skill | Score | Stars | Updated | Format |
|---|---|---|---|---|
| nw-agent-evals (this skill)by nWave-ai | 89 | 617 | 2mo ago | SKILL.md |
| claude-memby thedotmack | 100 | 117.6k | today | CLAUDE.md |
| Understand-Anythingby Egonex-AI | 100 | 92.8k | 1d ago | CLAUDE.md |
| headroomby headroomlabs-ai | 100 | 79.8k | today | CLAUDE.md |
| CowAgentby zhayujie | 100 | 48.0k | today | CLAUDE.md |
Frequently asked questions
- How do I install nw-agent-evals?
- Run
npx skills add nWave-ai/nWave --skill nw-agent-evals. The install tabs above show the steps for each supported agent. - Which AI agents does nw-agent-evals work with?
- It is written for Universal, as a SKILL.md file. Other agents that read the same format can often use it too.
- Is nw-agent-evals safe to use?
- Our scan of the whole file found no instruction hijacking, hidden characters, credential access, data exfiltration or destructive commands. An AI review of the same text found nothing harmful. It is MIT-licensed and scores 100/100 on trust signals. Skills are instructions an agent will follow, so read the file before installing it and do not approve commands you do not understand.
- Is nw-agent-evals still maintained?
- The repository was last updated about 2 months ago, so nw-agent-evals is actively maintained.
Skill content
View source on GitHubname: nw-agent-evals description: Lightweight eval method for testing nWave AGENTS and SKILLS (LLM behavior) as a lean alternative to heavy BDD/ATD. An eval = one prompt -> one captured run (trace + artifacts) -> a small set of checks -> a comparable score over time. Load when validating agent behavior, building a regression net for an agent/skill, or reducing agent-test bloat. user-invocable: false disable-model-invocation: true
Agent Evals
Why this exists
Agents are LLMs — non-deterministic. Traditional BDD/unit testing tests them badly and breeds bloat (measured: 294K test LOC, 5.5:1 test:src, step reuse 1.10x vs >=4x target). Evals are the lean way to verify agent behavior: a few targeted signals instead of a monolithic spec.
This skill is the antithesis of ATD over-specification. Keep it small. The anti-pattern is up-front exhaustive specification — that IS the bloat.
| | nw-agent-testing (sibling skill) | nw-agent-evals (this skill) |
|---|---|---|
| Form | static 5-layer manual checklist | executable dataset + grader + score over time |
| Use | one-shot design review of a spec | repeatable regression net for behavior |
| Output | pass/fail judgement | comparable score, trend across runs |
Use both: nw-agent-testing to vet the spec, nw-agent-evals to watch behavior over time.
What an eval is
One eval = prompt -> run -> checks -> score.
- prompt — a single input that should (or should NOT) trigger the agent/skill.
- run — one provider turn through the selected nWave adapter, with its available native evidence and artifacts captured.
- checks — a small set of targeted assertions (not one monolithic check).
- score — a comparable number you can track across runs to catch regressions.
Replaces "vibes" with measurable signals: did it invoke the right skill, run the expected tools, respect the conventions and produce the required observable effect? Never grade a terminal-text grammar as a behavioral outcome.
Definition of Done — before you write the eval
Write the success criteria FIRST, before implementing the agent/skill or its eval. Four check categories:
| Category | Question | Graded by | |---|---|---| | OUTCOME | Did the task complete through its required public effect and evidenced semantic result? | deterministic | | PROCESS | Was required skill delivery or use evidenced, with only behavior-relevant tool observations? | deterministic | | STYLE | Does the output respect nWave conventions (sections, format)? | model-graded | | EFFICIENCY | No useless commands / no token blowup? | deterministic |
If you cannot state DoD before writing the skill, the skill's job is not yet defined — stop and define it.
Workflow
Run these steps in order:
- Define success first — write a small set of falsifiable checks in the four categories above.
- Use one representative probe when needed — capture the provider-native evidence that the selected adapter actually exposes.
- Add only the cases needed — include a positive case and a relevant negative control; grow the set from observed failures.
- Grade deterministic evidence — check public effects, admitted tool evidence, and structured artifacts. Do not derive a verdict from terminal prose.
- Grade semantic quality separately — use the existing rubric facility; retain narrative feedback as diagnostic evidence, not terminal control input.
Capturing provider evidence
The selected adapter owns the provider projection. Claude roles run through
the Claude adapter; Codex roles run through the Codex adapter's codex exec
projection. Do not substitute one provider's trace shape, tool names, or
launcher for the other's.
Distinguish these three facts when evaluating a skill:
| Fact | Evidence | What it establishes |
|---|---|---|
| Preloaded knowledge | The role declares the skill in frontmatter and load_role_instructions materializes it in the provider instruction. | The role received the skill before the turn. It creates no native Read event. |
| Native read | A provider-native tool event reads a conditional SKILL.md path. Claude's existing transcript tracker records this form. | The turn fetched that file on demand. It does not prove that the knowledge affected the result. |
| Actual use | A predicted, observable effect or a capability-specific tool/result is present, with no simpler explanation. | The role applied the skill sufficiently for this eval. A catalog entry, preload, or Read alone is insufficient. |
Evaluate semantic results and public effects separately from their presentation. Terminal prose, stdout, stderr, recorder entries and native tool traces carry diagnostic/provenance evidence, not interchangeable proof of a durable effect. Use the selected host's native transcript/tool receipts rather than another parser. Inspect only events and recorder evidence that its adapter preserves. If the required observation is absent on the selected provider, report that check as INDETERMINATE; do not infer it from another projection.
Check an artifact on disk as well as its reported tool event. Only ask for a tool when the behavior requires it. Tool order and an exact command sequence are not general quality signals.
Deterministic graders (nWave-native signals)
Parse the trace, assert mechanically. nWave-specific, high-value signals:
| Signal | Assertion | Why it matters |
|---|---|---|
| Declared preload | resolved role instruction contains the declared skill once | verifies delivery of resident knowledge, not its use |
| Conditional read | native Read event for the resolved skill path, where the provider exposes one | verifies on-demand fetch, not semantic use |
| Capability-specific behavior | required public effect or justified tool/result is present | tests actual application of the skill |
| Semantic outcome observed | the semantic result and required public effect are evidenced | separates judgment from presentation |
| Artifact content | the required document contains the accepted facts | checks durable outcome rather than a completion claim |
| Efficiency | compare only a measured, relevant excess such as duplicate calls | avoids arbitrary ceilings |
| Negative control | the unsupported behavior or unnecessary skill effect is absent | guards against over-eager invocation |
Bind to the code-fact CLI where useful: e.g. assert the agent invoked code-fact query.callers-of rather than relying on a catalog entry (catalogued != wired). Note that feature-level change-scope analysis has no stable CLI today — an eval must not demand a capability the production CLI does not expose, or it grades the tooling rather than the agent.
Qualitative grader (model-graded rubric)
For STYLE / design-quality / review-quality, use the existing rubric facility. Keep each check single-purpose: identity, pass/fail judgement and evidence-located notes; retain the overall judgement and comparable score in the harness-owned result. Models supply judgements, not serialized rubric templates or terminal grammars. The harness owns result structure and recording; unavailable recording capability is INDETERMINATE, not permission to ask a role to print a data object.
When behavior under eval supplies a DESIGN decision, evaluate two linked observations: the values satisfy the Request, and the resulting authority contains them. Preserve one current authority rather than a competing ADR or brief. Revisions preserve unaffected facts; unsuccessful changes must not silently replace accepted authority.
For an EXAMINE eval, preserve the source-blind boundary. Give the examiner the available observation packet, not source access; an absent observation remains INDETERMINATE rather than becoming an inferred pass or failure.
Dataset
Use the smallest CSV dataset that covers the observed positive behavior and a relevant negative control. Minimum columns:
id,prompt,should_trigger,expected_skill,expected_observation,notes
ev-01,"Describe design facts for X",true,nw-design-patterns,"decision satisfies Request; resulting authority contains its facts",explicit
ev-07,"Just fix this typo",false,,,"negative control - no design facts"
- Mix: explicit-invocation, implicit-from-description (does
descriptionalone trigger it?), contextual, and NEGATIVE-CONTROLS (should_trigger=false). name+descriptionare the PRIMARY invocation signal — implicit rows test exactly that.- Coverage grows from real failures, never speculatively.
Where evals live
tests/evals/<agent-or-skill-name>/
dataset.csv # the prompt set
rubric.json # model-graded rubric (JSON-Schema)
README.md # DoD + how to run
runs/ # captured transcripts + scores per run (gitignored or pruned)
Reuse the existing tests/evals/ facility; do not create a new general
harness. Keep raw provider records out of committed bloat unless a test needs a
small, scrubbed fixture.
Principles
- Define success before you write the skill — no DoD, no skill.
- Small targeted checks beat monolithic ones — many cheap signals catch regressions early; one giant assertion hides them.
- Every manual fix is a future eval — coverage is earned from observed failures.
- Negative controls are mandatory — an agent that fires when it shouldn't is as broken as one that doesn't fire.
- name + description are the invocation contract — test them, don't bypass them with explicit invocation only.
- Least privilege — eval graders are read-only over traces + artifacts.
- Stay lean — over-specifying up front recreates the ATD bloat this method exists to avoid.
Example: eval for nw-solution-architect
DoD — the architect supplies valid semantic design facts and their resulting authority. When structural code facts are necessary and obtainable, they are provider-labelled and support the decision. A completion claim alone establishes neither valid content nor the durable effect.
Dataset rows (excerpt):
id,prompt,should_trigger,expected_skill,expected_observation,notes
sa-01,"Describe design facts for the handoff-state-algebra feature",true,nw-design-patterns,"decision satisfies Request; resulting ADR contains its facts",design
sa-02,"What ADRs exist for the gate layer?",true,,"provider-labelled code facts support the answer when code facts are needed",lookup
sa-03,"Rename this variable to camelCase",false,,,negative control - not an architecture task
Deterministic grader (over the captured trace + artifact):
- DELIVERY: the role's resolved instruction contains the declared preload once; a conditional native read is checked only when this case requires it.
- OUTCOME: the design facts satisfy the requested decision.
- OUTCOME: the declared ADR contains accepted facts, observed through the actual consumer surface.
- EVIDENCE: when structural analysis is necessary, its provider and confidence labels support the decision; do not demand a fixed shell spelling or order.
- NEGATIVE (sa-03): no architecture facts or durable-design production occurs.
Model-graded rubric (STYLE/quality):
| Check | Example judgement | Evidence note | |---|---|---| | Context states the problem | Pass | Clear problem framing in the generated ADR | | Decision is singular | Pass | One owned decision addresses the supplied gap | | Consequences quantify trade-offs | Fail | Trade-offs remain qualitative |
These are rubric judgements, not a role-output template. The harness constructs the stored result; terminal notes alone establish no recorded outcome.
Score = deterministic checks (binary, weighted) + rubric score, tracked per
comparable run. A missing required fact, generated document, or relevant supporting
observation flags a regression; unavailable evidence is reported as
INDETERMINATE.
Related Skills
claude-mem
117.6kPersistent Context Across Sessions for Every Agent – Captures everything your agent does during sessions, compresses it with AI, and injects relevant context back into future sessions. Works with Claude Code, OpenClaw, Codex, Gemini, Hermes, Copilot, OpenCode + More
Understand-Anything
92.8kGraphs that teach > graphs that impress. Turn any code into an interactive knowledge graph you can explore, search, and ask questions about. Works with Claude Code, Codex, Cursor, Copilot, Gemini CLI, and more.
headroom
79.8kCompress tool outputs, logs, files, and RAG chunks before they reach the LLM. 20% fewer tokens for coding agents, 60-95% fewer tokens for JSON, same answers. Library, proxy, MCP server.
CowAgent
48.0kOpen-source personal AI assistant & Agent Harness. Plans tasks, runs tools and skills, self-evolves with memory and knowledge. Multi-agent, multi-model, multi-channel. Lightweight, extensible, one-line install.
Languages
Trust signals
From repository metadata: license, adoption, age and documentation. Not a code audit — see the Safety scan above for what the skill file itself contains.
