eval-engineering
Inspect an agent repository and optional traces, interview the user, write reviewed Task Specs, build and audit Harbor tasks, and bootstrap reusable project World Knowledge Skills.
Install / Use
npx skills add langchain-ai/langchain-skills --skill eval-engineeringInstalls into whichever agent you are using.
SKILL.md
Installable skill definition
Quality Score
Category
Development & EngineeringSupported Platforms
Our assessment of eval-engineering
eval-engineering scores 90/100 on our quality scale, 1383rd of 4,698 Development & Engineering skills we index (top 30%).
Its SKILL.md is 15 KB long, well organised into 13 sections with 1 code example: a thorough specification that gives an agent plenty to work with.
With 1,246 GitHub stars, it is one of the more widely adopted skills in the catalogue.
Maintenance, license and trust
- The repository was last updated about 2 months ago, so eval-engineering is actively maintained.
- Our last check on 2026-10-30 found the source still online.
- It is released under the MIT license, a permissive license that allows use, modification and commercial use with attribution.
- Its trust signals score 100/100, with no cautions. These come from repository metadata, not a code audit — read the skill file before letting an agent act on it.
Safety scan
No issues foundOur scan of the whole file found no instruction hijacking, hidden characters, credential access, data exfiltration or destructive commands. An AI review of the same text found nothing harmful.
AI review by kimi-k2.7-code on 2026-10-04. Automated pattern scan on 2026-10-04. It catches known dangerous patterns, not every risk — read a skill before letting an agent act on it.
eval-engineering compared with similar skills
All 4 of these similar skills score higher than eval-engineering; compare them before choosing.
| Skill | Score | Stars | Updated | Format |
|---|---|---|---|---|
| eval-engineering (this skill)by langchain-ai | 90 | 1.2k | 54d ago | SKILL.md |
| taste-skillby Leonxlnx | 100 | 116.8k | 8d ago | CLAUDE.md |
| ai-job-searchby ai-job-search-dev | 100 | 72.1k | today | CLAUDE.md |
| claude-howtoby luongnv89 | 100 | 47.3k | 2d ago | CLAUDE.md |
| Claude-Code-Game-Studiosby Donchitos | 100 | 39.7k | 7d ago | CLAUDE.md |
Frequently asked questions
- How do I install eval-engineering?
- Run
npx skills add langchain-ai/langchain-skills --skill eval-engineering. The install tabs above show the steps for each supported agent. - Which AI agents does eval-engineering work with?
- It is written for Universal, as a SKILL.md file. Other agents that read the same format can often use it too.
- Is eval-engineering safe to use?
- Our scan of the whole file found no instruction hijacking, hidden characters, credential access, data exfiltration or destructive commands. An AI review of the same text found nothing harmful. It is MIT-licensed and scores 100/100 on trust signals. Skills are instructions an agent will follow, so read the file before installing it and do not approve commands you do not understand.
- Is eval-engineering still maintained?
- The repository was last updated about 2 months ago, so eval-engineering is actively maintained.
Skill content
View source on GitHubname: eval-engineering description: Inspect an agent repository and optional traces, interview the user, write reviewed Task Specs, build and audit Harbor tasks, and bootstrap reusable project World Knowledge Skills. Use for agent evals, benchmark design, Task generation, controlled Environments, synthetic data, Verifiers, Harbor runs, calibration, or continuous benchmark maintenance.
Eval Engineering
Flow
- Inspect all inputs first: the repository, Harness, optional traces, existing Tasks and runs, existing World knowledge, and the human goal. Identify the source files and skill references that apply before proposing work.
- Create or update the small project World Knowledge Skill from reusable facts in those inputs. Use it to propose one grounded Task.
- Draft the Task Spec and World Skill together. Show both to the user, keep exact
Task truth only in
Task.md, and refine both until the user approves them. - Implement the approved Task, validate its Environment and Verifier, run the real Harness, inspect the full evidence, and fix only non-agent failures.
- Reconcile the World Skill with what the run proved, then repeat this flow for the next Task.
Terms
- Task Spec:
Task.md, which describes the input, relevant agent conditions, Environment, scoring, fairness, and open decisions for one Task. - Task: the runnable instruction, Environment, and Verifier.
- Harness: the complete agent Harbor runs, including prompts, model loop, tools, hooks, memory, sessions, and adapter.
- Environment: the files, data, services, identity, permissions, network, clock, and mutable state around the Harness.
- Verifier: independent checks that score the result or mark a run invalid.
- World Knowledge Skill: a repository-local skill with reusable project-specific knowledge, references, scripts, assets, and tests that help generate Task Specs and build future Tasks.
- Spec2Task: the full loop that turns a reviewed Task Spec into an audited runnable Task. Follow Task implementation for its concrete build order and reference routing.
Reference routing
Read each reference when its decision appears:
| Need | Read | |---|---| | Inspect source, traces, the Harness, dependencies, access, and existing evals | Discovery | | Bootstrap or update reusable project knowledge | World knowledge | | Propose Tasks and write the single Task Spec | Task design | | Build data, services, access, state, and reset | Environment building | | Create structured or natural-language data | Synthetic data | | Define independent evidence and scoring | Verifier design | | Apply Spec2Task to turn a reviewed Spec into an audited Task | Task implementation | | Compare model runs and classify failures | Calibration | | Package and run Harbor tasks | Harbor | | Adapt a known benchmark design | Benchmark patterns | | Build multi-turn conversations | Multi-turn simulation | | See World knowledge learned across two Tasks | Service-desk example |
Reusable implementation resources:
- Multi-turn runner, model user, and Harbor adapter example
- Tool-schema comparison, which compares
supplied schema fragments but does not resolve external
$reftargets - Read-only SQLite state snapshot
1. Inspect inputs and existing World knowledge
Review every input the user provides before proposing a Task. Use the guidance that matches each available input:
- For a repository, Harness, traces, or dependencies, read Discovery.
- For existing Tasks and runs, inspect their instructions, Environments, Verifiers, rewards, trajectories, and final state. Read Calibration when run quality or failure causes affect the new design.
- For an existing project World Skill, read World knowledge, then check the sources and reusable methods that affect the new Task.
- For human goals and constraints, read Task design.
- For relevant benchmark examples, use the domain index and source callouts in Benchmark patterns.
Inspect the repository before asking questions that source and tests can answer. Follow the active Harness through prompts, models, tools, services, state, effects, and focused tests. Inspect existing Task instructions, parsers, Verifiers, reward paths, and run evidence.
If the user supplies traces, review complete runs or threads. Use traces to learn real requests, dependency behavior, state shapes, errors, and failure conditions. Do not treat a trace answer as independent truth.
If .agents/skills/<project>-world/SKILL.md exists, read it. Follow its routing
only for knowledge relevant to the current Task. Check cited repository paths,
commands, and scripts when their accuracy affects the design.
2. Propose and select a Task
Read Task design and use the index in Benchmark patterns to find the relevant domain and source callouts. Focus on that domain unless the Task crosses another one. In the first user-facing design response after inspection, propose one Task grounded in repository evidence, supplied traces, existing coverage, or a human priority. State:
- the real work and capability;
- the condition that makes the case non-trivial;
- the Environment and independent evidence it needs;
- the important failure it can detect;
- how it differs from existing Tasks; and
- the main open decision.
In the same response, show the relevant current World Skill content and the specific additions or corrections this Task suggests. If no World Skill exists, show the small initial contents that will help create this Task and future Tasks. Keep the Task's exact request, focal records, expected result, hidden truth, and exact scoring rules out of the World Skill.
Let the user revise the Task proposal and World knowledge together before implementation. Offer alternatives only when a real user choice changes the design.
3. Write and review the Task Spec and World Skill
Copy the Task template to
evals/<suite>/tasks/<task-id>/Task.md. Put all Task-specific design in this
one file. At the same time, create or update the project World Skill by
following World knowledge. Determine the
project skill location supported by the active agent and repository.
.agents/skills/<project>-world/SKILL.md and
.claude/skills/<project>-world/SKILL.md are common landing spots. Follow an
established project convention when one exists. Otherwise, explain the proposed
location and get user confirmation before creating the skill. Start from
the World Skill template when needed.
Keep each Task.md beside the Harbor task it describes:
evals/<suite>/tasks/<task-id>/
├── Task.md # human-reviewed control-plane spec
├── task.toml # required Harbor configuration
├── instruction.md # required agent input
├── environment/ # required Environment definition and visible state
│ ├── Dockerfile # use this or docker-compose.yaml
│ └── docker-compose.yaml # optional; primary service must be main
├── tests/
│ ├── test.sh # required Harbor Verifier entry point
│ ├── test_*.py # optional Verifier helpers
│ └── fixtures/ # optional hidden Verifier data
└── solution/
└── solve.sh # optional reference path
Never copy or mount Task.md into the evaluated agent's workspace or image.
The agent receives instruction.md and only the Environment state intended for
the run.
Include:
- purpose and source evidence;
- exact input and later turns;
- only the Harness conditions relevant to this Task;
- initial state, services, access, visibility, reset, and production differences;
- required results, prohibited effects, accepted alternatives, and independent Verifier evidence;
- fairness, leakage risks, and invalid-run conditions; and
- open decisions and assumptions.
Show the full Task Spec and the World Skill changes to the user. Explain what
is already in the World Skill, what this Task adds or corrects, and what stays
only in Task.md. Revise both through the same back-and-forth. Mark the Task
Spec approved only after explicit approval. Treat World Skill changes as
accepted only after the user reviews them. If the user requests an end-to-end
build without an approval pause, continue with an agent-reviewed
Status: Draft and label the World Skill changes as unreviewed.
If implementation changes the request, visible information, material
Environment behavior, or scoring boundary, update Task.md and show the
change. Set its status back to Draft. Show the diff and require explicit
reapproval before setting it to Approved again.
4. Apply Spec2Task
Follow Task implementation. It gives the build order and routes each decision to the Environment, synthetic-data, Verifier, Harbor, and calibration references.
For an existing project, use its pinned or supported Harbor version. Otherwise, use the installed supported version and record it. Upgrade only with user approval and a stated compatibility reason. Use the installed CLI help as the command contract.
Before a scored model run:
- Confirm the model, trial count, judge, timeout, and maximum expected cost with the user unless the user already authorized that run plan.
- Complete the package audit in Harbor. Confirm every required file, entry point, path, permission, configuration value, mount, service, and reward output needed for this exact Task is present and works through Harbor. Confirm hidden Task, Verifier, solution, and secret material is absent from the agent-visible image and workspace.
- Check setup and trial isolation in the way that fits the Environment. A fresh container or worktree can provide isolation by replacement. A reused mutable service needs a checked reset. Immutable frozen data needs only a checked load. See Task implementation.
- Exercise every operation the Task depends on.
- Run the reference path when one exists.
- Test the Verifier with a clear valid result, a valid alternative, a realistic wrong result, a shortcut, a prohibited collateral change, and missing or corrupt evidence.
- Confirm every completed Verifier path writes a valid reward and useful evidence without exposing hidden truth or secrets.
5. Run and audit
Run the actual Harness through Harbor. Read the complete trajectory, not only the reward. Inspect:
- messages, model calls, tool calls, results, retries, and errors;
- initial and final Environment state and external effects;
- service, setup, readiness, reset, and cleanup evidence;
- each Verifier criterion, its evidence, decision, and error; and
- the resolved Harness, model, Environment, and judge configuration.
Classify each unsuccessful run as an agent capability failure, missing information, Harness defect, Environment defect, Verifier false rejection, Verifier false acceptance, leakage, or infrastructure failure. Fix non-agent failures before using the score.
Model comparison is an optional c
Truncated for display — read the full file on GitHub.
Related Skills
taste-skill
116.8kAgent skills that give coding agents taste: art direction, reference images and a browser check instead of generic AI pages
ai-job-search
72.1kThe job search that runs on your machine. AI job application framework built on Claude Code: evaluate postings, tailor CVs, write cover letters, prep interviews. Fork it and own it.
claude-howto
47.3kA visual, example-driven guide to Claude Code — from basic concepts to advanced agents, with copy-paste templates that bring immediate value.
Claude-Code-Game-Studios
39.7kTurn Claude Code into a full game dev studio — 49 AI agents, 72 workflow skills, and a complete coordination system mirroring real studio hierarchy.
Languages
Trust signals
From repository metadata: license, adoption, age and documentation. Not a code audit — see the Safety scan above for what the skill file itself contains.
