SkillAgentSearch skills...

eval-integrity

Assess integrity of an eval, benchmark, metric, A/B test, holdout, LLM-judge, or user-study claim when the question is whether measurement, comparison, or evidence is independent and trustworthy.

Install / Use

npx skills add code-yeongyu/oh-my-openagent --skill eval-integrity

Installs into whichever agent you are using.

About this skill
📄

SKILL.md

Installable skill definition

Quality Score

89/100

Supported Platforms

Universal

Our assessment of eval-integrity

eval-integrity scores 89/100 on our quality scale, 409th of 947 AI & Machine Learning skills we index (top 44%).

Its SKILL.md is 5.8 KB long, well organised into 8 sections and no code examples: a solid amount of guidance for an agent.

With 69,362 GitHub stars, it is one of the more widely adopted skills in the catalogue.

Substance
26/30
Structure
13/20
Description
15/15
Adoption
20/20
Freshness
15/15

Maintenance, license and trust

  • The repository was last updated about 2 months ago, so eval-integrity is actively maintained.
  • No license is declared. By default that means all rights are reserved: you can read it, but reusing or redistributing it is not clearly permitted. Ask the author before building on it commercially.
  • Its trust signals score 88/100, with 1 caution from licensing, adoption, age or documentation. These come from repository metadata, not a code audit — read the skill file before letting an agent act on it.

Safety scan

No issues found

Our scan of the whole file found no instruction hijacking, hidden characters, credential access, data exfiltration or destructive commands. An AI review of the same text found nothing harmful.

AI review by kimi-k2.7-code on 2026-11-14. Automated pattern scan on 2026-11-14. It catches known dangerous patterns, not every risk — read a skill before letting an agent act on it.

eval-integrity compared with similar skills

All 4 of these similar skills score higher than eval-integrity; compare them before choosing.

SkillScoreStarsUpdatedFormat
eval-integrity (this skill)by code-yeongyu8969.4k53d agoSKILL.md
claude-memby thedotmack100117.6ktodayCLAUDE.md
Understand-Anythingby Egonex-AI10092.8k1d agoCLAUDE.md
headroomby headroomlabs-ai10079.8ktodayCLAUDE.md
CowAgentby zhayujie10048.0ktodayCLAUDE.md

Frequently asked questions

How do I install eval-integrity?
Run npx skills add code-yeongyu/oh-my-openagent --skill eval-integrity. The install tabs above show the steps for each supported agent.
Which AI agents does eval-integrity work with?
It is written for Universal, as a SKILL.md file. Other agents that read the same format can often use it too.
Is eval-integrity safe to use?
Our scan of the whole file found no instruction hijacking, hidden characters, credential access, data exfiltration or destructive commands. An AI review of the same text found nothing harmful. It declares no license and scores 88/100 on trust signals. Skills are instructions an agent will follow, so read the file before installing it and do not approve commands you do not understand.
Is eval-integrity still maintained?
The repository was last updated about 2 months ago, so eval-integrity is actively maintained.

name: eval-integrity description: "Assess integrity of an eval, benchmark, metric, A/B test, holdout, LLM-judge, or user-study claim when the question is whether measurement, comparison, or evidence is independent and trustworthy."

Evaluation Integrity

Provide a read-only audit of whether an evaluation can support its stated claim. Do not alter code, data, prompts, labels, or study operations. Begin with exactly one verdict line in the form **Verdict: PASS**, **Verdict: FAIL**, or **Verdict: INCONCLUSIVE**. Emit no other verdict line and do not restate the verdict later in the report. A favorable score is not itself evidence of integrity.

Route correctly

  • NOT for code review: route implementation, correctness, security, or maintainability review to review-work.
  • NOT for data processing: route cleaning, transforming, joining, modeling, or analyzing data to data-scientist.
  • Use this skill for the independence and validity of the measurement setup, not for producing a result from it.

Audit method

1. State the claim and map the evaluation

Write the claim in testable form: what comparison, metric, threshold, or causal statement is being made; for which population; and under which decision rule. Map every role before judging it:

| Element | Record | | --- | --- | | Claim | The conclusion and decision it is meant to justify. | | System under test | Exact model, version, configuration, prompt, policy, or treatment. | | Dataset and split | Sources, collection dates, inclusion rules, train/dev/test or holdout assignment. | | Designer | Who selected the task, data, prompts, treatments, metric, and stopping rule. | | Evaluator or judge | Who or what assigns quality judgments, including model and prompt versions. | | Scorer | The rule or system that turns outputs or judgments into the reported metric. | | Subjects | Humans, records, tasks, or units exposed to treatment or observation. |

Trace shared ownership, shared data, shared prompts, shared labels, and shared selection decisions across this map. Several models do not establish independence merely because there are several of them; identify distinct training, tuning, prompt design, labels, data, and decision authority.

2. Check the ground truth

Identify the independent ground truth that could confirm the claim without reusing the system's own preferences, outputs, labels, or optimization target. Test whether it is:

  • external to the system under test and its designer;
  • collected or adjudicated without seeing the treatment identity when blinding is feasible;
  • appropriate to the claim rather than a proxy selected because it favors one arm; and
  • fixed before comparison, or accompanied by a disclosed multiplicity and selection procedure.

If no independent ground truth exists, do not substitute agreement among related systems. Mark the claim INCONCLUSIVE unless the stated claim is explicitly limited to that related system's score.

3. Probe independence failures

Check only risks relevant to the mapped design:

  • Contamination: benchmark items, answers, or near duplicates reached training, prompt design, retrieval, tuning, or post-hoc selection.
  • Invalid controls: arms differ beyond the intended treatment, allocation is biased, exposure is unequal, or the control cannot answer the causal comparison.
  • Circular judging: the evaluated system, its designer, or a derived rubric defines its own success without external confirmation.
  • Scorer-label coupling: labels, thresholds, reward signals, or rubric choices are reused by the scorer in a way that makes agreement self-fulfilling.
  • Evaluator-designer coupling: the evaluator, judge prompt, or adjudication process is selected, tuned, or influenced by the comparison designer without an independent safeguard.
  • Split or annotator dependence: overlapping sources, duplicates, temporal leakage, shared annotators, or correlated annotation policies invalidate assumed independence.
  • Hypothesis priming: task framing, examples, pre-registration gaps, or interim results steer evaluators or analysts toward the expected outcome.
  • Observer and demand effects: subjects or observers infer the desired treatment or outcome, and behavior or ratings change because of that knowledge.

For each concern, distinguish documented evidence from an assumption. Missing provenance for data, splits, labels, prompts, evaluator selection, or scoring is INCONCLUSIVE, not evidence that the design was independent.

4. Reach the verdict

  • PASS: evidence supports an independent, valid measurement path for the stated claim, with no material unresolved dependency.
  • FAIL: evidence shows a dependency or invalid design that materially invalidates or overstates the stated claim.
  • INCONCLUSIVE: provenance or evidence is missing, the ground truth is not independently established, or remaining uncertainty prevents either result.

Do not convert uncertainty into PASS because a metric looks plausible, a result is statistically significant, or multiple related evaluators agree.

Report

Report the verdict first. Then report only findings that fired; do not pad the report with risks that were checked and did not fire. For each fired finding include:

  1. Evidence: the observed artifact, provenance record, or explicit absence of one.
  2. Impact: which claim or inference the issue weakens or invalidates.
  3. Minimal independence fix: the smallest separation, blind, held-out source, external adjudication, or control change that would address it.
  4. Missing evidence: what must be supplied to resolve the finding, if applicable.
  5. Residual risk: what remains uncertain after the minimal fix.

Keep conclusions bounded to the evidence. This audit advises; it does not certify a system, recompute results, or approve deployment.

Related Skills

View on GitHub
GitHub Stars69.4k
CategoryAI
Updated1mo ago
Forks5.7k

Languages

TypeScript

Trust signals

88/100

From repository metadata: license, adoption, age and documentation. Not a code audit — see the Safety scan above for what the skill file itself contains.

1 medium