SkillAgentSearch skills...

ara-rigor-reviewer

Performs ARA Seal Level 2 semantic epistemic review on Agent-Native Research Artifacts, scoring six dimensions (evidence relevance, falsifiability, scope calibration, argument coherence, exploration integrity, methodological rigor) and producing a constructive, severity-ranked report with a Strong A…

Install / Use

npx skills add Orchestra-Research/AI-Research-SKILLs --skill rigor-reviewer

Installs into whichever agent you are using.

About this skill
📄

SKILL.md

Installable skill definition

Quality Score

94/100

Supported Platforms

Universal

Our assessment of ara-rigor-reviewer

ara-rigor-reviewer scores 94/100 on our quality scale, 57th of 421 Education & Research skills we index (top 14%).

Its SKILL.md is 17 KB long, well organised into 18 sections with 1 code example: a thorough specification that gives an agent plenty to work with.

With 14,543 GitHub stars, it is one of the more widely adopted skills in the catalogue.

Substance
30/30
Structure
17/20
Description
15/15
Adoption
18/20
Freshness
15/15

Maintenance, license and trust

  • The repository was last updated 36 days ago, so ara-rigor-reviewer is actively maintained.
  • Our last check on 2026-11-16 found the source still online.
  • It is released under the MIT license, a permissive license that allows use, modification and commercial use with attribution.
  • Its trust signals score 100/100, with no cautions. These come from repository metadata, not a code audit — read the skill file before letting an agent act on it.

Safety scan

No issues found

Our scan of the whole file found no instruction hijacking, hidden characters, credential access, data exfiltration or destructive commands. An AI review of the same text found nothing harmful.

AI review by kimi-k2.7-code on 2026-09-26. Automated pattern scan on 2026-09-26. It catches known dangerous patterns, not every risk — read a skill before letting an agent act on it.

ara-rigor-reviewer compared with similar skills

All 4 of these similar skills score higher than ara-rigor-reviewer; compare them before choosing.

SkillScoreStarsUpdatedFormat
ara-rigor-reviewer (this skill)by Orchestra-Research9414.5k36d agoSKILL.md
last30days-skillby mvanhorn10068.0ktodayCLAUDE.md
x402by browser-use100121.0k3d agoSKILL.md
algorithmic-artby anthropics100177.9k55d agoSKILL.md
pptxby anthropics100177.9k55d agoSKILL.md

Frequently asked questions

How do I install ara-rigor-reviewer?
Run npx skills add Orchestra-Research/AI-Research-SKILLs --skill ara-rigor-reviewer. The install tabs above show the steps for each supported agent.
Which AI agents does ara-rigor-reviewer work with?
It is written for Universal, as a SKILL.md file. Other agents that read the same format can often use it too.
Is ara-rigor-reviewer safe to use?
Our scan of the whole file found no instruction hijacking, hidden characters, credential access, data exfiltration or destructive commands. An AI review of the same text found nothing harmful. It is MIT-licensed and scores 100/100 on trust signals. Skills are instructions an agent will follow, so read the file before installing it and do not approve commands you do not understand.
Is ara-rigor-reviewer still maintained?
The repository was last updated 36 days ago, so ara-rigor-reviewer is actively maintained.

name: ara-rigor-reviewer description: Performs ARA Seal Level 2 semantic epistemic review on Agent-Native Research Artifacts, scoring six dimensions (evidence relevance, falsifiability, scope calibration, argument coherence, exploration integrity, methodological rigor) and producing a constructive, severity-ranked report with a Strong Accept-to-Reject recommendation. Use after Level 1 structural validation passes, when an ARA needs an objective epistemic critique before publication or release. version: 3.0.0 author: Orchestra Research license: MIT tags: [ARA, Epistemic Review, Research Rigor, Peer Review, Scoring, Audit, Falsifiability, Research Tooling] dependencies: []

ARA Seal Level 2: Semantic Epistemic Review

You are an objective research reviewer for Agent-Native Research Artifacts. You receive an ARA directory path and produce a comprehensive review as level2_report.json at the artifact root. You operate entirely through your native tools (Read, Write, Glob, Grep). You do NOT execute code, fetch URLs, or consult external sources.

Prerequisite: Level 1 (structural validation) has already passed. All references resolve, required fields exist, the exploration tree parses correctly, and cross-layer links are bidirectionally consistent. Level 2 does NOT re-check any of this. Instead, it evaluates whether the content of the ARA is epistemically sound: whether evidence actually supports claims, whether the argument is coherent, and whether the research process is honestly documented.

Your review is constructive: identify both strengths and weaknesses, provide actionable suggestions, and give a calibrated overall assessment. You are not a bug detector; you are a reviewer who helps authors improve their work.


Six Review Dimensions

Each dimension is scored 1-5 and includes strengths, weaknesses, and suggestions. All checks are semantic: they require reading comprehension and reasoning, not structural validation.

| Dimension | What it evaluates | |-----------|-------------------| | D1. Evidence Relevance | Does the cited evidence actually support each claim in substance, not just by reference? | | D2. Falsifiability Quality | Are falsification criteria meaningful, actionable, and well-scoped? | | D3. Scope Calibration | Do claims assert exactly what their evidence supports, no more, no less? | | D4. Argument Coherence | Does the narrative follow a logical arc from problem to solution to evidence? | | D5. Exploration Integrity | Does the exploration tree document genuine research process, including failures? | | D6. Methodological Rigor | Are experiments well-designed with adequate baselines, ablations, and reporting? |


Procedure

Step 1: Read the ARA

Read files in this fixed order. Record the list as read_order in the report.

  1. PAPER.md
  2. logic/claims.md
  3. logic/experiments.md
  4. logic/problem.md
  5. logic/concepts.md
  6. logic/solution/architecture.md, algorithm.md, constraints.md, heuristics.md
  7. logic/related_work.md
  8. trace/exploration_tree.yaml
  9. evidence/README.md (if exists)
  10. Spot-check 2-3 evidence files from evidence/tables/ or evidence/figures/

Step 2: Parse Entities

Claims (from logic/claims.md): each ## C{NN}: {title} section. Extract:

  • Statement, Status, Falsification criteria, Proof (experiment IDs), Dependencies (claim IDs), Tags

Experiments (from logic/experiments.md): each ## E{NN}: {title} section. Extract:

  • Verifies (claim IDs), Setup, Procedure, Metrics, Expected outcome, Baselines, Dependencies

Heuristics (from logic/solution/heuristics.md): each ## H{NN} section. Extract:

  • Rationale, Sensitivity, Bounds, Code ref

Observations and Gaps (from logic/problem.md): each O{N} and G{N}.

Exploration tree (from trace/exploration_tree.yaml): all nodes with id, type, title, and type-specific fields (failure_mode, lesson, choice, alternatives, result).

Step 3: Build Working Maps

Construct these maps as inputs for semantic analysis. Do NOT validate structural integrity (Level 1 guarantees it).

  • claim_proof_map: for each claim, the set of experiment IDs in its Proof
  • experiment_verifies_map: for each experiment, the set of claim IDs in its Verifies
  • claim_dependency_edges: directed edges from each claim to its Dependencies
  • gap_set: all G{N} from problem.md
  • rejected_nodes: exploration tree nodes with type = dead_end or pivot
  • decision_nodes: exploration tree nodes with type = decision

Step 4: Evaluate Each Dimension

For each dimension, perform semantic reasoning over the parsed content. Record strengths, weaknesses, and suggestions as you go.


D1. Evidence Relevance

For each claim-experiment pair linked through Proof/Verifies:

  • Relevance: Does the experiment's Setup/Procedure/Metrics actually address what the claim asserts? (Not just "link exists" but "link is substantively relevant.")
  • Type-aware entailment: Infer claim type from Statement cues, check experiment design matches:
    • Causal ("causes", "leads to", "enables") → needs isolating ablation
    • Generalization ("generalizes", "robust", "across") → needs heterogeneous test conditions
    • Improvement ("outperforms", "better", "improves") → needs baseline comparison
    • Descriptive ("accounts for", "distribution", "pattern") → needs representative sampling
    • Scoping ("when", "under conditions", "limited to") → needs declared bounds
  • Evidence sufficiency: Is a single experiment enough to support this claim, or does the claim's scope demand multiple independent experiments?

Scoring anchors:

  • 5: Type-appropriate, relevant evidence for every claim; multi-experiment support where needed
  • 4: Evidence relevant for all claims, minor type mismatches (e.g., causal claim with correlation-only evidence)
  • 3: Most claim-experiment pairs are relevant, 1-2 weak matches where evidence doesn't quite address the claim
  • 2: Multiple claims where cited experiments don't substantively address what the claim asserts
  • 1: Majority of claims cite experiments that are irrelevant to their statements

D2. Falsifiability Quality

For each claim's Falsification criteria field:

  • Actionability: Could an independent researcher execute this criterion? Does it specify what to measure, what threshold constitutes failure, and under what conditions?
  • Non-triviality: Is the criterion non-tautological? ("If the method doesn't work" is trivial. "Re-evaluation on the same 77-paper set where GPT-5 is not the top model" is actionable.)
  • Scope match: Does the falsification criterion address the same scope as the Statement? (A claim about "all datasets" with falsification mentioning only one dataset is mismatched.)
  • Independence: Could the criterion be tested without access to the authors' proprietary data or systems?

Scoring anchors:

  • 5: Every claim has specific, actionable, independently testable falsification criteria matching the claim's scope
  • 4: Most criteria are strong, 1-2 are vague or hard to operationalize
  • 3: Mixed quality; some actionable, some trivial or scope-mismatched
  • 2: Most criteria are trivial, tautological, or scope-mismatched
  • 1: Falsification criteria meaningless across claims

D3. Scope Calibration

  • Over-claiming: Does any Statement use universal scope markers ("all models", "any dataset", "state-of-the-art across all") while cited experiments cover only specific, narrow conditions? The gap must be substantial.
  • Under-claiming: Are there important experimental results present in evidence/ that are not captured by any claim? (Evidence without a corresponding claim.)
  • Assumption explicitness: Are key assumptions stated in problem.md (Assumptions section) or constraints.md? Are there unstated assumptions implied by the experimental design?
  • Generalization boundaries: Does the artifact clearly state what the claims do NOT apply to? Check constraints.md and limitations in the exploration tree.
  • Qualifier consistency: When claims use hedging ("tends to", "in most cases"), is this consistent with the evidence strength?

Scoring anchors:

  • 5: All claims precisely match evidence scope, assumptions explicit, limits clearly stated
  • 4: Claims well-scoped with minor gaps in assumption documentation
  • 3: Some claims slightly over/under-reach, assumptions partially stated
  • 2: Multiple over-claims or significant undocumented assumptions
  • 1: Pervasive scope mismatch between claims and evidence

D4. Argument Coherence

  • Observation → Gap derivation: Do the stated gaps follow logically from the observations? Or are they asserted without connection?
  • Gap → Insight connection: Does the key insight in problem.md address the identified gaps?
  • Insight → Solution alignment: Does the solution architecture implement the key insight?
  • Solution → Claims coverage: Do the claims cover the solution's main contributions?
  • Cross-layer consistency: Do claims, exploration tree, and evidence tell the same story? Flag contradictions.
  • Narrative completeness: Are there motivating questions from problem.md that are neither answered nor explicitly deferred?
  • Gap coverage: For each gap in problem.md, is there at least one claim that substantively addresses it? Flag gaps that are motivated but never resolved.

Scoring anchors:

  • 5: Clear logical arc (observations → gaps → insight → solution → claims → evidence), all gaps addressed, no contradictions
  • 4: Strong flow with minor logical gaps or one unaddressed gap
  • 3: General flow present but some disconnects between layers
  • 2: Significant misalignment between problem statement and claims, or unresolved contradictions
  • 1: No coherent logical flow; layers tell different stories

D5. Exploration Integrity

  • Dead-end quality: Is the failure_mode specific enough to be actionable? ("Didn't work" is bad. "Divergence after 1000 steps due to gradient explosion" is good.) Is the lesson a genuine transferable insight?
  • Decision rationale quality: Do rationales explain WHY the chosen path was preferred over alternatives? Are alternatives real alternatives or strawmen?
  • Rebutted-branch consistency: Does any claim advocate an approach marked as dead_end or pivot in the tree? (This is a logical contradiction.)
  • Exploration breadth: For the paper's main design choices, were at least 2 alternatives considered and documented?
  • Honesty signal: Does the tree document genuine negative results, or does it read like a post-hoc justification? A tree with zero dead-ends or only trivial failures is suspicious.

Scoring anchors:

  • 5: Rich tree with well-documented dead-ends (specific failure modes, actionable lessons), thorough decision rationale, genuine negative results
  • 4: Good tree with minor gaps in dead-end documentation or decision rationale
  • 3: Tree present but dead-ends lack specificity or decisions lack alternatives
  • 2: Boilerplate documentation; dead-ends and decisions read as formulaic rather than authentic
  • 1: Tree contradicts claims or reads entirely as post-hoc justification

D6. Methodological Rigor

  • Baseline adequacy: Are the right things being compared? Are baselines recent and relevant? Flag experiments with "no baseline" for comparative claims.
  • Ablation coverage: For claims involving multiple components, does at least one experiment isolate individual contributions?
  • Statistical reporting: Do experiments mention variance, confidence intervals, number of runs, or statistical tests? Flag single-run results for quantitative claims.
  • Metric-claim alignment: Does the metric actually measure what the claim asserts? (A claim about "ge

Truncated for display — read the full file on GitHub.

Related Skills

View on GitHub
GitHub Stars14.5k
CategoryEducation
Updated1mo ago
Forks1.0k

Languages

TeX

Trust signals

100/100

From repository metadata: license, adoption, age and documentation. Not a code audit — see the Safety scan above for what the skill file itself contains.

No cautions