SkillAgentSearch skills...

evaluation-framework

Weighted scoring, rubrics, thresholds, agent-action utility by expected gain and cost. Use for quality gates or deciding if a tool call or dispatch is worth it.

Install / Use

npx skills add athola/claude-night-market --skill evaluation-framework

Installs into whichever agent you are using.

About this skill
📄

SKILL.md

Installable skill definition

Quality Score

86/100

Category

Operations

Supported Platforms

Universal

Tags

Our assessment of evaluation-framework

evaluation-framework scores 86/100 on our quality scale, 496th of 763 Operations skills we index.

Its SKILL.md is 5.8 KB long, well organised into 19 sections with 7 code examples: a solid amount of guidance for an agent.

It has 338 GitHub stars, a meaningful sign that others use it.

Substance
26/30
Structure
20/20
Description
15/15
Adoption
11/20
Freshness
15/15

Maintenance, license and trust

  • The repository was last updated about 2 months ago, so evaluation-framework is actively maintained.
  • It is released under the MIT license, a permissive license that allows use, modification and commercial use with attribution.
  • Its trust signals score 100/100, with no cautions. These come from repository metadata, not a code audit — read the skill file before letting an agent act on it.

Safety scan

No issues found

Our scan of the whole file found no instruction hijacking, hidden characters, credential access, data exfiltration or destructive commands.

Automated pattern scan on 2026-11-22. It catches known dangerous patterns, not every risk — read a skill before letting an agent act on it.

evaluation-framework compared with similar skills

All 4 of these similar skills score higher than evaluation-framework; compare them before choosing.

SkillScoreStarsUpdatedFormat
evaluation-framework (this skill)by athola8633859d agoSKILL.md
algorithmic-artby anthropics100177.9k2mo agoSKILL.md
pptxby anthropics100177.9k2mo agoSKILL.md
prospectingby coreyhaines3110060.1k11d agoSKILL.md
ad-creativeby coreyhaines3110060.1k11d agoSKILL.md

Frequently asked questions

How do I install evaluation-framework?
Run npx skills add athola/claude-night-market --skill evaluation-framework. The install tabs above show the steps for each supported agent.
Which AI agents does evaluation-framework work with?
It is written for Universal, as a SKILL.md file. Other agents that read the same format can often use it too.
Is evaluation-framework safe to use?
Our scan of the whole file found no instruction hijacking, hidden characters, credential access, data exfiltration or destructive commands. It is MIT-licensed and scores 100/100 on trust signals. Skills are instructions an agent will follow, so read the file before installing it and do not approve commands you do not understand.
Is evaluation-framework still maintained?
The repository was last updated about 2 months ago, so evaluation-framework is actively maintained.

name: evaluation-framework description: Weighted scoring, rubrics, thresholds, agent-action utility by expected gain and cost. Use for quality gates or deciding if a tool call or dispatch is worth it. alwaysApply: false category: infrastructure tags:

  • evaluation
  • scoring
  • decision-making
  • metrics
  • quality dependencies: [] provides: infrastructure:
    • weighted-scoring
    • threshold-decisions
    • evaluation-patterns patterns:
    • criteria-definition
    • scoring-methodology
    • decision-logic usage_patterns:
  • quality-evaluation
  • scoring-systems
  • decision-frameworks
  • rubric-design complexity: beginner estimated_tokens: 550 progressive_loading: true modules:
  • modules/scoring-patterns.md
  • modules/decision-thresholds.md
  • modules/evaluation-rubric.md
  • modules/multi-metric-evaluation-methodology.md
  • modules/quality-metrics.md
  • modules/utility.md
  • modules/state-builder.md
  • modules/gain.md
  • modules/step-cost.md
  • modules/uncertainty.md
  • modules/redundancy.md
  • modules/action-selector.md
  • modules/integration.md

Evaluation Framework

Overview

A generic framework for weighted scoring and threshold-based decision making. Provides reusable patterns for evaluating any artifact against configurable criteria with consistent scoring methodology.

This framework abstracts the common pattern of: define criteria → assign weights → score against criteria → apply thresholds → make decisions.

When To Use

  • Implementing quality gates or evaluation rubrics
  • Building scoring systems for artifacts, proposals, or submissions
  • Need consistent evaluation methodology across different domains
  • Want threshold-based automated decision making
  • Creating assessment tools with weighted criteria

When NOT To Use

  • Simple pass/fail without scoring needs

Core Pattern

1. Define Criteria

criteria:
  - name: criterion_name
    weight: 0.30          # 30% of total score
    description: What this measures
    scoring_guide:
      90-100: Exceptional
      70-89: Strong
      50-69: Acceptable
      30-49: Weak
      0-29: Poor

Verification: Run the command with --help flag to verify availability.

2. Score Each Criterion

scores = {
    "criterion_1": 85,  # Out of 100
    "criterion_2": 92,
    "criterion_3": 78,
}

Verification: Run the command with --help flag to verify availability.

3. Calculate Weighted Total

total = sum(score * weights[criterion] for criterion, score in scores.items())
# Example: (85 × 0.30) + (92 × 0.40) + (78 × 0.30) = 85.5

Verification: Run the command with --help flag to verify availability.

4. Apply Decision Thresholds

thresholds:
  80-100: Accept with priority
  60-79: Accept with conditions
  40-59: Review required
  20-39: Reject with feedback
  0-19: Reject

A score that gates a decision becomes the target. Agent output drifts from a visible proxy as the work grows, and standard maintainability metrics explain little of why agent code is harder to extend (docs/knowledge-corpus/decoherence.md). Pair each gating score with one check the score does not include: a held-out test, a run on real input, or a reviewer reading the output. Verification: Run the command with --help flag to verify availability.

Quick Start

Define Your Evaluation

  1. Identify criteria: What aspects matter for your domain?
  2. Assign weights: Which criteria are most important? (sum to 1.0)
  3. Create scoring guides: What does each score range mean?
  4. Set thresholds: What total scores trigger which decisions?

Example: Code Review Evaluation

criteria:
  correctness: {weight: 0.40, description: Does code work as intended?}
  maintainability: {weight: 0.25, description: Is it readable?}
  performance: {weight: 0.20, description: Meets performance needs?}
  testing: {weight: 0.15, description: Tests detailed?}

thresholds:
  85-100: Approve immediately
  70-84: Approve with minor feedback
  50-69: Request changes
  0-49: Reject, major issues

Verification: Run pytest -v to verify tests pass.

Evaluation Workflow

**Verification:** Run the command with `--help` flag to verify availability.
1. Review artifact against each criterion
2. Assign 0-100 score for each criterion
3. Calculate: total = Σ(score × weight)
4. Compare total to thresholds
5. Take action based on threshold range

Verification: Run the command with --help flag to verify availability.

Common Use Cases

Quality Gates: Code review, PR approval, release readiness Content Evaluation: Document quality, knowledge intake, skill assessment Resource Allocation: Backlog prioritization, investment decisions, triage

Integration Pattern

# In your skill's frontmatter
dependencies: [leyline:evaluation-framework]

Verification: Run the command with --help flag to verify availability.

Then customize the framework for your domain:

  • Define domain-specific criteria
  • Set appropriate weights for your context
  • Establish meaningful thresholds
  • Document what each score range means

Detailed Resources

  • Scoring Patterns: See modules/scoring-patterns.md for detailed methodology
  • Decision Thresholds: See modules/decision-thresholds.md for threshold design
  • Utility Scoring: Load modules/utility.md when deciding whether to dispatch an agent, make a tool call, or stop. It scores each action as gain minus weighted cost, uncertainty, and redundancy, and links its component modules (state-builder, gain, step-cost, uncertainty, redundancy, action-selector, integration)

Exit Criteria

  • [ ] Criteria defined with clear descriptions
  • [ ] Weights assigned and sum to 1.0
  • [ ] Scoring guides documented for each criterion
  • [ ] Thresholds mapped to specific actions
  • [ ] Each gating score names its check outside the score
  • [ ] Evaluation process documented and reproducible

Related Skills

View on GitHub
GitHub Stars338
CategoryOperations
Updated1mo ago
Forks35

Languages

Python

Trust signals

100/100

From repository metadata: license, adoption, age and documentation. Not a code audit — see the Safety scan above for what the skill file itself contains.

No cautions
evaluation-framework — Universal Skill: Install & Safety Check | SkillAgent