evaluation-framework
Weighted scoring, rubrics, thresholds, agent-action utility by expected gain and cost. Use for quality gates or deciding if a tool call or dispatch is worth it.
Install / Use
npx skills add athola/claude-night-market --skill evaluation-frameworkInstalls into whichever agent you are using.
SKILL.md
Installable skill definition
Quality Score
Category
OperationsSupported Platforms
Tags
Our assessment of evaluation-framework
evaluation-framework scores 86/100 on our quality scale, 496th of 763 Operations skills we index.
Its SKILL.md is 5.8 KB long, well organised into 19 sections with 7 code examples: a solid amount of guidance for an agent.
It has 338 GitHub stars, a meaningful sign that others use it.
Maintenance, license and trust
- The repository was last updated about 2 months ago, so evaluation-framework is actively maintained.
- It is released under the MIT license, a permissive license that allows use, modification and commercial use with attribution.
- Its trust signals score 100/100, with no cautions. These come from repository metadata, not a code audit — read the skill file before letting an agent act on it.
Safety scan
No issues foundOur scan of the whole file found no instruction hijacking, hidden characters, credential access, data exfiltration or destructive commands.
Automated pattern scan on 2026-11-22. It catches known dangerous patterns, not every risk — read a skill before letting an agent act on it.
evaluation-framework compared with similar skills
All 4 of these similar skills score higher than evaluation-framework; compare them before choosing.
| Skill | Score | Stars | Updated | Format |
|---|---|---|---|---|
| evaluation-framework (this skill)by athola | 86 | 338 | 59d ago | SKILL.md |
| algorithmic-artby anthropics | 100 | 177.9k | 2mo ago | SKILL.md |
| pptxby anthropics | 100 | 177.9k | 2mo ago | SKILL.md |
| prospectingby coreyhaines31 | 100 | 60.1k | 11d ago | SKILL.md |
| ad-creativeby coreyhaines31 | 100 | 60.1k | 11d ago | SKILL.md |
Frequently asked questions
- How do I install evaluation-framework?
- Run
npx skills add athola/claude-night-market --skill evaluation-framework. The install tabs above show the steps for each supported agent. - Which AI agents does evaluation-framework work with?
- It is written for Universal, as a SKILL.md file. Other agents that read the same format can often use it too.
- Is evaluation-framework safe to use?
- Our scan of the whole file found no instruction hijacking, hidden characters, credential access, data exfiltration or destructive commands. It is MIT-licensed and scores 100/100 on trust signals. Skills are instructions an agent will follow, so read the file before installing it and do not approve commands you do not understand.
- Is evaluation-framework still maintained?
- The repository was last updated about 2 months ago, so evaluation-framework is actively maintained.
Skill content
View source on GitHubname: evaluation-framework description: Weighted scoring, rubrics, thresholds, agent-action utility by expected gain and cost. Use for quality gates or deciding if a tool call or dispatch is worth it. alwaysApply: false category: infrastructure tags:
- evaluation
- scoring
- decision-making
- metrics
- quality
dependencies: []
provides:
infrastructure:
- weighted-scoring
- threshold-decisions
- evaluation-patterns patterns:
- criteria-definition
- scoring-methodology
- decision-logic usage_patterns:
- quality-evaluation
- scoring-systems
- decision-frameworks
- rubric-design complexity: beginner estimated_tokens: 550 progressive_loading: true modules:
- modules/scoring-patterns.md
- modules/decision-thresholds.md
- modules/evaluation-rubric.md
- modules/multi-metric-evaluation-methodology.md
- modules/quality-metrics.md
- modules/utility.md
- modules/state-builder.md
- modules/gain.md
- modules/step-cost.md
- modules/uncertainty.md
- modules/redundancy.md
- modules/action-selector.md
- modules/integration.md
Evaluation Framework
Overview
A generic framework for weighted scoring and threshold-based decision making. Provides reusable patterns for evaluating any artifact against configurable criteria with consistent scoring methodology.
This framework abstracts the common pattern of: define criteria → assign weights → score against criteria → apply thresholds → make decisions.
When To Use
- Implementing quality gates or evaluation rubrics
- Building scoring systems for artifacts, proposals, or submissions
- Need consistent evaluation methodology across different domains
- Want threshold-based automated decision making
- Creating assessment tools with weighted criteria
When NOT To Use
- Simple pass/fail without scoring needs
Core Pattern
1. Define Criteria
criteria:
- name: criterion_name
weight: 0.30 # 30% of total score
description: What this measures
scoring_guide:
90-100: Exceptional
70-89: Strong
50-69: Acceptable
30-49: Weak
0-29: Poor
Verification: Run the command with --help flag to verify availability.
2. Score Each Criterion
scores = {
"criterion_1": 85, # Out of 100
"criterion_2": 92,
"criterion_3": 78,
}
Verification: Run the command with --help flag to verify availability.
3. Calculate Weighted Total
total = sum(score * weights[criterion] for criterion, score in scores.items())
# Example: (85 × 0.30) + (92 × 0.40) + (78 × 0.30) = 85.5
Verification: Run the command with --help flag to verify availability.
4. Apply Decision Thresholds
thresholds:
80-100: Accept with priority
60-79: Accept with conditions
40-59: Review required
20-39: Reject with feedback
0-19: Reject
A score that gates a decision becomes the target. Agent output drifts
from a visible proxy as the work grows, and standard maintainability
metrics explain little of why agent code is harder to extend
(docs/knowledge-corpus/decoherence.md). Pair each gating score with
one check the score does not include: a held-out test, a run on real
input, or a reviewer reading the output.
Verification: Run the command with --help flag to verify availability.
Quick Start
Define Your Evaluation
- Identify criteria: What aspects matter for your domain?
- Assign weights: Which criteria are most important? (sum to 1.0)
- Create scoring guides: What does each score range mean?
- Set thresholds: What total scores trigger which decisions?
Example: Code Review Evaluation
criteria:
correctness: {weight: 0.40, description: Does code work as intended?}
maintainability: {weight: 0.25, description: Is it readable?}
performance: {weight: 0.20, description: Meets performance needs?}
testing: {weight: 0.15, description: Tests detailed?}
thresholds:
85-100: Approve immediately
70-84: Approve with minor feedback
50-69: Request changes
0-49: Reject, major issues
Verification: Run pytest -v to verify tests pass.
Evaluation Workflow
**Verification:** Run the command with `--help` flag to verify availability.
1. Review artifact against each criterion
2. Assign 0-100 score for each criterion
3. Calculate: total = Σ(score × weight)
4. Compare total to thresholds
5. Take action based on threshold range
Verification: Run the command with --help flag to verify availability.
Common Use Cases
Quality Gates: Code review, PR approval, release readiness Content Evaluation: Document quality, knowledge intake, skill assessment Resource Allocation: Backlog prioritization, investment decisions, triage
Integration Pattern
# In your skill's frontmatter
dependencies: [leyline:evaluation-framework]
Verification: Run the command with --help flag to verify availability.
Then customize the framework for your domain:
- Define domain-specific criteria
- Set appropriate weights for your context
- Establish meaningful thresholds
- Document what each score range means
Detailed Resources
- Scoring Patterns: See
modules/scoring-patterns.mdfor detailed methodology - Decision Thresholds: See
modules/decision-thresholds.mdfor threshold design - Utility Scoring: Load
modules/utility.mdwhen deciding whether to dispatch an agent, make a tool call, or stop. It scores each action as gain minus weighted cost, uncertainty, and redundancy, and links its component modules (state-builder,gain,step-cost,uncertainty,redundancy,action-selector,integration)
Exit Criteria
- [ ] Criteria defined with clear descriptions
- [ ] Weights assigned and sum to 1.0
- [ ] Scoring guides documented for each criterion
- [ ] Thresholds mapped to specific actions
- [ ] Each gating score names its check outside the score
- [ ] Evaluation process documented and reproducible
Related Skills
algorithmic-art
177.9kCreating algorithmic art using p5.js with seeded randomness and interactive parameter exploration. Use this when users request creating art using code, generative art, algorithmic art, flow fields, or particle systems.
pptx
177.9kUse this skill any time a .pptx or .potx file is involved in any way — as input, output, or both. This includes: creating slide decks, pitch decks, or presentations; reading, parsing, or extracting text from any .pptx or .potx file (even if the extracted content will be used elsewhere, like in an em…
prospecting
60.1kWhen the user wants to find and qualify prospects for outbound, or asks who to target first, which signal to prioritize, or where to spot buyers with the problem: buying signals, enrichment, verification, account research.
ad-creative
60.1kWhen the user wants to write or judge ad creative, or asks which ad, hook, or angle to run or kill: headlines, primary text, and variations for Google, Meta, LinkedIn, and TikTok, fresh angles when creative fatigues. Test hypotheses name the metric, comparison, and expected direction.
Languages
Trust signals
From repository metadata: license, adoption, age and documentation. Not a code audit — see the Safety scan above for what the skill file itself contains.
