SkillAgentSearch skills...

scholar-evaluation

Apply the ScholarEval framework to systematically evaluate scholarly work across quality dimensions (novelty, rigor, significance, and clarity). Use to critique academic papers, research proposals, literature reviews, or grant applications and produce a structured, criterion-based assessment.

Install / Use

npx skills add brycewang-stanford/Auto-Empirical-Research-Skills --skill scholar-evaluation

Installs into whichever agent you are using.

About this skill
📄

SKILL.md

Installable skill definition

Quality Score

94/100

Supported Platforms

Universal

Our assessment of scholar-evaluation

scholar-evaluation scores 94/100 on our quality scale, 64th of 421 Education & Research skills we index (top 16%).

Its SKILL.md is 12 KB long, well organised into 19 sections with 2 code examples: a thorough specification that gives an agent plenty to work with.

With 4,360 GitHub stars, it is one of the more widely adopted skills in the catalogue.

Substance
30/30
Structure
18/20
Description
15/15
Adoption
15/20
Freshness
15/15

Maintenance, license and trust

  • The repository was last updated about 2 months ago, so scholar-evaluation is actively maintained.
  • Our last check on 2026-11-16 found the source still online.
  • No license is declared. By default that means all rights are reserved: you can read it, but reusing or redistributing it is not clearly permitted. Ask the author before building on it commercially.
  • Its trust signals score 88/100, with 1 caution from licensing, adoption, age or documentation. These come from repository metadata, not a code audit — read the skill file before letting an agent act on it.

Safety scan

No issues found

Our scan of the whole file found no instruction hijacking, hidden characters, credential access, data exfiltration or destructive commands. An AI review of the same text found nothing harmful.

AI review by kimi-k2.7-code on 2026-09-29. Automated pattern scan on 2026-09-28. It catches known dangerous patterns, not every risk — read a skill before letting an agent act on it.

scholar-evaluation compared with similar skills

All 4 of these similar skills score higher than scholar-evaluation; compare them before choosing.

SkillScoreStarsUpdatedFormat
scholar-evaluation (this skill)by brycewang-stanford944.4k54d agoSKILL.md
last30days-skillby mvanhorn10068.0ktodayCLAUDE.md
x402by browser-use100121.0k3d agoSKILL.md
algorithmic-artby anthropics100177.9k55d agoSKILL.md
pptxby anthropics100177.9k55d agoSKILL.md

Frequently asked questions

How do I install scholar-evaluation?
Run npx skills add brycewang-stanford/Auto-Empirical-Research-Skills --skill scholar-evaluation. The install tabs above show the steps for each supported agent.
Which AI agents does scholar-evaluation work with?
It is written for Universal, as a SKILL.md file. Other agents that read the same format can often use it too.
Is scholar-evaluation safe to use?
Our scan of the whole file found no instruction hijacking, hidden characters, credential access, data exfiltration or destructive commands. An AI review of the same text found nothing harmful. It declares no license and scores 88/100 on trust signals. Skills are instructions an agent will follow, so read the file before installing it and do not approve commands you do not understand.
Is scholar-evaluation still maintained?
The repository was last updated about 2 months ago, so scholar-evaluation is actively maintained.

name: scholar-evaluation description: "Apply the ScholarEval framework to systematically evaluate scholarly work across quality dimensions (novelty, rigor, significance, and clarity). Use to critique academic papers, research proposals, literature reviews, or grant applications and produce a structured, criterion-based assessment." allowed-tools: [Read, Write, Edit, Bash]

<!-- ╔══════════════════════════════════════════════════════════════╗ ║ 本文件为开源 Skill 原始文档,收录仅供学习与研究参考 ║ ║ CoPaper.AI 收集整理 | https://copaper.ai ║ ╚══════════════════════════════════════════════════════════════╝ 来源仓库: https://github.com/K-Dense-AI/claude-scientific-writer 项目名称: claude-scientific-writer 开源协议: MIT License 收录日期: 2026-04-02 声明: 本文件版权归原作者所有。此处收录旨在为社会科学实证研究者 提供 AI Agent Skills 的集中参考。如有侵权,请联系删除。 -->

Scholar Evaluation

Overview

Apply the ScholarEval framework to systematically evaluate scholarly and research work. This skill provides structured evaluation methodology based on peer-reviewed research assessment criteria, enabling comprehensive analysis of academic papers, research proposals, literature reviews, and scholarly writing across multiple quality dimensions.

When to Use This Skill

Use this skill when:

  • Evaluating research papers for quality and rigor
  • Assessing literature review comprehensiveness and quality
  • Reviewing research methodology design
  • Scoring data analysis approaches
  • Evaluating scholarly writing and presentation
  • Providing structured feedback on academic work
  • Benchmarking research quality against established criteria
  • Assessing publication readiness for target venues
  • Providing quantitative evaluation to complement qualitative peer review

Visual Enhancement with Scientific Schematics

When creating documents with this skill, always consider adding scientific diagrams and schematics to enhance visual communication.

If your document does not already contain schematics or diagrams:

  • Use the scientific-schematics skill to generate AI-powered publication-quality diagrams
  • Simply describe your desired diagram in natural language
  • Nano Banana Pro will automatically generate, review, and refine the schematic

For new documents: Scientific schematics should be generated by default to visually represent key concepts, workflows, architectures, or relationships described in the text.

How to generate schematics:

python scripts/generate_schematic.py "your diagram description" -o figures/output.png

The AI will automatically:

  • Create publication-quality images with proper formatting
  • Review and refine through multiple iterations
  • Ensure accessibility (colorblind-friendly, high contrast)
  • Save outputs in the figures/ directory

When to add schematics:

  • Evaluation framework diagrams
  • Quality assessment criteria decision trees
  • Scholarly workflow visualizations
  • Assessment methodology flowcharts
  • Scoring rubric visualizations
  • Evaluation process diagrams
  • Any complex concept that benefits from visualization

For detailed guidance on creating schematics, refer to the scientific-schematics skill documentation.


Evaluation Workflow

Step 1: Initial Assessment and Scope Definition

Begin by identifying the type of scholarly work being evaluated and the evaluation scope:

Work Types:

  • Full research paper (empirical, theoretical, or review)
  • Research proposal or protocol
  • Literature review (systematic, narrative, or scoping)
  • Thesis or dissertation chapter
  • Conference abstract or short paper

Evaluation Scope:

  • Comprehensive (all dimensions)
  • Targeted (specific aspects like methodology or writing)
  • Comparative (benchmarking against other work)

Ask the user to clarify if the scope is ambiguous.

Step 2: Dimension-Based Evaluation

Systematically evaluate the work across the ScholarEval dimensions. For each applicable dimension, assess quality, identify strengths and weaknesses, and provide scores where appropriate.

Refer to references/evaluation_framework.md for detailed criteria and rubrics for each dimension.

Core Evaluation Dimensions:

  1. Problem Formulation & Research Questions

    • Clarity and specificity of research questions
    • Theoretical or practical significance
    • Feasibility and scope appropriateness
    • Novelty and contribution potential
  2. Literature Review

    • Comprehensiveness of coverage
    • Critical synthesis vs. mere summarization
    • Identification of research gaps
    • Currency and relevance of sources
    • Proper contextualization
  3. Methodology & Research Design

    • Appropriateness for research questions
    • Rigor and validity
    • Reproducibility and transparency
    • Ethical considerations
    • Limitations acknowledgment
  4. Data Collection & Sources

    • Quality and appropriateness of data
    • Sample size and representativeness
    • Data collection procedures
    • Source credibility and reliability
  5. Analysis & Interpretation

    • Appropriateness of analytical methods
    • Rigor of analysis
    • Logical coherence
    • Alternative explanations considered
    • Results-claims alignment
  6. Results & Findings

    • Clarity of presentation
    • Statistical or qualitative rigor
    • Visualization quality
    • Interpretation accuracy
    • Implications discussion
  7. Scholarly Writing & Presentation

    • Clarity and organization
    • Academic tone and style
    • Grammar and mechanics
    • Logical flow
    • Accessibility to target audience
  8. Citations & References

    • Citation completeness
    • Source quality and appropriateness
    • Citation accuracy
    • Balance of perspectives
    • Adherence to citation standards

Step 3: Scoring and Rating

For each evaluated dimension, provide:

Qualitative Assessment:

  • Key strengths (2-3 specific points)
  • Areas for improvement (2-3 specific points)
  • Critical issues (if any)

Quantitative Scoring (Optional): Use a 5-point scale where applicable:

  • 5: Excellent - Exemplary quality, publishable in top venues
  • 4: Good - Strong quality with minor improvements needed
  • 3: Adequate - Acceptable quality with notable areas for improvement
  • 2: Needs Improvement - Significant revisions required
  • 1: Poor - Fundamental issues requiring major revision

To calculate aggregate scores programmatically, use scripts/calculate_scores.py.

Step 4: Synthesize Overall Assessment

Provide an integrated evaluation summary:

  1. Overall Quality Assessment - Holistic judgment of the work's scholarly merit
  2. Major Strengths - 3-5 key strengths across dimensions
  3. Critical Weaknesses - 3-5 primary areas requiring attention
  4. Priority Recommendations - Ranked list of improvements by impact
  5. Publication Readiness (if applicable) - Assessment of suitability for target venues

Step 5: Provide Actionable Feedback

Transform evaluation findings into constructive, actionable feedback:

Feedback Structure:

  • Specific - Reference exact sections, paragraphs, or page numbers
  • Actionable - Provide concrete suggestions for improvement
  • Prioritized - Rank recommendations by importance and feasibility
  • Balanced - Acknowledge strengths while addressing weaknesses
  • Evidence-based - Ground feedback in evaluation criteria

Feedback Format Options:

  • Structured report with dimension-by-dimension analysis
  • Annotated comments mapped to specific document sections
  • Executive summary with key findings and recommendations
  • Comparative analysis against benchmark standards

Step 6: Contextual Considerations

Adjust evaluation approach based on:

Stage of Development:

  • Early draft: Focus on conceptual and structural issues
  • Advanced draft: Focus on refinement and polish
  • Final submission: Comprehensive quality check

Purpose and Venue:

  • Journal article: High standards for rigor and contribution
  • Conference paper: Balance novelty with presentation clarity
  • Student work: Educational feedback with developmental focus
  • Grant proposal: Emphasis on feasibility and impact

Discipline-Specific Norms:

  • STEM fields: Emphasis on reproducibility and statistical rigor
  • Social sciences: Balance quantitative and qualitative standards
  • Humanities: Focus on argumentation and scholarly interpretation

Resources

references/evaluation_framework.md

Detailed evaluation criteria, rubrics, and quality indicators for each ScholarEval dimension. Load this reference when conducting evaluations to access specific assessment guidelines and scoring rubrics.

Search patterns for quick access:

  • "Problem Formulation criteria"
  • "Literature Review rubric"
  • "Methodology assessment"
  • "Data quality indicators"
  • "Analysis rigor standards"
  • "Writing quality checklist"

scripts/calculate_scores.py

Python script for calculating aggregate evaluation scores from dimension-level ratings. Supports weighted averaging, threshold analysis, and score visualization.

Usage:

python scripts/calculate_scores.py --scores <dimension_scores.json> --output <report.txt>

Best Practices

  1. Maintain Objectivity - Base evaluations on established criteria, not personal preferences
  2. Be Comprehensive - Evaluate all applicable dimensions systematically
  3. Provide Evidence - Support assessments with specific examples from the work
  4. Stay Constructive - Frame weaknesses as opportunities for improvement
  5. Consider Context - Adjust expectations based on work stage and purpose
  6. Document Rationale - Explain the reasoning behind assessments and scores
  7. Encourage Strengths - Explicitly acknowledge what the work does well
  8. Prioritize Feedback - Focus on high-impact improvements first

Example Evaluation Workflow

User Request: "Evaluate this research paper on machine learning for drug discovery"

Response Process:

  1. Identify work type (empirical research paper) and scope (comprehensive evaluation)
  2. Load references/evaluation_framework.md for detailed criteria
  3. Systematically assess each dimension:
    • Problem formulation: Clear research question about ML model performance
    • Literature review: Comprehensive coverage of recent ML and drug discovery work
    • Methodology: Appropriate deep learning architecture with validation procedures
    • [Continue through all dimensions...]
  4. Calculate dimension scores and overall assessment
  5. Synthesize findings into structured report highlighting:
    • Strong methodology and reproducible code
    • Needs more diverse dataset evaluation
    • Writing could improve clarity in results section
  6. Provide prioritized recommendations with specific suggestions

Integration with Scientific Writer

This skill integrates seamlessly with the scientific writer workflow:

After Paper Generation:

  • Use Scholar Evaluation as an alternative or complement to peer review
  • Generate SCHOLAR_EVALUATION.md alongside PEER_REVIEW.md
  • Provide quantitative scores to track improvement across revisions

During Revision:

  • Re-evaluate specific dimensions after addressing feedback
  • Track score improvements over multiple versions
  • Identify persistent weaknesses requiring attention

Publication Preparation:

  • Assess readiness for target journal/conference
  • Identify gaps before submission
  • Benchmark against publication standards

Notes

  • Evaluation rigor should match the work's purpose and stage
  • Some dimensions may not apply to all work types (e.g., data collection for purely theoretical papers)
  • Cultural and disciplinary differences in scholarly norms should be considered
  • This framework complements, not replaces, domain-specific expertise
  • Use in combination with peer-review skill for comprehensive assessment

Citation

This skill is based on the ScholarEval framework introduced in:

**Moussa, H. N., Da Silva, P. Q., Adu-Ampratwum, D., East, A., Lu, Z., Puccetti, N., Xue, M., Sun, H., Majumder, B. P., &

Truncated for display — read the full file on GitHub.

Related Skills

View on GitHub
GitHub Stars4.4k
CategoryEducation
Updated1mo ago
Forks527

Languages

Stata

Trust signals

88/100

From repository metadata: license, adoption, age and documentation. Not a code audit — see the Safety scan above for what the skill file itself contains.

1 medium