SkillAgentSearch skills...

Building Paper Screening Rubrics

Collaboratively build and refine paper screening rubrics through brainstorming, test-driven development, and iterative feedback

Install / Use

npx skills add brycewang-stanford/Auto-Empirical-Research-Skills --skill building-screening-rubrics

Installs into whichever agent you are using.

About this skill
📄

SKILL.md

Installable skill definition

Quality Score

95/100

Category

Automation

Supported Platforms

Universal

Our assessment of Building Paper Screening Rubrics

Building Paper Screening Rubrics scores 95/100 on our quality scale, 319th of 2,780 Automation skills we index (top 12%).

Its SKILL.md is 17 KB long, well organised into 30 sections with 23 code examples: a thorough specification that gives an agent plenty to work with.

With 4,360 GitHub stars, it is one of the more widely adopted skills in the catalogue.

Substance
30/30
Structure
20/20
Description
15/15
Adoption
15/20
Freshness
15/15

Maintenance, license and trust

  • The repository was last updated about 2 months ago, so Building Paper Screening Rubrics is actively maintained.
  • Our last check on 2026-11-15 found the source still online.
  • No license is declared. By default that means all rights are reserved: you can read it, but reusing or redistributing it is not clearly permitted. Ask the author before building on it commercially.
  • Its trust signals score 88/100, with 1 caution from licensing, adoption, age or documentation. These come from repository metadata, not a code audit — read the skill file before letting an agent act on it.

Safety scan

No issues found

Our scan of the whole file found no instruction hijacking, hidden characters, credential access, data exfiltration or destructive commands. An AI review of the same text found nothing harmful.

AI review by kimi-k2.7-code on 2026-09-29. Automated pattern scan on 2026-09-29. It catches known dangerous patterns, not every risk — read a skill before letting an agent act on it.

Building Paper Screening Rubrics compared with similar skills

All 4 of these similar skills score higher than Building Paper Screening Rubrics; compare them before choosing.

SkillScoreStarsUpdatedFormat
Building Paper Screening Rubrics (this skill)by brycewang-stanford954.4k54d agoSKILL.md
Agent-Reachby Panniantong100112.3k8d agoCLAUDE.md
rufloby ruvnet10081.9ktodayCLAUDE.md
Claude-Code-Game-Studiosby Donchitos10039.7k7d agoCLAUDE.md
Scraplingby D4Vinci100103.6k2d agoMCP Server

Frequently asked questions

How do I install Building Paper Screening Rubrics?
Run npx skills add brycewang-stanford/Auto-Empirical-Research-Skills --skill "Building Paper Screening Rubrics". The install tabs above show the steps for each supported agent.
Which AI agents does Building Paper Screening Rubrics work with?
It is written for Universal, as a SKILL.md file. Other agents that read the same format can often use it too.
Is Building Paper Screening Rubrics safe to use?
Our scan of the whole file found no instruction hijacking, hidden characters, credential access, data exfiltration or destructive commands. An AI review of the same text found nothing harmful. It declares no license and scores 88/100 on trust signals. Skills are instructions an agent will follow, so read the file before installing it and do not approve commands you do not understand.
Is Building Paper Screening Rubrics still maintained?
The repository was last updated about 2 months ago, so Building Paper Screening Rubrics is actively maintained.

name: Building Paper Screening Rubrics description: Collaboratively build and refine paper screening rubrics through brainstorming, test-driven development, and iterative feedback when_to_use: Starting new literature search. When automated screening misclassifies papers. When need to screen 50+ papers efficiently. Before creating screening scripts. When rescreening papers with updated criteria. version: 1.0.0

<!-- ╔══════════════════════════════════════════════════════════════╗ ║ 本文件为开源 Skill 原始文档,收录仅供学习与研究参考 ║ ║ CoPaper.AI 收集整理 | https://copaper.ai ║ ╚══════════════════════════════════════════════════════════════╝ 来源仓库: https://github.com/kthorn/research-superpower 项目名称: research-superpower 开源协议: MIT License 收录日期: 2026-04-02 声明: 本文件版权归原作者所有。此处收录旨在为社会科学实证研究者 提供 AI Agent Skills 的集中参考。如有侵权,请联系删除。 -->

Building Paper Screening Rubrics

Overview

Core principle: Build screening rubrics collaboratively through brainstorming → test → refine → automate → review → iterate.

Good rubrics come from understanding edge cases upfront and testing on real papers before bulk screening.

When to Use

Use this skill when:

  • Starting a new literature search that will screen 50+ papers
  • Current rubric misclassifies papers (false positives/negatives)
  • Need to define "relevance" criteria before automated screening
  • Want to update criteria and re-screen cached papers
  • Building helper scripts for evaluating-paper-relevance

When NOT to use:

  • Small searches (<20 papers) - manual screening is fine
  • Rubric already works well - no need to rebuild
  • One-off exploratory searches

Two-Phase Process

Phase 1: Collaborative Rubric Design

Step 1: Brainstorm Relevance Criteria

Ask domain-agnostic questions to understand what makes papers relevant:

Core Concepts:

  • "What are the key terms/concepts for your research question?"
    • Examples: specific genes, proteins, compounds, diseases, methods, organisms, theories
  • "Are there synonyms or alternative names?"
  • "Any terms that should EXCLUDE papers (false positives)?"

Data Types & Artifacts:

  • "What type of information makes a paper valuable?"
    • Quantitative measurements (IC50, expression levels, population sizes, etc.)
    • Protocols or methods
    • Datasets with accessions (GEO, SRA, PDB, etc.)
    • Code or software
    • Chemical structures
    • Sequences or genomes
    • Theoretical models
  • "Do you need the actual data in the paper, or just that such data exists?"

Paper Types:

  • "What types of papers are relevant?"
    • Primary research only?
    • Reviews or meta-analyses?
    • Methods papers?
    • Clinical trials?
    • Preprints acceptable?

Relationships & Context:

  • "Are papers about related/analogous concepts relevant?"
    • Example: "If studying protein X, are papers about homologs/paralogs relevant?"
    • Example: "If studying compound A, are papers about analogs/derivatives relevant?"
    • Example: "If studying disease X, are papers about related diseases relevant?"
  • "Does the paper need to be ABOUT your topic, or just MENTION it?"
  • "Are synthesis/methods papers relevant even without activity data?"

Edge Cases:

  • "Can you think of papers that would LOOK relevant but aren't?"
  • "Papers that might NOT look relevant but actually are?"

Document responses in screening-criteria.json

Step 2: Build Initial Rubric

Based on brainstorming, propose scoring logic:

Scoring (0-10):

Keywords Match (0-3 pts):
  - Core term 1: +1 pt
  - Core term 2 OR synonym: +1 pt
  - Related term: +1 pt

Data Type Match (0-4 pts):
  - Measurement type (IC50, Ki, EC50, etc.): +2 pts
  - Dataset/code available: +1 pt
  - Methods described: +1 pt

Specificity (0-3 pts):
  - Primary research: +3 pts
  - Methods paper: +2 pts
  - Review: +1 pt

Special Rules:
  - If mentions exclusion term: score = 0

Threshold: ≥7 = relevant, 5-6 = possibly relevant, <5 = not relevant

Present to user and ask: "Does this logic match your expectations?"

Save initial rubric to screening-criteria.json:

{
  "version": "1.0.0",
  "created": "2025-10-11T15:30:00Z",
  "keywords": {
    "core_terms": ["term1", "term2"],
    "synonyms": {"term1": ["alt1", "alt2"]},
    "related_terms": ["related1", "related2"],
    "exclusion_terms": ["exclude1", "exclude2"]
  },
  "data_types": {
    "measurements": ["IC50", "Ki", "MIC"],
    "datasets": ["GEO:", "SRA:", "PDB:"],
    "methods": ["protocol", "synthesis", "assay"]
  },
  "scoring": {
    "keywords_max": 3,
    "data_type_max": 4,
    "specificity_max": 3,
    "relevance_threshold": 7
  },
  "special_rules": [
    {
      "name": "scaffold_analogs",
      "condition": "mentions target scaffold AND (analog OR derivative)",
      "action": "add 3 points"
    }
  ]
}

Phase 2: Test-Driven Refinement

Step 1: Create Test Set

Do a quick PubMed search to get candidate papers:

# Search for 20 papers using initial keywords
curl "https://eutils.ncbi.nlm.nih.gov/entrez/eutils/esearch.fcgi?db=pubmed&term=YOUR_QUERY&retmax=20&retmode=json"

Fetch abstracts for first 10-15 papers:

curl "https://eutils.ncbi.nlm.nih.gov/entrez/eutils/efetch.fcgi?db=pubmed&id=PMID1,PMID2,...&retmode=xml&rettype=abstract"

Present abstracts to user one at a time:

Paper 1/10:
Title: [Title]
PMID: [12345678]
DOI: [10.1234/example]

Abstract:
[Full abstract text]

Is this paper RELEVANT to your research question? (y/n/maybe)

Record user judgments in test-set.json:

{
  "test_papers": [
    {
      "pmid": "12345678",
      "doi": "10.1234/example",
      "title": "Paper title",
      "abstract": "Full abstract text...",
      "user_judgment": "relevant",
      "timestamp": "2025-10-11T15:45:00Z"
    }
  ]
}

Continue until have 5-10 papers with clear judgments

Step 2: Score Test Papers with Rubric

Apply rubric to each test paper:

for paper in test_papers:
    score = calculate_score(paper['abstract'], rubric)
    predicted_status = "relevant" if score >= 7 else "not_relevant"
    paper['predicted_score'] = score
    paper['predicted_status'] = predicted_status

Calculate accuracy:

correct = sum(1 for p in test_papers
              if p['predicted_status'] == p['user_judgment'])
accuracy = correct / len(test_papers)

Step 3: Show Results to User

Present classification report:

RUBRIC TEST RESULTS (5 papers):

✓ PMID 12345678: Score 9 → relevant (user: relevant) ✓
✗ PMID 23456789: Score 4 → not_relevant (user: relevant) ← FALSE NEGATIVE
✓ PMID 34567890: Score 8 → relevant (user: relevant) ✓
✓ PMID 45678901: Score 3 → not_relevant (user: not_relevant) ✓
✗ PMID 56789012: Score 7 → relevant (user: not_relevant) ← FALSE POSITIVE

Accuracy: 60% (3/5 correct)
Target: ≥80%

--- FALSE NEGATIVE: PMID 23456789 ---
Title: "Novel analogs of compound X with improved potency"
Score breakdown:
  - Keywords: 1 pt (matched "compound X")
  - Data type: 2 pts (mentioned IC50 values)
  - Specificity: 1 pt (primary research)
  - Total: 4 pts → not_relevant

Why missed: Paper discusses "analogs" but didn't trigger scaffold_analogs rule
Abstract excerpt: "We synthesized 12 analogs of compound X..."

--- FALSE POSITIVE: PMID 56789012 ---
Title: "Review of kinase inhibitors"
Score breakdown:
  - Keywords: 2 pts
  - Data type: 3 pts
  - Specificity: 2 pts (review, not primary)
  - Total: 7 pts → relevant

Why wrong: Review paper, user wants primary research only

Step 4: Iterative Refinement

Ask user for adjustments:

Current accuracy: 60% (below 80% threshold)

Suggestions to improve rubric:
1. Strengthen scaffold_analogs rule - should "synthesized N analogs" always trigger?
2. Lower points for review papers (currently 2 pts, maybe 0 pts?)
3. Add more synonym terms for core concepts?

What would you like to adjust?

Update screening-criteria.json based on feedback

Example update:

{
  "special_rules": [
    {
      "name": "scaffold_analogs",
      "condition": "mentions target scaffold AND (analog OR derivative OR synthesized)",
      "action": "add 3 points"
    }
  ],
  "paper_types": {
    "primary_research": 3,
    "methods": 2,
    "review": 0  // Changed from 1
  }
}

Step 5: Re-test Until Satisfied

Re-score test papers with updated rubric

Show new results:

UPDATED RUBRIC TEST RESULTS (5 papers):

✓ PMID 12345678: Score 9 → relevant (user: relevant) ✓
✓ PMID 23456789: Score 7 → relevant (user: relevant) ✓ (FIXED!)
✓ PMID 34567890: Score 8 → relevant (user: relevant) ✓
✓ PMID 45678901: Score 3 → not_relevant (user: not_relevant) ✓
✓ PMID 56789012: Score 5 → not_relevant (user: not_relevant) ✓ (FIXED!)

Accuracy: 100% (5/5 correct) ✓
Target: ≥80% ✓

Rubric is ready for bulk screening!

If accuracy ≥80%: Proceed to bulk screening If <80%: Continue iterating

Phase 3: Bulk Screening

Once rubric validated on test set:

  1. Run on full PubMed search results
  2. Save all abstracts to abstracts-cache.json:
{
  "10.1234/example": {
    "pmid": "12345678",
    "title": "Paper title",
    "abstract": "Full abstract text...",
    "fetched": "2025-10-11T16:00:00Z"
  }
}
  1. Score all papers, save to papers-reviewed.json:
{
  "10.1234/example": {
    "pmid": "12345678",
    "status": "relevant",
    "score": 9,
    "source": "pubmed_search",
    "timestamp": "2025-10-11T16:00:00Z",
    "rubric_version": "1.0.0"
  }
}
  1. Generate summary report:
Screened 127 papers using validated rubric:
- Highly relevant (≥8): 12 papers
- Relevant (7): 18 papers
- Possibly relevant (5-6): 23 papers
- Not relevant (<5): 74 papers

All abstracts cached for re-screening.
Results saved to papers-reviewed.json.

Review offline and provide feedback if any misclassifications found.

Phase 4: Offline Review & Re-screening

User reviews papers offline, identifies issues:

User: "I reviewed the results. Three papers were misclassified:
- PMID 23456789 scored 4 but is actually relevant (discusses scaffold analogs)
- PMID 34567890 scored 8 but not relevant (wrong target)
- PMID 45678901 scored 6 but is highly relevant (has key dataset)

Can we update the rubric?"

Update rubric based on feedback:

  1. Analyze why misclassifications occurred
  2. Propose rubric adjustments
  3. Re-score ALL cached papers with new rubric
  4. Show diff of what changed

Re-screening workflow:

# Load all abstracts from abstracts-cache.json
# Apply updated rubric to each
# Generate change report

RUBRIC UPDATE: v1.0.0 → v1.1.0

Changes:
- Added "derivative" to scaffold_analogs rule
- Increased dataset bonus from +1 to +2 pts

Re-screening 127 cached papers...

Status changes:
  not_relevant → relevant: 3 papers
    - PMID 23456789 (score 4→7)
    - PMID 45678901 (score 6→8)
  relevant → not_relevant: 1 paper
    - PMID 34567890 (score 8→6)

Updated papers-reviewed.json with new scores.
New summary:
- Highly relevant: 13 papers (+1)
- Relevant: 19 papers (+1)

File Structure

research-sessions/YYYY-MM-DD-topic/
├── screening-criteria.json      # Rubric definition (weights, rules, version)
├── test-set.json               # Ground truth papers used for validation
├── abstracts-cache.json        # Full abstracts for all screened papers
├── papers-reviewed.json        # Simple tracking: DOI, score, status
└── rubric-changelog.md         # History of rubric changes and why

Integration with Other Skills

Before evaluating-paper-relevance:

  • Use this skill to build and validate rubric first
  • Creates screening-criteria.json and abstracts-cache.json
  • Then use evaluating-paper-relevance with validated rubric

When creating helper scripts:

  • Use screening-criteria.json to parameterize scoring logic
  • Reference abstracts-cache.json to avoid re-fetching
  • Easy to update rubric without rewriting script

**During answering-research-q

Truncated for display — read the full file on GitHub.

Related Skills

View on GitHub
GitHub Stars4.4k
CategoryAutomation
Updated1mo ago
Forks527

Languages

Stata

Trust signals

88/100

From repository metadata: license, adoption, age and documentation. Not a code audit — see the Safety scan above for what the skill file itself contains.

1 medium