SkillAgentSearch skills...

paper-writing-bench

Reverse-engineer raw materials (Sparse idea, Dense idea, experimental log) from an existing AI research paper to build a benchmark case for evaluating paper-writing pipelines. Replicates the PaperWritingBench dataset construction procedure from arXiv:2604.05018 §3 / App. C.

Install / Use

npx skills add Ar9av/PaperOrchestra --skill paper-writing-bench

Installs into whichever agent you are using.

About this skill
📄

SKILL.md

Installable skill definition

Quality Score

81/100

Category

Automation

Supported Platforms

Universal

Tags

Our assessment of paper-writing-bench

paper-writing-bench scores 81/100 on our quality scale, 2404th of 2,768 Automation skills we index.

Its SKILL.md is 5.5 KB long, well organised into 11 sections and no code examples: a solid amount of guidance for an agent.

It has 664 GitHub stars, a meaningful sign that others use it.

Substance
26/30
Structure
13/20
Description
15/15
Adoption
12/20
Freshness
15/15

Maintenance, license and trust

  • The repository was last updated about 2 months ago, so paper-writing-bench is actively maintained.
  • Our last check on 2026-11-08 found the source still online.
  • No license is declared. By default that means all rights are reserved: you can read it, but reusing or redistributing it is not clearly permitted. Ask the author before building on it commercially.
  • Its trust signals score 88/100, with 1 caution from licensing, adoption, age or documentation. These come from repository metadata, not a code audit — read the skill file before letting an agent act on it.

Safety scan

No issues found

Our scan of the whole file found no instruction hijacking, hidden characters, credential access, data exfiltration or destructive commands. An AI review of the same text found nothing harmful.

AI review by kimi-k2.7-code on 2026-10-05. Automated pattern scan on 2026-10-04. It catches known dangerous patterns, not every risk — read a skill before letting an agent act on it.

paper-writing-bench compared with similar skills

All 4 of these similar skills score higher than paper-writing-bench; compare them before choosing.

SkillScoreStarsUpdatedFormat
paper-writing-bench (this skill)by Ar9av8166458d agoSKILL.md
Agent-Reachby Panniantong100113.0k10d agoCLAUDE.md
rufloby ruvnet10082.0ktodayCLAUDE.md
Claude-Code-Game-Studiosby Donchitos10040.1k9d agoCLAUDE.md
Scraplingby D4Vinci100104.2k1d agoMCP Server

Frequently asked questions

How do I install paper-writing-bench?
Run npx skills add Ar9av/PaperOrchestra --skill paper-writing-bench. The install tabs above show the steps for each supported agent.
Which AI agents does paper-writing-bench work with?
It is written for Universal, as a SKILL.md file. Other agents that read the same format can often use it too.
Is paper-writing-bench safe to use?
Our scan of the whole file found no instruction hijacking, hidden characters, credential access, data exfiltration or destructive commands. An AI review of the same text found nothing harmful. It declares no license and scores 88/100 on trust signals. Skills are instructions an agent will follow, so read the file before installing it and do not approve commands you do not understand.
Is paper-writing-bench still maintained?
The repository was last updated about 2 months ago, so paper-writing-bench is actively maintained.

name: paper-writing-bench description: Reverse-engineer raw materials (Sparse idea, Dense idea, experimental log) from an existing AI research paper to build a benchmark case for evaluating paper-writing pipelines. Replicates the PaperWritingBench dataset construction procedure from arXiv:2604.05018 §3 / App. C. TRIGGER when the user asks to "build a benchmark case from this paper", "reverse-engineer raw materials", or "evaluate my pipeline against PaperWritingBench".

PaperWritingBench (§3)

Faithful implementation of the PaperWritingBench dataset construction procedure from PaperOrchestra (Song et al., 2026, arXiv:2604.05018, §3 and App. C, F.2).

The original benchmark contains 200 papers (100 CVPR 2025 + 100 ICLR 2025). For each paper, the authors reverse-engineer the (I, E) tuple by stripping narrative flow from the original PDF using the three prompts in App. F.2. You can use this skill to reverse-engineer your own benchmark cases from any paper PDF.

What this skill does

Given an existing AI research paper (PDF or markdown extract), produce:

  • idea.md (Sparse variant) — high-level concept note, no math, no experimental results
  • idea.md (Dense variant) — detailed technical proposal with LaTeX equations and variable definitions, but still no experimental results
  • experimental_log.md — exhaustive raw experimental setup, numeric data, and qualitative observations, with all narrative references stripped

These three files form a complete (I, E) input pair for the paper-orchestra pipeline. You can then run the pipeline and compare its output to the original paper using paper-autoraters.

Inputs

  • A paper PDF or extracted markdown text. The paper uses MinerU (Wang et al., 2024) for PDF→markdown extraction; you (the host agent) should use whatever PDF extractor your environment provides.
  • For controlled experiments, you may also extract figures separately (PDFFigures 2.0 in the paper).

Outputs

  • bench/<paper_id>/idea_sparse.md — Sparse variant
  • bench/<paper_id>/idea_dense.md — Dense variant
  • bench/<paper_id>/experimental_log.md — Experimental log

Workflow

For each paper, run three independent LLM calls using the verbatim prompts below:

1. Sparse idea generation

Load references/sparse-idea-prompt.md. Pass the paper text (or markdown extract) as {paper_content}. The prompt instructs the model to:

  • Stop extracting at empirical verification (no Experiments / Results / Comparisons)
  • Use first-person future tense ("We propose to explore...")
  • Avoid LaTeX math; describe components by function
  • Anonymize authors and titles

Output: idea_sparse.md with the four sections (Problem Statement, Core Hypothesis, Proposed Methodology high-level, Expected Contribution).

2. Dense idea generation

Load references/dense-idea-prompt.md. Same input. The prompt instructs the model to:

  • Preserve mathematical formulations using LaTeX
  • Define every variable used in equations
  • Include specific architectural choices and dimensions
  • Same exclusion zone (no experiments)

Output: idea_dense.md with the four sections (Problem Statement, Core Hypothesis, Proposed Methodology detailed, Expected Contribution).

3. Experimental log generation

Load references/experimental-log-prompt.md. Same input. The prompt instructs the model to:

  • Use past-tense persona ("We ran...", "The results were...")
  • Strip all references to figure/table numbers
  • Deconstruct tables into raw numeric data
  • Log figure findings as factual observations
  • Anonymize authors

Output: experimental_log.md with sections for Setup, Raw Numeric Data, and Qualitative Observations.

Critical rules from the prompts

These are excerpted from App. F.2. The host agent MUST honor them:

  • No citations. None of the three outputs may contain \cite, reference numbers, or author names from the source paper.
  • No URLs. Strip all hyperlinks.
  • Anonymize. Author identities, affiliations, acknowledgements all removed.
  • Self-contained. Each file must make sense without the original paper.
  • No experimental leakage in idea files. The Sparse and Dense ideas must stop where empirical verification begins. They describe what will be done, not what was done.
  • No table/figure references in experimental log. No "as shown in Table 1", "see Fig. 5". The downstream paper-orchestra pipeline will generate its own figures and tables — the log must not assume any particular ones exist.
  • 100% numeric accuracy in experimental log. This becomes the ground truth for the section-writing-agent and content-refinement-agent's hallucination check.

How the bench is used

After producing (idea_sparse.md, idea_dense.md, experimental_log.md) for a paper:

  1. Pick a variant (Sparse or Dense) — the paper ablates both, with Dense producing more rigorous methodology and Sparse exercising the system's robustness on under-specified inputs.
  2. Drop the chosen idea.md, plus experimental_log.md, plus a template.tex for the target conference, plus a conference_guidelines.md, into a paper-orchestra workspace.
  3. Run the pipeline.
  4. Compare the generated paper against the original using paper-autoraters (citation F1, lit review quality, SxS paper quality).

Resources

  • references/bench-overview.md — the 200-paper bench, venue cutoffs, sizes
  • references/sparse-idea-prompt.md — verbatim from App. F.2
  • references/dense-idea-prompt.md — verbatim from App. F.2
  • references/experimental-log-prompt.md — verbatim from App. F.2

Related Skills

View on GitHub
GitHub Stars664
CategoryAutomation
Updated1mo ago
Forks92

Languages

Python

Trust signals

88/100

From repository metadata: license, adoption, age and documentation. Not a code audit — see the Safety scan above for what the skill file itself contains.

1 medium
paper-writing-bench — Universal Skill: Install & Safety Check | SkillAgent