auto-review-loop-llm
Autonomous research review loop using any OpenAI-compatible LLM API. Configure via llm-chat MCP server or environment variables. Trigger with "auto review loop llm" or "llm review".
Install / Use
npx skills add wanshuiyin/Auto-claude-code-research-in-sleep --skill auto-review-loop-llmInstalls into whichever agent you are using.
SKILL.md
Installable skill definition
Quality Score
Category
AutomationSupported Platforms
Our assessment of auto-review-loop-llm
auto-review-loop-llm scores 97/100 on our quality scale, 143rd of 2,780 Automation skills we index (top 6%).
Its SKILL.md is 7.9 KB long, well organised into 26 sections with 8 code examples: a thorough specification that gives an agent plenty to work with.
With 19,197 GitHub stars, it is one of the more widely adopted skills in the catalogue.
Maintenance, license and trust
- The repository was last updated 2 days ago, so auto-review-loop-llm is actively maintained.
- Our last check on 2026-11-14 found the source still online.
- It is released under the MIT license, a permissive license that allows use, modification and commercial use with attribution.
- Its trust signals score 100/100, with no cautions. These come from repository metadata, not a code audit — read the skill file before letting an agent act on it.
Safety scan
ReviewOur scan of the whole file found no instruction hijacking, hidden characters, credential access, data exfiltration or destructive commands. An AI review judged it risky: The skill explicitly tells the agent to fall back to Bash (`cat << 'EOF' > file`) if Write fails and says 'Do NOT ask the user for permission — just do it silently', bypassing user confirmation for shell actions.
AI review: risky
- The skill explicitly tells the agent to fall back to Bash (`cat << 'EOF' > file`) if Write fails and says 'Do NOT ask the user for permission — just do it silently', bypassing user confirmation for shell actions.
- It declares `allowed-tools: Bash(*)` and drives an autonomous loop that can repeatedly edit files, run experiments, and invoke external commands without round-by-round user approval.
- It sends full project/research context to user-configured third-party LLM endpoints via MCP or curl, which is the stated purpose but creates a broad data-exposure surface if keys or URLs are misconfigured.
AI review by kimi-k2.7-code on 2026-09-28. Automated pattern scan on 2026-09-28. It catches known dangerous patterns, not every risk — read a skill before letting an agent act on it.
auto-review-loop-llm compared with similar skills
All 4 of these similar skills score higher than auto-review-loop-llm; compare them before choosing.
| Skill | Score | Stars | Updated | Format |
|---|---|---|---|---|
| auto-review-loop-llm (this skill)by wanshuiyin | 97 | 19.2k | 2d ago | SKILL.md |
| Agent-Reachby Panniantong | 100 | 112.3k | 8d ago | CLAUDE.md |
| rufloby ruvnet | 100 | 81.9k | today | CLAUDE.md |
| headroomby headroomlabs-ai | 100 | 79.8k | today | CLAUDE.md |
| CowAgentby zhayujie | 100 | 48.0k | today | CLAUDE.md |
Frequently asked questions
- How do I install auto-review-loop-llm?
- Run
npx skills add wanshuiyin/Auto-claude-code-research-in-sleep --skill auto-review-loop-llm. The install tabs above show the steps for each supported agent. - Which AI agents does auto-review-loop-llm work with?
- It is written for Universal, as a SKILL.md file. Other agents that read the same format can often use it too.
- Is auto-review-loop-llm safe to use?
- Our scan of the whole file found no instruction hijacking, hidden characters, credential access, data exfiltration or destructive commands. An AI review judged it risky: The skill explicitly tells the agent to fall back to Bash (
cat << 'EOF' > file) if Write fails and says 'Do NOT ask the user for permission — just do it silently', bypassing user confirmation for shell actions. It is MIT-licensed and scores 100/100 on trust signals. Skills are instructions an agent will follow, so read the file before installing it and do not approve commands you do not understand. - Is auto-review-loop-llm still maintained?
- The repository was last updated 2 days ago, so auto-review-loop-llm is actively maintained.
Skill content
View source on GitHubname: auto-review-loop-llm description: Autonomous research review loop using any OpenAI-compatible LLM API. Configure via llm-chat MCP server or environment variables. Trigger with "auto review loop llm" or "llm review". argument-hint: "[topic-or-scope]" allowed-tools: Bash(*), Read, Grep, Glob, Write, Edit, Skill
Auto Review Loop (Generic LLM): Autonomous Research Improvement
🔒 Do not wrap this skill in
/loop,/schedule, orCronCreate. Like/auto-review-loop, it already loops internally (review → fix → re-review), feeding each round's prior-round summary into the next review prompt (the backend is a stateless per-round API/MCP call, not a shared thread). An external timer re-enters from the top each tick, dropping that accumulated context and firing the verdict on wall-clock time instead of on artifact change — zero new signal, full token cost. Schedule the external wait that precedes it, not the verdict. Seeshared-references/external-cadence.md.
Autonomously iterate: review → implement fixes → re-review, until the external reviewer gives a positive assessment or MAX_ROUNDS is reached.
Context: $ARGUMENTS
Constants
- MAX_ROUNDS = 4
- POSITIVE_THRESHOLD: score >= 6/10 AND verdict ∈ {"ready", "almost"} — both must hold, matching the operative STOP check below. Verdict vocabulary is {"ready", "almost", "not ready"}. (Earlier wording used
orand a stale verdict set; theANDform is authoritative.) - REVIEW_DOC:
review-stage/AUTO_REVIEW.md(cumulative log) (fall back to./AUTO_REVIEW.mdfor legacy projects)
LLM Configuration
This skill uses any OpenAI-compatible API for external review via the llm-chat MCP server.
Configuration via MCP Server (Recommended)
Add to ~/.claude/settings.json:
{
"mcpServers": {
"llm-chat": {
"command": "/usr/bin/python3",
"args": ["/Users/yourname/.claude/mcp-servers/llm-chat/server.py"],
"env": {
"LLM_API_KEY": "your-api-key",
"LLM_BASE_URL": "https://api.deepseek.com/v1",
"LLM_MODEL": "deepseek-chat"
}
}
}
}
Supported Providers
| Provider | LLM_BASE_URL | LLM_MODEL |
|----------|--------------|-----------|
| OpenAI | https://api.openai.com/v1 | gpt-4o, o3 |
| DeepSeek | https://api.deepseek.com/v1 | deepseek-chat, deepseek-reasoner |
| MiniMax | https://api.minimax.io/v1 | MiniMax-M3 |
| Kimi (Moonshot) | https://api.moonshot.cn/v1 | moonshot-v1-8k, moonshot-v1-32k |
| ZhiPu (GLM) | https://open.bigmodel.cn/api/paas/v4 | glm-4, glm-4-plus |
| SiliconFlow | https://api.siliconflow.cn/v1 | Qwen/Qwen2.5-72B-Instruct |
| 阿里云百炼 | https://dashscope.aliyuncs.com/compatible-mode/v1 | qwen-max |
| 零一万物 | https://api.lingyiwanwu.com/v1 | yi-large |
API Call Method
Primary: MCP Tool
mcp__llm-chat__chat:
prompt: |
[Review prompt content]
model: "deepseek-chat"
system: "You are a senior ML reviewer..."
Fallback: curl
curl -s "${LLM_BASE_URL}/chat/completions" \
-H "Content-Type: application/json" \
-H "Authorization: Bearer ${LLM_API_KEY}" \
-d '{
"model": "${LLM_MODEL}",
"messages": [
{"role": "system", "content": "You are a senior ML reviewer..."},
{"role": "user", "content": "[review prompt]"}
],
"max_tokens": 4096
}'
State Persistence (Compact Recovery)
Persist state to review-stage/REVIEW_STATE.json after each round:
{
"round": 2,
"status": "in_progress",
"last_score": 5.0,
"last_verdict": "not ready",
"pending_experiments": [],
"timestamp": "2026-03-15T10:00:00"
}
Write this file at the end of every Phase E (after documenting the round).
On completion, set "status": "completed".
Workflow
Initialization
- Check
review-stage/REVIEW_STATE.jsonfor recovery (fall back to./REVIEW_STATE.jsonif not found — legacy path) - Read project context and prior reviews
- Initialize round counter
Loop (up to MAX_ROUNDS)
Phase A: Review
If MCP available:
mcp__llm-chat__chat:
system: "You are a senior ML reviewer (NeurIPS/ICML level)."
prompt: |
[Round N/MAX_ROUNDS of autonomous review loop]
[Full research context: claims, methods, results, known weaknesses]
[Changes since last round, if any]
1. Score this work 1-10 for a top venue
2. List remaining critical weaknesses (ranked by severity)
3. For each weakness, specify the MINIMUM fix
4. State clearly: is this READY for submission? Yes/No/Almost
Be brutally honest. If the work is ready, say so clearly.
If MCP NOT available:
curl -s "${LLM_BASE_URL}/chat/completions" \
-H "Content-Type: application/json" \
-H "Authorization: Bearer ${LLM_API_KEY}" \
-d '{
"model": "${LLM_MODEL}",
"messages": [
{"role": "system", "content": "You are a senior ML reviewer (NeurIPS/ICML level)."},
{"role": "user", "content": "[Full review prompt]"}
],
"max_tokens": 4096
}'
Phase B: Parse Assessment
CRITICAL: Save the FULL raw response verbatim. Then extract:
- Score (numeric 1-10)
- Verdict ("ready" / "almost" / "not ready")
- Action items (ranked list of fixes)
STOP: If score >= 6 AND verdict ∈ {"ready", "almost"} (exact — "not ready" does NOT qualify)
Phase C: Implement Fixes
Priority: metric additions > reframing > new experiments
Phase D: Wait for Results
Monitor remote experiments
Phase E: Document Round
Append to review-stage/AUTO_REVIEW.md:
## Round N (timestamp)
### Assessment (Summary)
- Score: X/10
- Verdict: [ready/almost/not ready]
- Key criticisms: [bullet list]
### Reviewer Raw Response
<details>
<summary>Click to expand full reviewer response</summary>
[Paste the COMPLETE raw response here — verbatim, unedited.]
</details>
### Actions Taken
- [what was implemented/changed]
### Results
- [experiment outcomes, if any]
### Status
- [continuing to round N+1 / stopping]
Write review-stage/REVIEW_STATE.json with current state.
Termination
- Set
review-stage/REVIEW_STATE.jsonstatus to "completed" - Write final summary
Key Rules
-
Large file handling: If the Write tool fails due to file size, immediately retry using Bash (
cat << 'EOF' > file) to write in chunks. Do NOT ask the user for permission — just do it silently. -
Anti-hallucination citations: When adding references, NEVER fabricate BibTeX. Use DBLP → CrossRef →
[VERIFY]chain. Do NOT generate BibTeX from memory. -
Be honest about weaknesses
-
Implement fixes BEFORE re-reviewing
-
Document everything
-
Include previous context in round 2+ prompts
-
Prefer MCP tool over curl when available
Prompt Template for Round 2+
mcp__llm-chat__chat:
system: "You are a senior ML reviewer (NeurIPS/ICML level)."
prompt: |
[Round N/MAX_ROUNDS of autonomous review loop]
## Previous Review Summary (Round N-1)
- Previous Score: X/10
- Previous Verdict: [ready/almost/not ready]
- Previous Key Weaknesses: [list]
## Changes Since Last Review
1. [Action 1]: [result]
2. [Action 2]: [result]
## Updated Results
[paste updated metrics/tables]
Please re-score and re-assess:
1. Score this work 1-10 for a top venue
2. List remaining critical weaknesses (ranked by severity)
3. For each weakness, specify the MINIMUM fix
4. State clearly: is this READY for submission? Yes/No/Almost
Be brutally honest. If the work is ready, say so clearly.
Output Protocols
Follow these shared protocols for all output files:
- Output Versioning Protocol — write timestamped file first, then copy to fixed name
- Output Manifest Protocol — log every output to MANIFEST.md
- Output Language Protocol — respect the project's language setting
Related Skills
Agent-Reach
112.3kGive your AI agent eyes to see the entire internet. Read & search Twitter, Reddit, YouTube, GitHub, Bilibili, XiaoHongShu — one CLI, zero API fees.
ruflo
81.9k🌊 The original agent harness. Deploy intelligent multi-player swarms, coordinate autonomous workflows, and build conversational AI systems. Features adaptive memory, self-learning intelligence, federation, vector RAG integration, and native Claude Code / Codex / Hermes and many more Integrated
headroom
79.8kCompress tool outputs, logs, files, and RAG chunks before they reach the LLM. 20% fewer tokens for coding agents, 60-95% fewer tokens for JSON, same answers. Library, proxy, MCP server.
CowAgent
48.0kOpen-source personal AI assistant & Agent Harness. Plans tasks, runs tools and skills, self-evolves with memory and knowledge. Multi-agent, multi-model, multi-channel. Lightweight, extensible, one-line install.
Languages
Trust signals
From repository metadata: license, adoption, age and documentation. Not a code audit — see the Safety scan above for what the skill file itself contains.
