catobot-autoexperiment-mcp
Domain-agnostic MCP server for autonomous experimentation with metric-driven keep/rollback decisions and reproducible experiment history.
Install / Use
claude mcp add IamCatoBot -- npx -y github:IamCatoBot/catobot-autoexperiment-mcpIf the server publishes to npm under a different name, use that package instead ā check the repo README.
MCP Server
Model Context Protocol server
Quality Score
Category
AutomationSupported Platforms
Tags
Our assessment of catobot-autoexperiment-mcp
catobot-autoexperiment-mcp scores 78/100 on our quality scale, 2508th of 2,769 Automation skills we index.
Its MCP Server is 16 KB long, well organised into 34 sections with 13 code examples: a thorough specification that gives an agent plenty to work with.
It has 3 GitHub stars, so there is little community track record yet; judge it on its content.
Maintenance, license and trust
- The repository was last updated about 8 months ago. That is recent enough to be usable, but agent tooling moves fast, so check the instructions against your agent's current version.
- Our last check on 2026-09-20 found the source still online.
- It is released under the Apache-2.0 license, a permissive license that allows use, modification and commercial use with attribution.
- Its trust signals score 86/100, with 2 cautions from licensing, adoption, age or documentation. These come from repository metadata, not a code audit ā read the skill file before letting an agent act on it.
Safety scan
No issues foundOur scan of the whole file found no instruction hijacking, hidden characters, credential access, data exfiltration or destructive commands (2 minor notes below). An AI review of the same text found nothing harmful.
- noteInstalls by piping a downloaded script into a shellline 101
curl -LsSf https://astral.sh/uv/install.sh | sh - noteInstalls by piping a downloaded script into a shellline 107
powershell -ExecutionPolicy ByPass -c "irm https://astral.sh/uv/install.ps1 | iex"
AI review by kimi-k2.7-code on 2026-10-02. Automated pattern scan on 2026-10-02. It catches known dangerous patterns, not every risk ā read a skill before letting an agent act on it.
catobot-autoexperiment-mcp compared with similar skills
All 4 of these similar skills score higher than catobot-autoexperiment-mcp; compare them before choosing.
| Skill | Score | Stars | Updated | Format |
|---|---|---|---|---|
| catobot-autoexperiment-mcp (this skill)by IamCatoBot | 78 | 3 | 8mo ago | MCP Server |
| Agent-Reachby Panniantong | 100 | 112.8k | 10d ago | CLAUDE.md |
| rufloby ruvnet | 100 | 82.0k | today | CLAUDE.md |
| headroomby headroomlabs-ai | 100 | 79.9k | today | CLAUDE.md |
| CowAgentby zhayujie | 100 | 48.0k | today | CLAUDE.md |
Frequently asked questions
- How do I install catobot-autoexperiment-mcp?
- Run
claude mcp add IamCatoBot -- npx -y github:IamCatoBot/catobot-autoexperiment-mcp. The install tabs above show the steps for each supported agent. - Which AI agents does catobot-autoexperiment-mcp work with?
- It is written for Claude Code and Claude Desktop, as a MCP Server file. Other agents that read the same format can often use it too.
- Is catobot-autoexperiment-mcp safe to use?
- Our scan of the whole file found no instruction hijacking, hidden characters, credential access, data exfiltration or destructive commands (2 minor notes below). An AI review of the same text found nothing harmful. It is Apache-2.0-licensed and scores 86/100 on trust signals. Skills are instructions an agent will follow, so read the file before installing it and do not approve commands you do not understand.
- Is catobot-autoexperiment-mcp still maintained?
- The repository was last updated about 8 months ago. That is recent enough to be usable, but agent tooling moves fast, so check the instructions against your agent's current version.
Skill content
View source on GitHubCatoBot autoexperiment MCP Server
Domain-Agnostic MCP Server for Autonomous Experimentation
A generalisation of Karpathy's autoresearch pattern into a reusable Model Context Protocol (MCP) server that any AI agent can drive, pointed at any domain.
Documentation Map
- Core usage and setup: this README
- Example catalog:
example_experiments/README.md - Experiment design guide:
example_experiments/Autoexperiment_Design_Guide.md - Contribution guide:
CONTRIBUTING.md - Security policy:
SECURITY.md - Support channels:
SUPPORT.md - License and notices:
LICENSE,NOTICE
Example Experiments
- Shell-only experiment with CatoBot autoexperiment MCP:
example_experiments/shell/ - External orchestration of CatoBot autoexperiment MCP + Text2Sim MCP):
example_experiments/external/DES_Text2Sim/
The Pattern
modify something ā run it ā measure a result ā keep or discard ā repeat
The server exposes this loop as a standard set of MCP tools. The domain (what gets modified, how it runs, and what gets measured) is defined entirely in a JSON config file. The agent-side logic stays the same regardless of domain.
Architecture
āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā
ā AI Agent (Claude Code, Codex, etc.) ā
ā ā
ā Reads status ā plans change ā edits file ā ā
ā runs experiment ā checks result ā keeps/discards ā
āāāāāāāāāāāāāāāā¬āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā
ā MCP (stdio)
āāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā
ā autoexperiment MCP server ā
ā ā
ā Tools: ā
ā autoexp_get_status ā session overview ā
ā autoexp_read_file ā read allowed file ā
ā autoexp_update_file ā full file replace ā
ā autoexp_patch_file ā targeted find/repl ā
ā autoexp_run_experiment ā execute + measure ā
ā autoexp_begin_experiment ā open pending record ā
ā autoexp_complete_experiment ā close with metric ā
ā autoexp_set_baseline ā mark as baseline ā
ā autoexp_rollback ā revert to last good ā
ā autoexp_get_history ā review past runs ā
ā autoexp_run_setup ā one-time setup ā
ā ā
ā Resources: ā
ā autoexp://status ā session status (JSON) ā
ā autoexp://history ā experiment history ā
ā autoexp://file/{path} ā read allowed files ā
ā ā
ā Config: autoexperiment.json (domain adapter) ā
ā Ledger: .autoexperiment_ledger.json (state) ā
āāāāāāāāāāāāāāāā¬āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā
ā subprocess / external MCP server
āāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā
ā Your domain ā
ā (training script, benchmark, simulation, etc.) ā
āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā
Code Structure
The server is implemented as a Python package (autoexperiment_mcp/) with a thin server.py entry point:
autoexperiment-mcp-server/
āāā server.py # Entry point: imports mcp, calls mcp.run()
āāā autoexperiment_mcp/
āāā models.py # Pydantic models (DomainConfig, ExperimentRecord, ā¦)
āāā utils.py # Pure utilities (git, hash, path, regex, time, coercion)
āāā store.py # State I/O, snapshot management, TSV logging, query helpers
āāā experiment.py # Core lifecycle: begin/complete experiment, keep decision
āāā lifespan.py # Startup validation, app_lifespan context manager
āāā app.py # mcp = FastMCP("autoexperiment_mcp", lifespan=ā¦)
āāā tools.py # All 11 @mcp.tool() registrations
āāā resources.py # All 3 @mcp.resource() registrations
āāā __init__.py # Imports app + triggers tool/resource registration
Installation
Prerequisites
- Python 3.12 or higher
uvpackage manager
Install uv
macOS / Linux
curl -LsSf https://astral.sh/uv/install.sh | sh
Windows (PowerShell)
powershell -ExecutionPolicy ByPass -c "irm https://astral.sh/uv/install.ps1 | iex"
Learn more: astral-sh/uv
Clone the repository
git clone https://github.com/IamCatoBot/catobot-autoexperiment-mcp.git
cd catobot-autoexperiment-mcp
Install dependencies
uv sync
Quick Start
1. Prepare your experiment folder
Your experiment folder needs a working baseline, an evaluation script, and a config file:
my-experiment/
āāā autoexperiment.json ā config (you write this)
āāā solution.py ā editable (agent modifies this)
āāā benchmark.py ā evaluation (read-only)
āāā data.csv ā test data (read-only)
2. Create autoexperiment.json
{
"project_name": "My Experiment",
"description": "What you're trying to optimise",
"workspace_dir": "/absolute/path/to/my-experiment",
"editable_files": ["solution.py"],
"read_only_files": ["benchmark.py", "data.csv"],
"run_command": "python benchmark.py 2>&1",
"timeout_seconds": 60,
"metric_name": "rmse",
"metric_regex": "^rmse:\\s*([\\d.]+)",
"metric_direction": "lower",
"use_git": true
}
3. Initialise git in the experiment folder
Git tracking is enabled by default (use_git: true). The experiment folder must be a git repository with an initial commit before the server will start.
cd /path/to/my-experiment
git init
git add -A
git commit -m "initial baseline"
4. Verify the run command works
Run your experiment command manually and check the output contains the metric in the expected format:
cd /path/to/my-experiment
python benchmark.py
# Should print something like: rmse: 12.345678
5. Register the MCP server
Recommended: pass AUTOEXPERIMENT_CONFIG pointing to your config file. MCP hosts may launch the server process from a different working directory, so an explicit path is the safest default.
Claude Code (-e for env vars):
claude mcp add autoexperiment \
-e AUTOEXPERIMENT_CONFIG=/path/to/my-experiment/autoexperiment.json \
-- uv run \
--project PATH_TO_AUTOEXPERIMENT_MCP_SERVER \
python PATH_TO_AUTOEXPERIMENT_MCP_SERVER/server.py
Codex (--env for env vars):
codex mcp add autoexperiment \
--env AUTOEXPERIMENT_CONFIG=/path/to/my-experiment/autoexperiment.json \
-- uv run \
--project PATH_TO_AUTOEXPERIMENT_MCP_SERVER \
python PATH_TO_AUTOEXPERIMENT_MCP_SERVER/server.py
Replace /path/to/my-experiment/autoexperiment.json with the absolute path to your config file.
Optional shortcut: if the server process is launched from your experiment folder and the config filename is autoexperiment.json, you can omit the environment variable.
You only need to register the MCP server once per MCP client profile; after that, reconnect normally in new sessions.
Note: Replace
PATH_TO_AUTOEXPERIMENT_MCP_SERVERwith the actual path to your cloned repository. If theuvcommand is not found, runwhich uv(Unix) orGet-Command uv(PowerShell) and use the full path in the"command"field.
6. Start experimenting
Launch Claude Code, Codex, or another MCP client from your experiment folder and prompt it:
Read the experiment status, review the editable and read-only files, run the baseline first, then iterate until improvements plateau and no meaningful gains remain.
Security Warning
setup_command and run_command execute shell commands on your host machine. This server does not provide sandboxing or container isolation by default.
Startup Validation
The server validates the configuration at startup and will refuse to start if:
workspace_dirdoes not exist or is not a directory- Any file in
editable_filesorread_only_filesis missing - A file appears in both
editable_filesandread_only_files metric_regexis not a valid regular expressionuse_gitistruebut the workspace is not a git repository
Error messages are specific and tell you exactly what to fix.
Configuration
Everything domain-specific lives in autoexperiment.json:
| Field | Required | Default | Description |
|---|---|---|---|
| project_name | yes | | Human-readable name |
| description | no | "" | What you're trying to achieve |
| workspace_dir | yes | | Absolute path to the experiment folder |
| editable_files | yes | | Files the agent is allowed to modify (at least one) |
| read_only_files | no | [] | Files the agent can read but not change |
| execution_mode | no | "hybrid" | "shell", "external", or "hybrid" |
| run_command | shell/hybrid | | Shell command to run one experiment |
| timeout_seconds | no | 300 | Max time per experiment (10ā7200s) |
| setup_command | no | null | One-time setup (deps, data download, etc.) |
| metric_name | yes | | Name of the metric being optimised |
| metric_regex | shell/hybrid | | Regex with one capture group to extract a float from stdout |
| metric_direction | yes | | "lower" or "higher" |
| require_baseline_first | no | true | Require a baseline experiment before non-baseline runs |
| use_git | no | true | Track experiments as git commits. Requires the workspace to be a git repo with an initial commit. |
| git_branch_prefix | no | "autoexp" | Prefix for experiment branches |
| keep_policy | no | see below | Multi-gate keep/discard policy |
Keep Policy
The keep_policy object controls when a completed experiment is kept vs discarded. All gates must pass for a run to be kept.
| Field | Default | Description |
|---|---|---|
| required_true_keys | [] | Metadata keys that must be boolean true |
| numeric_min | {} | Metadata keys with a floor value (e.g. {"utilization": 45}) |
| numeric_max | {} | Metadata keys with a ceiling value (e.g. {"latency_ms": 250}) |
| require_numeric_keys_present | true | If true, missing keys in numeric_min/numeric_max cause discard |
| allow_equal_metric_if_simpler | true | Keep a tied run if its complexity_score is lower |
| equal_metric_tolerance | 1e-9 | Tolerance for treating two metric values as equal |
| complexity_key | "complexity_score" | Metadata key used for complexity tie-breaking |
The agent sees the full keep_policy in autoexp_get_status and receives a required_metadata_keys reminder in every autoexp_begin_experiment response ā so it always knows exactly what to include in the metadata argument when calling autoexp_complete_experiment.
Tool Reference
| Tool | Purpose | Destructive? |
|---|---|---|
| autoexp_get_status | Session overview, best score, editable files, keep_policy gates | No |
| autoexp_read_file | Read any allowed file | No |
| autoexp_update_file | Replace entire file contents | Yes |
| autoexp_patch_file | Targeted find-and-replace | No |
| autoexp_run_experiment | Execute the run command, extract metric (shell mode) | No (but slow) |
| autoexp_begin_experiment | Open a pending experiment record (external/hybrid mode) | No |
| `aut
Truncated for display ā read the full file on GitHub.
Related Skills
Agent-Reach
112.8kGive your AI agent eyes to see the entire internet. Read & search Twitter, Reddit, YouTube, GitHub, Bilibili, XiaoHongShu ā one CLI, zero API fees.
ruflo
82.0kš The original agent harness. Deploy intelligent multi-player swarms, coordinate autonomous workflows, and build conversational AI systems. Features adaptive memory, self-learning intelligence, federation, vector RAG integration, and native Claude Code / Codex / Hermes and many more Integrated
headroom
79.9kCompress tool outputs, logs, files, and RAG chunks before they reach the LLM. 20% fewer tokens for coding agents, 60-95% fewer tokens for JSON, same answers. Library, proxy, MCP server.
CowAgent
48.0kOpen-source personal AI assistant & Agent Harness. Plans tasks, runs tools and skills, self-evolves with memory and knowledge. Multi-agent, multi-model, multi-channel. Lightweight, extensible, one-line install.
Languages
Trust signals
From repository metadata: license, adoption, age and documentation. Not a code audit ā see the Safety scan above for what the skill file itself contains.
