agentic-experiments

<p align="center"> <img src="docs/logo-square.png" width="180" /> </p> <h1 align="center">agentic-experiments</h1> <p align="center"> When your agent runs ML experiments, make it run them <b>like a scientist</b>. </p> <p align="center"> <a href="https://github.com/KadenMc/agentic-experiments/actions"><img src="https://img.shields.io/github/actions/workflow/status/KadenMc/agentic-experiments/ci.yml?label=tests" alt="Tests"></a> <img src="https://img.shields.io/badge/status-beta-yellow" alt="Beta"> <img src="https://img.shields.io/badge/python-3.11%2B-blue" alt="Python 3.11+"> <a href="https://pypi.org/project/agentic-experiments/"><img src="https://img.shields.io/pypi/v/agentic-experiments.svg" alt="PyPI"></a> <a href="LICENSE"><img src="https://img.shields.io/badge/license-MIT-green" alt="MIT License"></a> <a href="https://github.com/sponsors/KadenMc"><img src="https://img.shields.io/badge/sponsor-%E2%9D%A4-ea4aaa" alt="Sponsor"></a> </p> <p align="center"> <a href="#quick-start">Quick Start</a> &bull; <a href="#how-it-works">How It Works</a> &bull; <a href="#why-this-is-different">Why It's Different</a> &bull; <a href="#features">Features</a> &bull; <a href="#architecture">Architecture</a> &bull; <a href="docs/concepts.md">Docs</a> </p>

---

agentic-experiments (import name aexp) is an opinionated research harness for ML experimentation done with an AI agent — typically Claude Code. It forces a Hypothesis → Experiment → Finding chain on every run, ties that chain to git commits, and validates citation integrity at every turn.

What this looks like in practice

• Your agent optionally captures forward-looking research directions as threads (T###) — concerns broader than a single hypothesis, with their own lifecycle (PROPOSED → EXPLORING → PROMOTED | CLOSED) — and promotes them into hypotheses when concrete claims emerge (aexp new-hypothesis --thread T###)
• It proposes a hypothesis and writes it to kb/research/hypotheses/H001-*.md — session-start hooks refuse work that skips this step; the H/E/F template enforces a ## Caveats + dual-mode ## Test Plan (pre-registered vs. exploratory) so authors can't fabricate retroactive thresholds
• It designs an experiment that explicitly cites the hypothesis; a pre-write hook blocks orphaned experiments. Experiment frontmatter can declare named conditions: blocks — --sp condition=full resolves to the full config at queue-time and is frozen into signac, so later edits to conditions.full can't retroactively change what ran (drift-proof provenance)
• It creates signac-backed runs via the MCP tool new_run (or in-process via aexp.tracked_run for managed wandb integration) — each run records its git commit, experiment ID, hypothesis ID, and resolved sp on the job document
• For batches, it uses the queue layer: aexp queue add --sweep "condition=full|cls,seed=0..3" registers the Cartesian product as pending jobs; aexp queue materialize --runner slurm emits a runner script that executes wherever the user's compute lives; aexp queue run iterates the pending jobs in-batch-script. Designed for agent-on-laptop, training-on-cluster workflows
• A W&B run (optional) is bound via three first-class modes: tracked_run (managed), prepare_tracker + ctx.bind(run) (BYO wandb.init), or bind_tracker (CLI/adapter). Group slug is deterministic from (hypothesis, experiment, condition)
• When it writes a finding, the supporting_runs array must cite real jobs — aexp validate flags dangling references; templates enforce a ## Caveats section distinct from ## Remaining Debt
• Delete an experiment by accident? Every run pointing at it is flagged run.broken_experiment_link; missing required template headers raise missing_template_header on the next validation pass

Principles

• Hypothesis-first, not metric-first — you can't start a run without a live hypothesis; you can't ship a finding without cited runs
• Git is the source of truth — every run carries its commit SHA; the knowledge base lives in git; nothing load-bearing is ephemeral
• Integrate, don't reinvent — signac for run state, W&B for observability, and a bundled research harness for the H→E→F artifact model, templates, and methodology skills. aexp is the glue and the discipline
• Portable by default — the MCP server runs via uvx from PyPI; .mcp.json is identical on every machine and committable to git

---

The Problem

Agents are great at running experiments. Left unattended, they are also great at running a lot of experiments with no shared thread — ablation sprawl, metric-chasing, findings with no clear question behind them, and a W&B workspace full of orphan runs nobody can reconstruct a month later.

The missing layer is not another tracker. It's a grammar — a structure the agent has to operate within, enforced deterministically by hooks rather than by reminder text in the prompt. Hypothesis before experiment. Experiment before run. Finding cites runs. Runs tied to commits.

aexp provides that grammar. Your agent proposes, designs, runs, and concludes; the harness makes sure the chain stays intact and the paper trail is reproducible.

---

How It Works

aexp stacks three concerns — research grammar, run state, and observability — glued together with a typed Python API and three agent-facing surfaces.

| Layer | What lives here | |---|---| | Research grammar | kb/ artifact graph — Hypothesis → Experiment → Finding plus Literature / Challenge Review / Strategic Review. Claude Code hooks enforce the H→E→F chain at write time. Research-methodology skills (experiment-rigor, exploratory-sota-research, research-devil-advocate, build-maintainable-software) install into .claude/skills/ | | Local run state (signac) | .runs/.signac/ plus one .runs/workspace/<job_id>/ directory per run. job.sp carries identity params; job.doc carries the artifact link, tracker IDs, status, and summary metrics | | Observability (W&B, optional [wandb] extra) | Remote runs grouped by a deterministic slug derived from (hypothesis_id, experiment_id, condition). Offline-by-default on HPC — aexp sync-offline walks the run store and syncs every pending run in one call from a login node |

Three surfaces, one canonical API

Every operation exists in three places, all thin wrappers over the same Python functions in aexp.*:

| Surface | Triggered by | Best for | |---|---|---| | MCP tools (new_run, list_runs, validate, …) | The agent during a turn | Structured queries, programmatic chaining, typed JSON returns | | Slash commands (/aexp-new-hypothesis, /aexp-new-run, /aexp-finding-from-batch, …) | User typing /aexp-… | Guided multi-step workflows | | CLI (aexp new-run ...) | Human at a terminal | Scripts, CI, PowerShell sessions |

The hooks are a fourth surface — invisible to the user, they inject kb/ACTIVE.md at session start, block HEF-chain violations, validate KB writes, and run structural validation at turn end.

---

Why This Is Different

Most ML experiment infrastructure records what happened. aexp polices what's allowed to happen.

• Unlike generic trackers (W&B, MLflow, Aim) — they log the numbers beautifully, but they don't care whether those numbers answer a question. aexp refuses runs that don't name their hypothesis and experiment.
• Unlike notebook-driven research — no commit ties, no structural validation, no citation integrity when you share the notebook three months later.
• Unlike DIY harnesses — this ships with working MCP integration, hook-enforced chain discipline, and a validation pass that catches broken references before they rot.

The design bet: agents already know how to run experiments. What they need is a runtime that makes rigorous research the path of least resistance.

---

Features

Research grammar

| | | |---|---| | H→E→F artifact graph | Every run descends from an Experiment, which descends from a Hypothesis. Findings cite runs with strong references (either specific job IDs or batch selectors). | | Hook-enforced discipline | SessionStart, PreToolUse, PostToolUse, and Stop hooks inject active context, block chain violations, and validate KB integrity at turn end. Hooks ship inside the installed package and upgrade via pip install -U. | | Research methodology skills | SKILL.md files install into .claude/skills/ — experiment rigor, exploratory SOTA research, devil's advocate review, and build-maintainable-software. Trigger with $experiment-rigor etc. |

Run state + observability

| | | |---|---| | signac-backed runs | Identity-hashed workspaces; idempotent creation keyed on state point; status and summary metrics in job.doc. Re-run at a new commit produces a distinct persistent workspace, both preserved. | | W&B binding | aexp.tracked_run(job, project=...) for managed wandb runs; aexp.prepare_tracker(job, ...) if your code already calls wandb.init and you just want the disciplined payload + signac binding. Group slug is deterministic; offline-first; co-locates with its signac workspace. | | HPC-friendly sync | aexp sync-offline walks the run store and runs wandb sync on every offline run — one command from a login node, no shell gymnastics. | | Tracker ABC | TrackerAdapter is a small ABC used by the legacy bind_tracker(job, adapter, ...) path — kept for NoopAdapter and backend-agnostic code. See docs/tracker-adapters.md for which surface to pick. | | Queue + runner materialization | aexp queue add / materialize registers pending runs on one machine and emits a runner script (shell / slurm / manual) for another. Drift-proof provenance via named conditions: blocks in the experiment frontmatter — --sp condition=full resolves to the full config at queue-time so later edits to conditions.full can't retroactively change what ran. See docs/queue.md. |

Agent surfaces

| | | |---|---| | MCP server | FastMCP covering artifact creation (H/E/F/T), run lifecycle, batch queries, queue management (incl. queue_stop for live-job interruption), tracker binding, and validation. Runs via uvx --from agentic-experiments[mcp] aexp-mcp-server — no absolute paths, no per-machine config, .mcp.json committable to git. | | Slash commands | Artifact creation: /aexp-new-hypothesis, /aexp-new-experiment, /aexp-new-run. Threads (forward-looking research concerns broader than a hypothesis): /aexp-new-thread, /aexp-list-threads, /aexp-show-thread, /aexp-close-thread. Finding creation (pick by what the finding cites): /aexp-finding-from-run, /aexp-finding-from-batch, /aexp-finding-placeholder. Read / inspect: /aexp-show-run, /aexp-show-batch, /aexp-list-runs, /aexp-status, /aexp-validate. Queue: /aexp-queue-add, /aexp-queue-list, /aexp-queue-materialize, /aexp-queue-stop. Notebook lifecycle (when --with-jupyter is configured): /aexp-jupyter-iterate (test loop), /aexp-promote-nb (promote working cells into a tracked-run script). Sandbox scaffolding: /aexp-new-sandbox (create an exploratory notebook subdir under notebooks/_sandbox/). | | CLI | Verbs covering install, artifact creation (H/E/F/T + thread lifecycle), run lifecycle, batch queries, tracker binding, validation, offline sync, optional jupyter-setup, the queue subcommand group (add/list/remove/stop/clear/materialize/run) + run-queued, and sandbox scaffolding (new-sandbox). See aexp --help for the full list. Python API is a one-line from aexp import .... | | Typed JSON contracts | Pydantic models (RunLink, BatchSelector, Issue, …) back the schema; MCP