<img src="docs/images/titration-logo.svg" alt="" width="36" height="36"> Titration

Let your coding agent fix a prompt until it actually works, and prove it did.

Point your agent at a prompt that misbehaves. It changes the prompt, reruns it, and Titration grades every attempt against a baseline that doesn't move, with judges from other AI vendors. The loop keeps going until the problem is gone, or until Titration tells you the prompt was never the problem.

Real run: billing tickets wrongly marked urgent went from 90% → 0% in one measured change (worked example).

An open-source MCP server with 18 tools. Self-hosted. Configuration examples for Claude Code, Codex and Cursor. Five-slide overview (PDF)

Why you can trust the loop

An agent that iterates against a score will happily game the score. Titration is built so it can't:

• Other vendors grade the work. Your agent's own vendor is never on the judge panel, and a score needs judges from at least two vendor families to answer. Fewer, and the attempt is refused with the reason and never scored.
• The baseline doesn't move. A baseline freezes its rubric, its outputs and its judges, so "better" always means better against the same yardstick.
• Noise is not a win. A change inside the judges' disagreement band comes back inconclusive, never passed, and a gain that breaks another group of cases doesn't ship.
• It tells you when it isn't the prompt. Every failure is classified into one of nine origins. Only one of them means "edit the prompt"; the others point at your test set, your rubric, your judges or the code around the model.
• Memory that compounds. What each run learns becomes searchable cards your agent can use next time, alongside a starter pack of 66 evaluation methods.
• Yours. Your Postgres, your keys, your judges. Nothing leaves your machine except the calls to the judge and embedding providers you choose.

---

How it works

"The first principle is that you must not fool yourself, and you are the easiest person to fool." (Richard Feynman)

"When a measure becomes a target, it ceases to be a good measure." (Goodhart's law, as put by Marilyn Strathern)

Titration never pulls or runs your code. Your agent sends the outputs it wants judged; Titration grades only those declared outputs against the rubric. It runs as a local MCP server over stdio, calls the judges you pick, and stores everything in your own Postgres.

Quickstart (about 5 minutes)

You need Node.js 22.11+, Git on the server process PATH (for repository discovery), Docker, and access to judges from three vendor families other than your agent's own (a panel is three judges from three families, and your agent's family never judges its own work).

• An OpenRouter API key covers this on its own (12 models across 9 families) and also turns on memory search. This is the simplest start.
• The subscription CLIs can fill up to two seats at no per-call cost: claude (npm i -g @anthropic-ai/claude-code), codex (npm i -g @openai/codex), and grok. With them, a panel is typically two CLIs plus one OpenRouter model.

Fastest: let your agent install it

Start Docker, then paste this into Claude Code, Codex or Cursor:

Install Titration from https://github.com/kaithoughtarchitect/titration by following the
Quickstart in its README, including "Notes for an installing agent". Register it as an MCP
server for yourself, then tell me to restart you.

You approve its commands as it goes, give it your OpenRouter key if you have one, and restart your agent at the end. MCP servers load when a session starts, so the agent cannot use Titration in the session that installed it.

By hand

Clone it into its own folder, not inside a project you want to measure:

git clone https://github.com/kaithoughtarchitect/titration.git
cd titration
docker compose up -d          # Postgres + pgvector on localhost:5432
npm install
cp .env.example .env          # then set OPENROUTER_API_KEY if you have one
npm run setup                 # applies the schema, loads the base starter pack

npm run setup is safe to re-run. Without an OpenRouter key it still loads the starter pack, but card search (vector search) stays off until you add a key and run npm run embed.

Notes for an installing agent

• Check first: node --version is 22.11 or later, git --version works, and docker info succeeds. If Docker is not running, stop and ask the user to start it; do not try to install it.
• Clone into its own folder (for example ~/titration), never inside the user's current project.
• If port 5432 is already in use, change the host side of ports in docker-compose.yml (for example "55432:5432") and use the same port in TITRATION_DATABASE_URL in .env.
• Ask the user for an OpenRouter key and write it to .env. Without one, leave it blank; setup still succeeds.
• Setup has worked when it ends with [setup] done or the line saying the base is loaded but unembedded. Anything else is a failure: show the user the error.
• Register the server with the clone's absolute path, as shown under "Connect your agent" below, then tell the user to restart you.

Connect your agent

The server speaks MCP over stdio. Point your client at server/server.ts in your clone (replace the path). These are configuration examples, not client certifications: local or global configuration works when that connection supplies the correct local repository roots, or (for clients without roots support) launches the server in the project directory:

Claude Code

claude mcp add --scope user titration -- npx tsx /absolute/path/to/titration/server/server.ts

--scope user makes it available in every project. Without it, Claude Code registers the server only for the folder you ran the command in.

Codex (~/.codex/config.toml)

[mcp_servers.titration]
command = "npx"
args = ["tsx", "/absolute/path/to/titration/server/server.ts"]

Cursor (.cursor/mcp.json)

{
  "mcpServers": {
    "titration": { "command": "npx", "args": ["tsx", "/absolute/path/to/titration/server/server.ts"] }
  }
}

The server reads .env from the clone itself, so no secrets go into client config. Restart your agent after adding the server or changing its environment, so it starts a fresh MCP process. For normal use, omit project and leave TITRATION_PROJECT unset or blank. See Automatic repository memory for context requirements and refusals.

Optional: agent skills

skills/ contains three skills that walk an agent through the whole flow — scout (find what is worth measuring), harness (design and validate the measurement), improve (baseline, then improve against it). See skills/README.md.

Choosing judges

The first time your agent establishes a baseline, it calls referee_panel_mint and a small page opens in your browser (served on 127.0.0.1 only). Pick three judges from three different vendor families; your agent's own family is greyed out, and so is the vendor of the model your app calls, when the agent passes it as sut_model (a judge may favour its own vendor). The panel is locked to that baseline, so every later comparison uses the same judges. (Diagnosing a failure with classify_failure is not grading, so that panel may include your agent's own vendor.)

| Door | Cost | Notes | | --- | --- | --- | | claude CLI | your Claude subscription | verified | | codex CLI | your ChatGPT subscription | verified | | grok CLI | your Grok subscription | built; unverified — help wanted | | OpenRouter | pay per token | 12 curated models across 9 vendor families |

Edit judges-roster.json to change the pool. To add an OpenRouter model, first record its proof calls with npx tsx scripts/record-openrouter-fixtures.ts <slug> (a fraction of a cent). For unattended runs or CI, set TITRATION_JUDGES in .env to a comma-separated list of roster ids, or to auto to let Titration pick three families you have access to (subscription CLIs first, never the Player's family).

What grading costs: each judged output is one call per judge. A 20-output baseline with a three-judge panel is 60 judge calls — free on subscription CLIs (within their usage limits), and billed per token on OpenRouter models.

Tools

| Area | Tools | | --- | --- | | Memory | card_search, card_get, card_create, card_relate, card_distill, run_capture, propose_cards, edge_propose | | Measurement design | harness_design, harness_validate | | Grading | referee_panel_mint, referee_panel_status, establish_baseline, verify, classify_failure, job_status | | Improvement loop | goal_titrate, goal_titrate_step |

Automatic repository memory

For the 14 stateful tools above (all except classify_failure, harness_design, propose_cards, and harness_validate), standard stdio automatically selects repository memory. Omit project and leave TITRATION_PROJECT unset or blank. No tenant name needs to be computed.

The MCP client must supply local repository roots. Valid file URIs and unambiguous raw absolute local paths use the same normalization and strict validation; Windows paths with spaces are supported. Supplied roots always win over the server's launch directory. Failed lookup, empty, malformed/relative/nonlocal roots, or roots resolving to different repositories refuse rather than guess a first root or use default.

Clients without roots support (observed: Codex CLI 0.160) start the server in the project they opened. Only when a client advertises no roots capability, the server uses that launch directory instead, with the same identity rules. It refuses if the launch directory is not inside a Git repository, or is inside the Titration clone the server runs from (a launch there says nothing about your project). Multiple roots are accepted only when they resolve to the same identity. Git must be available to the server, and the supplied roots must resolve to usable Git metadata.

Identity uses current Git evidence: origin wins, otherwise the sole remote; verified local remote links are followed, and a repository with no remote uses its canonical Git common directory. Branches, worktrees, subdirectories and clones share memory in the same database when they resolve to equal current anchors. On github.com and gitlab.com only, HTTPS, SSH and SCP-style remotes (for example https://github.com/team/repo.git, ssh://git@github.com/team/repo.git and git@github.com:team/repo.git) share an anchor, with an optional .git suffix. SSH forms require an explicit git user for this equivalence. Elsewhere, transport/address form, SSH user and the exact path including .git remain distinct: an absolute SSH path is not a home-relative SCP path. Equivalent self-hosted addresses may therefore select separate memory; identical addresses still share. Git's explicit insteadOf configuration is applied by remote get-url before identity is derived. Unsupported or ambiguous forms, including explicit ports and SSH passwords, refuse conservatively. Basenames, shared history and a fork's upstream do not establish equivalence; un