Riteway

The standard testing framework for AI Driven Development (AIDD) and software agents.

Riteway is a testing assertion style and philosophy which leads to simple, readable, helpful unit tests for humans and AI agents.

It lets you write better, more readable tests with a fraction of the code that traditional assertion frameworks would use, and the riteway ai CLI lets you write AI agent prompt evals as easily as you would write a unit testing suite.

Riteway is the AI-native way to build a modern test suite. It pairs well with Vitest, Playwright, Claude Code, Cursor Agent, Google Antigravity, and more.

• Readable
• Isolated/Integrated
• Thorough
• Explicit

Riteway forces you to write Readable, Isolated, and Explicit tests, because that's the only way you can use the API. It also makes it easier to be thorough by making test assertions so simple that you'll want to write more of them.

Why Riteway for AI Driven Development?

Riteway's structured approach makes it ideal for AIDD:

📖 Learn more: Better AI Driven Development with Test Driven Development

• Clear requirements: The given, should expectations and 5-question framework help AI better understand exactly what to build
• Readable by design: Natural language descriptions make tests comprehensible to both humans and AI
• Simple API: Minimal surface area reduces AI confusion and hallucinations
• Token efficient: Concise syntax saves valuable context window space

The 5 Questions Every Test Must Answer

There are 5 questions every unit test must answer. Riteway forces you to answer them.

1. What is the unit under test (module, function, class, whatever)?
2. What should it do? (Prose description)
3. What was the actual output?
4. What was the expected output?
5. How do you reproduce the failure?

Installing

npm install --save-dev riteway

Then add an npm command in your package.json:

"test": "riteway test/**/*-test.js",

For projects using both core Riteway tests and JSX component tests, you can use a dual test runner setup:

"test": "node source/test.js && vitest run",

Now you can run your tests with npm test. Riteway also supports full TAPE-compatible usage syntax, so you can have an advanced entry that looks like:

"test": "nyc riteway test/**/*-rt.js | tap-nirvana",

In this case, we're using nyc, which generates test coverage reports. The output is piped through an advanced TAP formatter, tap-nirvana that adds color coding, source line identification and advanced diff capabilities.

Requirements

Riteway requires Node.js 16+ and uses native ES modules. Add "type": "module" to your package.json to enable ESM support. For JSX component testing, you'll need a build tool that can transpile JSX (see JSX Setup below).

riteway ai — AI Prompt Evaluations

The riteway ai CLI runs your AI agent prompt evaluations against a configurable pass-rate threshold. Write a .sudo test file, run it through any supported AI agent, and get a TAP-formatted report with per-assertion pass rates across multiple runs.

Authentication

All agents use OAuth authentication — no API keys needed. Authenticate once before running evals:

| Agent | Command | Docs | |-------|---------|------| | Claude | claude setup-token | Claude Code docs | | Cursor | agent login | Cursor docs | | OpenCode | See docs | opencode.ai/docs/cli |

Writing a test file

AI evals are written in .sudo files using SudoLang syntax:

# my-feature-test.sudo

import 'path/to/spec.mdc'

userPrompt = """
Implement the sum function as described.
"""

- Given the spec, should name the function sum
- Given the spec, should accept two parameters named a and b
- Given the spec, should return the correct sum of the two parameters

Each - Given ..., should ... line becomes an independently judged assertion. The agent is asked to respond to the userPrompt (with any imported spec as context), and a judge agent scores each assertion across all runs.

Running an eval

riteway ai path/to/my-feature-test.sudo

By default this runs 4 passes, requires 75% pass rate, uses the claude agent, runs up to 4 tests concurrently, and allows 300 seconds per agent call.

# Specify runs, threshold, and agent
riteway ai path/to/test.sudo --runs 10 --threshold 80 --agent opencode

# Use a Cursor agent with color output
riteway ai path/to/test.sudo --agent cursor --color

# Use a custom agent config file (mutually exclusive with --agent)
riteway ai path/to/test.sudo --agent-config ./my-agent.json

Options

| Flag | Default | Description | |------|---------|-------------| | --runs N | 4 | Number of passes per assertion | | --threshold P | 75 | Required pass percentage (0–100) | | --timeout MS | 300000 | Per-agent-call timeout in milliseconds | | --agent NAME | claude | Agent: claude, opencode, cursor, or a custom name from riteway.agent-config.json | | --agent-config FILE | — | Path to a flat single-agent JSON config {"command","args","outputFormat"} — mutually exclusive with --agent | | --concurrency N | 4 | Max concurrent test executions | | --color | off | Enable ANSI color output | | --save-responses | off | Save raw agent responses and judge details to a companion .responses.md file |

Results are written as a TAP markdown file under ai-evals/ in the project root.

Saving raw responses for debugging

When --save-responses is passed, a companion .responses.md file is written alongside the .tap.md output. It contains the raw result agent response and per-run judge details (passed, actual, expected, score) for every assertion — useful for debugging failures without adding console noise.

riteway ai path/to/test.sudo --save-responses

Each test file produces its own uniquely-named pair of files (e.g. 2026-03-17-test-abc12.tap.md and 2026-03-17-test-abc12.responses.md), so multiple test files never conflict.

Capturing responses as CI artifacts

In GitHub Actions, use --save-responses and upload the ai-evals/ directory as an artifact:

- name: Run AI prompt evaluations
  run: npx riteway ai path/to/test.sudo --save-responses

- name: Upload AI eval responses
  if: always()
  uses: actions/upload-artifact@v4
  with:
    name: ai-eval-responses
    path: ai-evals/*.responses.md
    retention-days: 14

The if: always() ensures responses are uploaded even when assertions fail, so you can inspect exactly what the agent produced.

Partial results on timeout

If some runs complete before another times out, the completed runs' responses are still written to the responses file. The timed-out run's partial agent output is also captured, followed by a [RITEWAY TIMEOUT] marker showing when and where the timeout occurred. This lets you debug why a run took too long and potentially optimize the prompt to run faster.

Custom agent configuration

riteway ai init writes all built-in agent configs to riteway.agent-config.json in your project root, so you can add custom agents or tweak existing flags:

riteway ai init           # create riteway.agent-config.json
riteway ai init --force   # overwrite existing file

The generated file is a keyed registry. Add a custom agent entry and use it with --agent:

{
  "claude":   { "command": "claude",   "args": ["-p", "--output-format", "json", "--no-session-persistence"], "outputFormat": "json"  },
  "opencode": { "command": "opencode", "args": ["run", "--format", "json"],                                   "outputFormat": "ndjson" },
  "cursor":   { "command": "agent",    "args": ["--print", "--output-format", "json"],                        "outputFormat": "json"  },
  "my-agent": { "command": "my-tool",  "args": ["--json"],                                                    "outputFormat": "json"  }
}

riteway ai path/to/test.sudo --agent my-agent

Once riteway.agent-config.json exists, any agent key defined in it supersedes the library's built-in defaults for that agent.

---

Example Usage

import { describe, Try } from 'riteway/index.js';

// a function to test
const sum = (...args) => {
  if (args.some(v => Number.isNaN(v))) throw new TypeError('NaN');
  return args.reduce((acc, n) => acc + n, 0);
};

describe('sum()', async assert => {
  const should = 'return the correct sum';

  assert({
    given: 'no arguments',
    should: 'return 0',
    actual: sum(),
    expected: 0
  });

  assert({
    given: 'zero',
    should,
    actual: sum(2, 0),
    expected: 2
  });

  assert({
    given: 'negative numbers',
    should,
    actual: sum(1, -4),
    expected: -3
  });

  assert({
    given: 'NaN',
    should: 'throw',
    actual: Try(sum, 1, NaN),
    expected: new TypeError('NaN')
  });  
});

Testing React Components

import render from 'riteway/render-component';
import { describe } from 'riteway/index.js';

describe('renderComponent', async assert => {
  const $ = render(<div className="foo">testing</div>);

  assert({
    given: 'some jsx',
    should: 'render markup',
    actual: $('.foo').html().trim(),
    expected: 'testing'
  });
});

N