tokf

tokf.net — reduce LLM context consumption from CLI commands by 60–90%.

Commands like git push, cargo test, and docker build produce verbose output packed with progress bars, compile noise, and boilerplate. tokf intercepts that output, applies a TOML filter, and emits only what matters — so your AI agent sees a clean signal instead of hundreds of wasted tokens.

Before / After

cargo test — 61 lines → 1 line:

<table> <tr> <th>Without tokf</th> <th>With tokf</th> </tr> <tr> <td>

   Compiling tokf v0.2.0 (/home/user/tokf)
   Compiling proc-macro2 v1.0.92
   Compiling unicode-ident v1.0.14
   Compiling quote v1.0.38
   Compiling syn v2.0.96
   Compiling serde_derive v1.0.217
   Compiling serde v1.0.217
   ...
running 47 tests
test config::tests::test_load ... ok
test filter::tests::test_skip ... ok
test filter::tests::test_keep ... ok
test filter::tests::test_extract ... ok
...
test result: ok. 47 passed; 0 failed; 0 ignored
  finished in 2.31s

</td> <td>

✓ 47 passed (2.31s)

</td> </tr> </table>

git push — 8 lines → 1 line:

<table> <tr> <th>Without tokf</th> <th>With tokf</th> </tr> <tr> <td>

Enumerating objects: 5, done.
Counting objects: 100% (5/5), done.
Delta compression using up to 10 threads
Compressing objects: 100% (3/3), done.
Writing objects: 100% (3/3), 312 bytes | 312.00 KiB/s, done.
Total 3 (delta 2), reused 0 (delta 0), pack-reused 0
remote: Resolving deltas: 100% (2/2), completed with 2 local objects.
To github.com:user/repo.git
   a1b2c3d..e4f5a6b  main -> main

</td> <td>

ok ✓ main

</td> </tr> </table>

Quickstart

brew install mpecan/tokf/tokf   # or: cargo install tokf
tokf setup                      # detect your AI tools and install hooks

That's it. Every command your AI agent runs is now automatically filtered.

Run tokf gain to see how many tokens you've saved, or tokf setup --refresh to re-run detection.

Installation

Homebrew (macOS and Linux)

brew install mpecan/tokf/tokf

cargo

cargo install tokf

Build from source

git clone https://github.com/mpecan/tokf
cd tokf
cargo build --release
# binary at target/release/tokf

How it works

tokf run git push origin main

tokf looks up a filter for git push, runs the command, and applies the filter. The filter logic lives in plain TOML files — no recompilation required. Anyone can author, share, or override a filter.

Set up automatic filtering

If you use an AI coding tool, install the hook so every command is filtered automatically — no tokf run prefix needed:

# Claude Code (recommended: --global so it works in every project)
tokf hook install --global

# OpenCode
tokf hook install --tool opencode --global

# OpenAI Codex CLI
tokf hook install --tool codex --global

Drop --global to install for the current project only. See Claude Code hook for details on each tool, the --path flag, and optional extras like the filter-authoring skill.

Usage

Run a command with filtering

tokf run git push origin main
tokf run cargo test
tokf run docker build .

Apply a filter to a fixture

tokf apply filters/git/push.toml tests/fixtures/git_push_success.txt --exit-code 0

Verify filter test suites

tokf verify                    # run all test suites
tokf verify git/push           # run a specific suite
tokf verify --list             # list available suites and case counts
tokf verify --json             # output results as JSON
tokf verify --require-all      # fail if any filter has no test suite
tokf verify --list --require-all  # show coverage per filter
tokf verify --scope project    # only project-local filters (.tokf/filters/)
tokf verify --scope global     # only user-level filters (~/.config/tokf/filters/)
tokf verify --scope stdlib     # only built-in stdlib (filters/ in CWD)
tokf verify --safety           # run safety checks (prompt injection, shell injection, hidden unicode)
tokf verify git/push --safety  # safety check a specific filter

Task runner filtering

tokf automatically wraps make and just so that each recipe line is individually filtered:

make check    # each recipe line (cargo test, cargo clippy, ...) is filtered
just test     # same — each recipe runs through tokf

See Rewrite configuration for details and customization.

Explore available filters

tokf ls                    # list all filters
tokf which "cargo test"    # which filter would match
tokf show git/push         # print the TOML source

Customize a built-in filter

tokf eject cargo/build            # copy to .tokf/filters/ (project-local)
tokf eject cargo/build --global   # copy to ~/.config/tokf/filters/ (user-level)

This copies the filter TOML and its test suite to your config directory, where it shadows the built-in. Edit the ejected copy freely — tokf's priority system ensures your version is used instead of the original.

Flags

| Flag | Description | |---|---| | --timing | Print how long filtering took | | --verbose | Show which filter was matched (also explains skipped rewrites) | | --no-filter | Pass output through without filtering | | --no-cache | Bypass the filter discovery cache | | --no-mask-exit-code | Disable exit-code masking. By default tokf exits 0 and prepends Error: Exit code N on failure | | --preserve-color | Preserve ANSI color codes in filtered output (env: TOKF_PRESERVE_COLOR=1). See Color passthrough below | | --baseline-pipe | Pipe command for fair baseline accounting (injected by rewrite) | | --prefer-less | Compare filtered vs piped output and use whichever is smaller (requires --baseline-pipe) |

Color passthrough

By default, filters with strip_ansi = true permanently remove ANSI escape codes. The --preserve-color flag changes this: tokf strips ANSI internally for pattern matching (skip, keep, dedup) but restores the original colored lines in the final output. When --preserve-color is active it overrides strip_ansi = true in the filter config.

tokf does not force commands to emit color — you must ensure the child command outputs ANSI codes (e.g. via FORCE_COLOR=1 or --color=always):

# Node.js / Vitest / Jest
FORCE_COLOR=1 tokf run --preserve-color npm test

# Cargo
tokf run --preserve-color cargo test -- --color=always

# Or set the env var once for all invocations
export TOKF_PRESERVE_COLOR=1
FORCE_COLOR=1 tokf run npm test

Limitations: color passthrough applies to the skip/keep/dedup pipeline (stages 2–2.5). The match_output, parse, and lua_script stages operate on clean text and are unaffected by this flag. [[replace]] rules run on the raw text before the color split, so when --preserve-color is enabled their patterns may need to account for ANSI escape codes, similar to branch-level skip patterns, which also match against the restored colored text.

Built-in filter library

| Filter | Command | |---|---| | git/add | git add | | git/commit | git commit | | git/diff | git diff | | git/log | git log | | git/push | git push | | git/show | git show | | git/status | git status — runs git status --porcelain -b; shows branch name + one porcelain-format line per changed file (e.g. M src/main.rs, ?? scratch.rs) | | cargo/build | cargo build | | cargo/check | cargo check | | cargo/clippy | cargo clippy | | cargo/fmt | cargo fmt | | cargo/install | cargo install * | | cargo/test | cargo test | | docker/* | docker build, docker compose, docker images, docker ps | | npm/run | npm run * | | npm/test | npm test, pnpm test, yarn test (with vitest/jest variants) | | pnpm/* | pnpm add, pnpm install | | go/* | go build, go vet | | gradle/* | gradle build, gradle test, gradle dependencies | | gh/* | gh pr list, gh pr view, gh pr checks, gh issue list, gh issue view | | kubectl/* | kubectl get pods | | next/* | next build | | prisma/* | prisma generate | | pytest | Python test runner | | tsc | TypeScript compiler | | ls | ls |

Generic Commands

When no dedicated filter exists for a command, three built-in subcommands provide useful compression for arbitrary output:

| Command | Purpose | Default context | |---------|---------|----------------| | tokf err <cmd> | Extract errors and warnings | 3 lines | | tokf test <cmd> | Extract test failures | 5 lines | | tokf summary <cmd> | Heuristic summary | 30 lines max |

`tokf err` — Error extraction

Scans output for error/warning patterns across common toolchains (Rust, Python, Node, Go, Java) and shows only the relevant lines with surrounding context.

# Show only errors from a build
tokf err cargo build

# Adjust context lines around each error
tokf err -C 5 cargo build

# Works with any command
tokf err python train.py
tokf err npm run build

Patterns matched: error:, warning:, FAILED, Traceback, panic, npm ERR!, fatal:, Python/Java exception types, and more.

Behaviour:

Empty output: [tokf err] no errors detected (empty output)
Short output (< 10 lines): shows [tokf err] header with full output
No errors + exit 0: prints [tokf err] no errors detected
No errors + exit ≠ 0: includes full output (something failed but no recognized pat

Tokf

Install / Use

README

tokf

Before / After

Quickstart

Installation

Homebrew (macOS and Linux)

cargo

Build from source

How it works

Set up automatic filtering

Usage

Run a command with filtering

Apply a filter to a fixture

Verify filter test suites

Task runner filtering

Explore available filters

Customize a built-in filter

Flags

Color passthrough

Built-in filter library

Generic Commands

`tokf err` — Error extraction

Tokf

Install / Use

README

tokf

Before / After

Quickstart

Installation

Homebrew (macOS and Linux)

cargo

Build from source

How it works

Set up automatic filtering

Usage

Run a command with filtering

Apply a filter to a fixture

Verify filter test suites

Task runner filtering

Explore available filters

Customize a built-in filter

Flags

Color passthrough

Built-in filter library

Generic Commands

tokf err — Error extraction

`tokf err` — Error extraction