trimtok
Cut token usage in Claude Code, Cursor, Codex & Gemini: deterministic in-process context compression for AI coding agents. Lossless and retrievable (#ref), AST code skeletons, TOON JSON tables, stdio MCP. No LLM, no network, MIT.
Install / Use
claude mcp add Stxyu-p -- npx -y github:Stxyu-p/trimtokIf the server publishes to npm under a different name, use that package instead — check the repo README.
MCP Server
Model Context Protocol server
Quality Score
Category
AI & Machine LearningSupported Platforms
Our assessment of trimtok
trimtok scores 83/100 on our quality scale, 661st of 947 AI & Machine Learning skills we index.
Its MCP Server is 39 KB long, well organised into 66 sections with 25 code examples: a thorough specification that gives an agent plenty to work with.
It has 3 GitHub stars, so there is little community track record yet; judge it on its content.
Maintenance, license and trust
- The repository was last updated yesterday, so trimtok is actively maintained.
- It is released under the MIT license, a permissive license that allows use, modification and commercial use with attribution.
- Its trust signals score 92/100, with 1 caution from licensing, adoption, age or documentation. These come from repository metadata, not a code audit — read the skill file before letting an agent act on it.
Safety scan
No issues foundOur scan of the whole file found no instruction hijacking, hidden characters, credential access, data exfiltration or destructive commands. An AI review of the same text found nothing harmful.
AI review by kimi-k2.7-code on 2026-11-15. Automated pattern scan on 2026-11-15. It catches known dangerous patterns, not every risk — read a skill before letting an agent act on it.
trimtok compared with similar skills
All 4 of these similar skills score higher than trimtok; compare them before choosing.
| Skill | Score | Stars | Updated | Format |
|---|---|---|---|---|
| trimtok (this skill)by Stxyu-p | 83 | 3 | 1d ago | MCP Server |
| claude-memby thedotmack | 100 | 117.6k | today | CLAUDE.md |
| Agent-Reachby Panniantong | 100 | 112.3k | 8d ago | CLAUDE.md |
| Understand-Anythingby Egonex-AI | 100 | 92.8k | 1d ago | CLAUDE.md |
| rufloby ruvnet | 100 | 81.9k | today | CLAUDE.md |
Frequently asked questions
- How do I install trimtok?
- Run
claude mcp add Stxyu-p -- npx -y github:Stxyu-p/trimtok. The install tabs above show the steps for each supported agent. - Which AI agents does trimtok work with?
- It is written for Claude Code, Claude Desktop, Cursor, Gemini CLI and OpenAI Codex, as a MCP Server file. Other agents that read the same format can often use it too.
- Is trimtok safe to use?
- Our scan of the whole file found no instruction hijacking, hidden characters, credential access, data exfiltration or destructive commands. An AI review of the same text found nothing harmful. It is MIT-licensed and scores 92/100 on trust signals. Skills are instructions an agent will follow, so read the file before installing it and do not approve commands you do not understand.
- Is trimtok still maintained?
- The repository was last updated yesterday, so trimtok is actively maintained.
Skill content
View source on GitHub<a id="at-a-glance"></a>
⚡ At a Glance: Before vs After
Autonomous coding agents burn over 60% of their context window budget on repetitive, low-entropy noise. trimtok compresses this in-process without losing a single line of actionable truth:
| Target Workload | Raw Input | trimtok Compact | Token Reduction | Core Mechanism |
| :--- | :---: | :---: | :---: | :--- |
| pytest Test Suite (200 tests) | 56,000 chars | 320 chars | -99.4% | Cross-turn stateful delta ([tt:delta]) |
| Source Code File (300 functions) | 13,500 toks | 2,400 toks | -82.2% | Multi-language AST skeletonizer |
| FastAPI / Python Traceback | 12,400 chars | 890 chars | -92.8% | Crash-site anchor & framework frame prune |
| Web Docs & Search Scraping | 15,200 chars | 1,800 chars | -88.1% | Tiered anchor & link de-bloating |
| MCP Tool Schemas (25 tools) | 5,800 toks | 780 toks | -86.5% | Bounded JSON-RPC stdio proxy |
| Dense Uniform JSON Array | 8,900 chars | 3,100 chars | -65.1% | Lossless columnar TOON v2 encoding |
| Mixed-Schema JSON Data | 11,240 chars | 6,630 chars | -41.0% | Multi-schema merged TOON tables |
| Build & Compiler Logs | 15,960 toks | 4,869 toks | -69.5% | Adaptive error-rail balanced windowing |
| Whole Repo Codebase Map | 12,000 toks | 480 toks | -96.0% | 2k-budget hierarchical tree map |
The 32-payload corpus evaluation is a reproducible fixture run (1,135,441 characters): 76.07% character reduction (863,784 chars) with zero gate failures. Every removed character is verifiable through trimtok_retrieve; eval/audit_honesty.py reports 76.07% claimed and proven, with 0 unproven characters. This corpus result is not a production-traffic estimate.
Production telemetry reports characters, not measured model tokens. usage.jsonl records transform attempts across gates, while audit.jsonl records successful compression events; they are different populations and are rotated independently. Do not combine their totals into one live-traffic percentage or convert characters to tokens without a model-specific tokenizer. Use trimtok gain (optionally with a time window) for the retained usage-log view; older retained logs cannot reproduce historical all-time claims.
Rows in the table are single-payload spot checks over the shapes stored in eval/corpus/; they are not the headline figure. The audited number is the corpus line above, which eval/audit_honesty.py recomputes from the recovery store.
<a id="how-it-works"></a>
🏗️ How It Works
trimtok operates 100% locally and in-process using the Python standard library, RapidFuzz C++ alignment, optional tree-sitter AST parsing (pip install "trimtok[multilang]"), and Zstandard compression. No external daemons, no Docker containers, and zero open network ports. The default install parses Python with the stdlib ast; the other grammar packs are opt-in.
┌────────────────────────────────────────────────────────┐
│ Tier 0: Pre-Flight Context Shield (Zero Disk Footprint)│
│ ├── Bounded MCP Proxy: filters tools/list schemas │
│ └── Codebase Symbol Map: 2k-budget symbol directory map│
└──────────────────────────┬─────────────────────────────┘
▼
Tool / Terminal Output
│
▼
┌────────────────────────────────────────────────────────┐
│ Tier 1: In-Process Hot-Path (sub-ms typ., guard 200ms) │
│ ├── 0. Content Policy Gate (secrets / edits verbatim) │
│ ├── 1. Cross-Turn Delta Engine (pytest, cargo test) │
│ ├── 2. Command-Specific Filters (git, npm, pip, etc.) │
│ ├── 3. Structured Traceback Pruning (locks crash-site) │
│ ├── 4. Lossless & Merged TOON v2 (JSON arrays) │
│ ├── 5. Multi-Language AST Skeletonizer (opt-in grammar)│
│ └── 6. Tiered Web & Doc Link De-bloating │
└──────────────────────────┬─────────────────────────────┘
│ level: light │ normal │ aggressive
│ format: TOON │ minified JSON
┌────────────────┴────────────────┐
▼ ▼
Agent Context Window SQLite WAL + Zstandard
(Clean, dense token stream) (Tier 2 Recovery via `#c1`)
📑 Table of Contents
| 🏛️ Architecture & Verification | ⚡ Core Engineering Pillars | 🛠️ Setup & Operations |
| :--- | :--- | :--- |
| • ⚡ At a Glance: Before vs After<br>• 🏗️ How It Works<br>• 🥊 Architecture & Benchmark Matrix<br>• 🔬 Empirical Quality & Fidelity | • Pillar 1: Terminal Compression<br> ▫ Cross-Turn Delta Engine<br> ▫ Command-Specific Filters<br> ▫ Structured Traceback Pruning<br> ▫ Fault-Signature Folding<br>• Pillar 2: Intake Shield<br> ▫ Tiered Web & File Intake<br> ▫ AST Code Skeletonizer<br> ▫ Lossless TOON v2 (JSON)<br>• Pillar 3: Zero Data Loss<br> ▫ CCR Recovery (#ref)<br> ▫ Pointer Tier (pointer_large)<br> ▫ Telemetry & Diagnostics<br> ▫ Fail-Open Budget Guards<br>• Pillar 4: Policy & Verification<br> ▫ Content Policy Gate<br> ▫ Adaptive Compression Level<br> ▫ Model-Aware Output Format<br> ▫ Fix-Accuracy Verification | • First-Time Setup (5 Mins)<br> ▫ Step 1: Install trimtok<br> ▫ Step 2: Setup Wizard<br> ▫ Step 3: Health Verification<br>• Hermes Plugin Setup<br> ▫ Configuration Options<br>• Slash Commands & CLI<br> ▫ In-Chat Slash Commands<br> ▫ KV-Cache Stabilizer (kv-lock)<br> ▫ Codebase Map (map)<br> ▫ Bounded MCP Proxy (mcp-proxy)<br> ▫ Standalone Wrapper (wrap)<br> ▫ Command Rewriting (rewrite)<br>• Autonomous Agent Ecosystem<br>• Verification & Tests<br>• License (MIT) |
<a id="market-comparison"></a>
🥊 Architecture & Benchmark Matrix: In-Process vs Alternatives
Direct architectural comparison across context compression paradigms:
| Architectural Dimension | trimtok 🌟 | Subprocess CLI (e.g. RTK) | Local HTTP Proxy (e.g. Distil / Headroom) |
| :--- | :---: | :---: | :---: |
| Execution Layer | In-process hook + CLI | External subprocess fork | Local HTTP daemon / reverse proxy |
| Runtime Overhead | sub-ms hook (microsecond in-process call) | 10 – 50 ms (process fork) | proxy queue + network hop |
| Data Loss Model | Lossless CCR (SQLite WAL + Zstd) | ❌ Irreversible truncation | ⚠️ Disk cache / ephemeral handles |
| Ref Intelligence | ✅ trimtok_exec (compute on #ref) | ❌ None | ⚠️ Whole-block retrieve only |
| Cross-Turn State | ✅ [tt:delta] (>95% on reruns) | ❌ None (stateless per command) | ❌ None (stateless stream) |
| Structured JSON | ✅ Lossless TOON v2 (55% – 65%) | ⚠️ Lossy head sample (99.6% lossy) | ⚠️ Reversible / dictionary encoder |
| Diff Safety Gate | ✅ 100% changed lines preserved | ⚠️ Fixed window line sampling | ⚠️ Token budget truncation |
| Prompt Alignment | ✅ kv-lock (KV-Cache friendly) | ❌ None | ⚠️ Sliding window prefix cache |
| Network Footprint | ✅ 0 ports (100% in-process) | ✅ 0 ports | ❌ Opens local TCP port |
🎯 Why In-Process Context Intelligence Wins
- Deterministic & Sub-2ms: Runs directly inside agent runtime hooks with zero daemon processes, zero network ports, and fail-open deadline guards (
guard_ms: 200.0). - Zero Data Loss Guarantee: Every omitted byte is backed by SQLite WAL + Zstandard. Agents recall slices via
trimtok_retrieveor execute Python aggregate queries viatrimtok_execwithout inflating prompt context. - Cross-Turn Stateful Delta: Repeated test suites (
pytest,cargo test) and git operations track cross-turn state, surfacing only new regressions and active diffs.
<a id="empirical-verification"></a>
🔬 Empirical Quality & Task Fidelity Verification
Context compression is only valuable if downstream language models retain 100% of reasoning accuracy and task comprehension. To prove that trimtok preserves actionable intelligence without inducing information loss or "lost-in-the-middle" degradation, the pipeline is evaluated across model inference, needle retention, and adversarial stress suites.
1. End-to-End LLM Task Fidelity Benchmark
Evaluated across diverse tool-use tasks (Python tracebacks, function return annotations, lockfile configs, structured logs) by presenting paired identical queries against Raw vs. trimtok-Compressed payloads to coding models:
| Metric | Raw Baseline | trimtok Compressed | Impact / Advantage |
| :--- | :---: | :---: | :---: |
| Total Prompt Tokens | 22,092 tokens | 10,326 tokens | -53.3% tokens saved |
| Task Accuracy | 80.0% (4/5) | 100.0% (5/5) | +20.0% (n=5, indicative only) |
| Information Fidelity | 100.0% | 100.0% | Zero hallucinations / zero loss |
Why Compressed Input Yielded Higher Accuracy: On oversized raw payloads, models frequently suffer from Lost-in-the-Middle degradation (e.g. failing to locate function signatures buried inside 4,000+ token raw code files). trimtok's selective skeletonization eliminates noise, enabling the model to achieve higher answer precision on compressed inputs than on uncompressed raw text.
This run is
n=5on hand-built tasks. It shows the mechanism is plausible and that compression did not cost accuracy here; it is not a statistically established accuracy gain. Read the 4/5 → 5/5 delta as indicative, not as a headline number.
2. Needle-In-A-Haystack (NIAH) Retention Benchmark
Concealed critical operational signals (security overrides, replica split-brain fault codes, diagnostic signatures) at depths of 25%, 50% (exact center), and 75% within dense 1,200-line server log streams:
| Needle Depth | Target Payload | Raw Retrieval | Compressed Retrieval | Retention Status | | :---: | :--- | :---: | :---: | :---: | | 25% Depth | `CRITICA
Truncated for display — read the full file on GitHub.
Related Skills
claude-mem
117.6kPersistent Context Across Sessions for Every Agent – Captures everything your agent does during sessions, compresses it with AI, and injects relevant context back into future sessions. Works with Claude Code, OpenClaw, Codex, Gemini, Hermes, Copilot, OpenCode + More
Agent-Reach
112.3kGive your AI agent eyes to see the entire internet. Read & search Twitter, Reddit, YouTube, GitHub, Bilibili, XiaoHongShu — one CLI, zero API fees.
Understand-Anything
92.8kGraphs that teach > graphs that impress. Turn any code into an interactive knowledge graph you can explore, search, and ask questions about. Works with Claude Code, Codex, Cursor, Copilot, Gemini CLI, and more.
ruflo
81.9k🌊 The original agent harness. Deploy intelligent multi-player swarms, coordinate autonomous workflows, and build conversational AI systems. Features adaptive memory, self-learning intelligence, federation, vector RAG integration, and native Claude Code / Codex / Hermes and many more Integrated
Languages
Trust signals
From repository metadata: license, adoption, age and documentation. Not a code audit — see the Safety scan above for what the skill file itself contains.
