TONL - Token-Optimized Notation Language

</div>

TONL (Token-Optimized Notation Language)

TONL is a production-ready data platform that combines compact serialization with powerful query, modification, indexing, and streaming capabilities. Designed for LLM token efficiency while providing a rich API for data access and manipulation.

🎉 Latest Release: v2.5.2 - Documentation & Testing Excellence

📚 v2.5.2 (December 20, 2025)

216 new tests - Total 698 tests across 162 suites with 100% pass rate
Browser documentation - Complete docs/BROWSER.md and docs/ERROR_HANDLING.md
4 browser examples - React 18 and Vue 3 interactive demos
5 security fixes - All npm vulnerabilities resolved
Updated dependencies - All packages at latest versions

🔧 v2.5.1 (December 11, 2025)

8 critical bug fixes including DoS prevention and async handling

🛡️ v2.5.0 (December 3, 2025)

Enterprise security hardening and optimization module

🧪 Testing Excellence:

698 Comprehensive Tests - All passing with 100% success rate
96 Security Tests - Covering all attack vectors
Concurrency Tests - Thread safety validation
Browser Tests - Cross-platform compatibility

🏠 Homepage: tonl.dev 📦 GitHub: github.com/tonl-dev/tonl 📖 Documentation: Complete Guides

Why TONL?

🗜️ Up to 60% Smaller - Reduce JSON size and LLM token costs 👁️ Human-Readable - Clear text format, not binary 🚀 Blazingly Fast - 10-1600x faster than targets 🔒 Production Secure - 100% security hardened (v2.0.3) 🛠️ TypeScript-First - Full type safety & IntelliSense 📦 Zero Dependencies - Pure TypeScript, no bloat 🌐 Browser Ready - 10.5 KB gzipped bundle (IIFE/UMD) ✅ 100% Tested - 496/496 tests passing (core functionality)

🚀 Quick Start

Installation

npm install tonl

Basic Usage

import { TONLDocument, encodeTONL, decodeTONL } from 'tonl';

// Create from JSON
const doc = TONLDocument.fromJSON({
  users: [
    { id: 1, name: "Alice", role: "admin", age: 30 },
    { id: 2, name: "Bob", role: "user", age: 25 }
  ]
});

// Query with JSONPath-like syntax
doc.get('users[0].name');                          // 'Alice'
doc.query('users[*].name');                        // ['Alice', 'Bob']
doc.query('users[?(@.role == "admin")]');          // [{ id: 1, ... }]
doc.query('$..age');                               // All ages recursively

// Aggregation (v2.4.0)
doc.count('users[*]');                             // 2
doc.sum('users[*]', 'age');                        // 55
doc.avg('users[*]', 'age');                        // 27.5
doc.groupBy('users[*]', 'role');                   // { admin: [...], user: [...] }
doc.aggregate('users[*]').stats('age');            // { count, sum, avg, min, max, stdDev }

// Fuzzy Matching (v2.4.0)
import { fuzzySearch, soundsLike } from 'tonl/query';
fuzzySearch('Jon', ['John', 'Jane', 'Bob']);       // [{ value: 'John', score: 0.75 }]
soundsLike('Smith', 'Smyth');                      // true

// Temporal Queries (v2.4.0)
import { parseTemporalLiteral, isDaysAgo } from 'tonl/query';
parseTemporalLiteral('@now-7d');                   // 7 days ago
isDaysAgo(someDate, 30);                           // within last 30 days?

// Modify data
doc.set('users[0].age', 31);
doc.push('users', { id: 3, name: "Carol", role: "editor", age: 28 });

// Navigate and iterate
for (const [key, value] of doc.entries()) {
  console.log(key, value);
}

doc.walk((path, value, depth) => {
  console.log(`${path}: ${value}`);
});

// Export
const tonl = doc.toTONL();
const json = doc.toJSON();
await doc.save('output.tonl');

// Classic API (encode/decode)
const data = { users: [{ id: 1, name: "Alice" }] };
const tonlText = encodeTONL(data);
const restored = decodeTONL(tonlText);

// Advanced Optimization (v2.0.1+)
import { AdaptiveOptimizer, BitPacker, DeltaEncoder } from 'tonl/optimization';

// Automatic optimization
const optimizer = new AdaptiveOptimizer();
const result = optimizer.optimize(data);  // Auto-selects best strategies

// Specific optimizers
const packer = new BitPacker();
const packed = packer.packBooleans([true, false, true]);

const delta = new DeltaEncoder();
const timestamps = [1704067200000, 1704067201000, 1704067202000];
const compressed = delta.encode(timestamps, 'timestamp');

CLI Usage

🎮 Interactive CLI (NEW v2.3.1)

# Interactive stats dashboard
tonl stats data.json --interactive
tonl stats data.json -i --theme neon

# File comparison mode
tonl stats data.json --compare --theme matrix

# Interactive exploration
tonl stats --interactive  # Launch without file for menu-driven exploration

📊 Standard Commands

# Get started (shows help)
tonl

# Version info
tonl --version

# Encode JSON to TONL (perfect round-trip, quotes special keys)
tonl encode data.json --out data.tonl --smart --stats

# Encode with preprocessing (clean, readable keys)
tonl encode data.json --preprocess --out data.tonl

# Decode TONL to JSON
tonl decode data.tonl --out data.json

# Query data
tonl query users.tonl "users[?(@.role == 'admin')]"
tonl get data.json "user.profile.email"

# Validate against schema
tonl validate users.tonl --schema users.schema.tonl

# Format and prettify
tonl format data.tonl --pretty --out formatted.tonl

# Compare token costs
tonl stats data.json --tokenizer gpt-5

🎨 Interactive Themes (v2.3.1)

# Available themes: default, neon, matrix, cyberpunk
tonl stats data.json -i --theme neon        # Bright neon colors
tonl stats data.json -i --theme matrix      # Green matrix style
tonl stats data.json -i --theme cyberpunk   # Cyan/purple cyberpunk
tonl stats data.json -i --theme default     # Clean terminal colors

⚖️ File Comparison (v2.3.1)

# Compare JSON and TONL files side-by-side
tonl stats data.json --compare
tonl stats data.json --compare --theme neon

# Interactive comparison mode
tonl stats data.json -i --compare

📊 Format Overview

Arrays of Objects (Tabular Format)

JSON (245 bytes, 89 tokens):

{
  "users": [
    { "id": 1, "name": "Alice", "role": "admin" },
    { "id": 2, "name": "Bob, Jr.", "role": "user" },
    { "id": 3, "name": "Carol", "role": "editor" }
  ]
}

TONL (158 bytes, 49 tokens - 45% reduction):

#version 1.0
users[3]{id:u32,name:str,role:str}:
  1, Alice, admin
  2, "Bob, Jr.", user
  3, Carol, editor

Nested Objects

JSON:

{
  "user": {
    "id": 1,
    "name": "Alice",
    "contact": {
      "email": "alice@example.com",
      "phone": "+123456789"
    },
    "roles": ["admin", "editor"]
  }
}

TONL:

#version 1.0
user{id:u32,name:str,contact:obj,roles:list}:
  id: 1
  name: Alice
  contact{email:str,phone:str}:
    email: alice@example.com
    phone: +123456789
  roles[2]: admin, editor

✨ Complete Feature Set

🔄 Core Serialization

Compact Format - 32-45% smaller than JSON (bytes + tokens)
Human-Readable - Clear text format with minimal syntax
Round-Trip Safe - Perfect bidirectional JSON conversion
Smart Encoding - Auto-selects optimal delimiters and formatting
Type Hints - Optional schema information for validation

🔍 Query & Navigation API

JSONPath Queries - users[?(@.age > 25)], $..email
Filter Expressions - ==, !=, >, <, &&, ||, contains, matches
Wildcard Support - users[*].name, **.email
Tree Traversal - entries(), keys(), values(), walk()
LRU Cache - >90% cache hit rate on repeated queries

✏️ Modification API

CRUD Operations - set(), get(), delete(), push(), pop()
Bulk Operations - merge(), update(), removeAll()
Change Tracking - diff() with detailed change reports
Snapshots - Document versioning and comparison
Atomic File Edits - Safe saves with automatic backups

⚡ Performance & Indexing

Hash Index - O(1) exact match lookups
BTree Index - O(log n) range queries
Compound Index - Multi-field indexing
Stream Processing - Handle multi-GB files with <100MB memory
Pipeline Operations - Chainable filter/map/reduce transformations

🗜️ Advanced Optimization

Dictionary Encoding - Value compression via lookup tables (30-50% savings)
Delta Encoding - Sequential data compression (40-60% savings)
Run-Length Encoding - Repetitive value compression (50-80% savings)
Bit Packing - Boolean and small integer bit-level compression (87.5% savings)
Numeric Quantization - Precision reduction for floating-point numbers (20-40% savings)
Schema Inheritance - Reusable column schemas across data blocks (20-40% savings)
Hierarchical Grouping - Common field extraction for nested structures (15-30% savings)
Tokenizer-Aware - LLM tokenizer optimization for minimal token usage (5-15% savings)
Column Reordering - Entropy-based ordering for better compression
**Adap

Tonl

Install / Use

README