Tenko

A "pixel perfect" 100% spec compliant JavaScript parser written in JavaScript, parsing ES6/ES2015 - ES2025.

REPL: https://pvdz.github.io/tenko/repl

Supports:
- Anything stage 4 up to and including ES2025
- Explicit resource ("using") stage 3 (under flag)
- Regex syntax (deep)
- Parsing modes:
  - Sloppy / non-strict
  - Web compat / AnnexB
  - Strict
  - Module
AST
- Is optional, enabled by default
- Estree (default)
- Acorn
- Babel (anything stage 4, except comments, didn't check after ES2021 tbh)
- Supports location data (matching Acorn/Babel for reference)
Tests
- 29k input syntax tests
- Passes test262 suite (at least as per March 2026), without exception (./t t)

Name

The name is short for "The Parser Formerly Known As ZeParser3".

It's also an anagram for "Token", perfectly fitting this project.

In Japanese it's a divine beast ("heavenly fox" or "celestial fox"), playing into my nicknames.

REPL

You can find the REPL in repl/index.html, github link: https://pvdz.github.io/tenko/repl

The REPL runs on dev master branch and needs a very new browser due to es module syntax.

Usage

import {Tenko, GOAL_MODULE, COLLECT_TOKENS_ALL} from 'src/index.mjs';
const {
  ast,                 // estree compatible AST
  tokens,              // array of numbers (see Lexer)
  tokenCountSolid,     // number of non-whitespace tokens
  tokenCountAny,       // number of tokens of any kind
} = Tenko(
  inputCode,           // string
  {
    // Parse with script or module goal (module allows import/export)
    goalMode = GOAL_MODULE, // GOAL_MODULE | GOAL_SCRIPT | "module" | "script"
    // Do you want to collect generated tokens at all?
    collectTokens = COLLECT_TOKENS_ALL, // COLLECT_TOKENS_ALL | COLLECT_TOKENS_SOLID | COLLECT_TOKENS_NONE | COLLECT_TOKENS_TYPES | "all" | "solid" | "none" | "types"
    // Apply Annex B rules? (Only works in sloppy mode)
    webCompat = true,
    // Start parsing as if in strict mode? (Works with script goal)
    strictMode = false,
    // Output a Babel compatible AST? Note: comment nodes are not properly mirrored
    babelCompat = false,
    // Add a loc (with `{start: {line, column}, stop: {line, column}}`) to each token?
    babelTokenCompat = false,
    // Pass on a reference that will be used as the AST root
    astRoot = null,
    // Should it normalize \r and \r\n to \n in the .raw of template nodes?
    // Estree spec but makes it hard to serialize template nodes losslessly
    templateNewlineNormalization = true,
    // Pass on a reference to store the tokens
    tokenStorage = [],
    // Callback to receive the lexer instance once its created
    getLexer = null, // getLexer(lexer)
    // You use this to parse `eval` code
    allowGlobalReturn = false,
    // Target a very specific ecmascript version (like, reject async). Number; 6 - 16, or 2015 - 2025, or Infinity.
    targetEsVersion = lastVersion, // (Last supported version is currently ES2025)
    // Top-level await in Module: undefined = allow when target ES2022+; true = force on; false = force off
    toplevelAwait = undefined,
    // Explicit opt-in for `using` and `await using` declarations (not yet stage 4 at time of writing)
    allowUsingDeclaration = false,
    // Leave built up scope information in the ASTs (good luck)
    exposeScopes = false,
    // Assign each node a unique incremental id
    astUids = false,
    // Do you want to print a code frame with error messages? (Part of the input around the point of error)
    errorCodeFrame = true,
    // For the code frame, do you want to always show the entire input, regardless of size? Or just a small context
    truncCodeFrame = true,
    // You can override the logging functions to catch or squash all output
    $log = console.log,
    $warn = console.warn,
    $error = console.error,
    // Value ot use for the `source` field of each `loc` object
    sourceField = '',
    // Generate a `range: {start: number, end: number}` property on all loc objects (does not require `locationTracking`)
    ranges = false,
    // Generate a `range: [start: number, end: number]` property on all nodes. `input.slice(range[0], range[1])` should get you the text for a node.
    nodeRange = false,
    // Do not populate loc properties on AST nodes (property will be undefined). Since v<unpublished>
    locationTracking = true,
    // Prevent syntax error for octal escapes regardless of strict mode or anything else
    alwaysAllowOctalEscapes = false,
  }
);

Development

There is a single entry point in the root project called t which calls tests/t.sh which calls out to various development related scripts.

ES modules

Note that the files use import and export declarations and import(), which requires node 10+ or a cutting edge browser.

At the time of writing node requires the experimental --experimental-modules flag.

It's a burden in some ways and nice in others. A prod build would not have any modules.

Test cases

All test cases are in "special" plain-text .md files. See tests/testcases/README.md for details on formatting those.

Entry point

Some interesting usages of ./t:

# Show help
./t --help

# Run and update (inline) all tests. 
# Use git diff to see changes. Will bail fast on unexpected or assertion errors.
# This tests four modes (sloppy, strict, module, and sloppy-web-compat)
# This also tests the printer on the first successful parse
./t u
# Run all tests step-by-step (same as above) and ask what to do for any changes
./t m

# Same as `./t u` but compare it against Babel or Acorn. Recorded changes should be discarded afterwards.
# Use this to test against AST differences. If there are any they will be printed explicitly.
# Acorn:
./t a
# Babel:
./t b

# Test a particular input from cli
./t i "some.input()"
# Test a particular test file
./t f "tests/testcases/regexes/foo.md"
# Test a particular test file or folder and force-write the result(s)
./t ff "tests/testcases/regexes"
# Use entire contents of given file as input
./t F "test262/test/annexB/built-ins/foo.js"

# Generate prod builds
# Generate a build. Strips ASSERT*, inline many constants
./t z
# Same as above but explicitly set `acornCompat` and `babelCompat` to `false`.
./t z --no-compat
# Generate pretty builds for debugging without asserts:
./t --pretty
# Minified build with Terser (will lower performance due to inlining)
./t --min

# Run test262 tests (requires cloning https://github.com/tc39/test262 into tenko/ignore/test262)
./t t

# Fuzz the parser
./t fuzz

# Regenerate all autogen test files. Regenerates files still need to be updated (`./t u`).
# All files, regardless:
./t g
# Only create new files:
./t G

# Find out which tests execute a particular code branch in the parser
# Add `HIT()` to any part of the code in src
# Reports (only) all inputs that trigger a `HIT()` call in Tenko
./t s

Some tooling that requires additional setup;

# Benchmarks (requires benchmark files in projroot/ignore/perf);
# Simply spawn new node process and run test:
./t p
# Run benchmarks repeatedly and report results
./t p6
# Configure machine to be as stable as possible (DANGEROUS, read the script before using it, requires root). All 
# changes should be reset after reboot. Then run the benchmarks in the shielded cpus at RT prio (also requires root).
./t stable
./t p6 --stabled
# Same as above but without running `./t stable` previously, and tries to undo certain (but not all) things afterwards
./t p6 --stable

# Investigate v8 perf regressions with deoptigate:
./t deoptigate

# Profile the parser in Chrome devtools (open the tab through `about://inspect`)
./t devtools

# Run a visual heatmap profiler for counts based investigation (private)
./t hf

There are many flags. Some are specific to an action, others are generic. Some examples:

--sloppy             Run in non-strict mode (but non-web compat!)
--strict             Run with script goal but consider the code strict
--module             Run with module goal (enabling strict mode by default)
--web                Run with script goal, non-strict, and enable web compat (AnnexB rules)
--annexb             Force enable AnnexB rules, regardless of mode

6                    Run as close to the rules as of ES6  / ES2015 as possible
7                    Run as close to the rules as of ES7  / ES2016 as possible
8                    Run as close to the rules as of ES8  / ES2017 as possible
9                    Run as close to the rules as of ES9  / ES2018 as possible
10                   Run as close to the rules as of ES10 / ES2019 as possible
11                   Run as close to the rules as of ES11 / ES2020 as possible
12                   Run as close to the rules as of ES12 / ES2021 as possible
13                   Run as close to the rules as of ES13 / ES2022 as possible (ie. top-level await)
14                   Run as close to the rules as of ES14 / ES2023 as possible (ie. hashbang)
15                   Run as close to the rules as of ES15 / ES2024 as possible (ie. RegExp v flag / unicode sets)
16                   Run as close to the rules as of ES16 / ES2025 as possible (ie. import with, RegExp modifiers)
2015
2016
2017
2018
2019
2020
2021
2022
2023
2024
2025

--min                Given a broken input, brute force minify the input while maintaining the same error message
--acorn              Output a Acorn compatible AST
--babel              Output a Babel compatible AST
--test-acorn         Compare the `--acorn` output to the actual output of Acorn on same input (./t a)
--test-babel         Compare the `--babel` output to the actual output of Babel on same input (./t b)
--test-node          Compile input in a `Function()` and report whether that throws when Tenko throws, for fuzzer
--build              Use a prod build (from standard output location), inst

Tenko

Install / Use

README

Tenko

Name

REPL

Usage

Development

ES modules

Test cases

Entry point