SweetLine Syntax Highlighting Engine

Overview

SweetLine is a cross-platform, high-performance, and extensible syntax highlighting engine designed for modern code editors and code display scenarios. Built on the Oniguruma regex engine and a finite state machine model, it processes large code files in real-time with accurate syntax highlighting.

Screenshots

Core Features

High Performance

Built on Oniguruma regex engine for fast pattern matching
Incremental update algorithm that only reanalyzes changed portions, ideal for real-time editor highlighting
Multi-line state preservation to avoid full document reanalysis

High Accuracy

Finite State Machine (FSM) based model supporting complex syntax rule nesting
Multiple capture group style mapping for fine-grained highlighting control
SubStates mechanism for handling nested syntax structures (e.g., generics, template parameters)
Zero-width match support for context-sensitive state transitions

Highly Extensible

JSON-based syntax rule configuration – add new language support without writing code
Variable substitution, fragments (include / includes), and pattern reuse to reduce rule redundancy
33 built-in language syntax rules (Java, C/C++, Python, Kotlin, Rust, Go, TypeScript, etc.)

Cross-Platform

Core engine written in C++17
C API wrapper for easy FFI integration
Native support for Android (JNI), Java 22 (FFM), WebAssembly (Emscripten), HarmonyOS (NAPI), .NET/WinForms (P/Invoke)
Supports Windows, Linux, macOS and other desktop platforms

Architecture Overview

┌────────────────────────────────────────────────────────────────────────────────────┐
│                              SweetLine Architecture                               │
├────────────────────────────────────────────────────────────────────────────────────┤
│ Application / Platform Bindings                                                   │
│  Android(JNI) | Java22(FFM) | .NET/C#(P/Invoke) | WASM(Emscripten)               │
│  HarmonyOS(NAPI) | C API(FFI) | C++ Native API                                    │
├────────────────────────────────────────────────────────────────────────────────────┤
│                         SweetLine C++ Core (C++17)                                │
│                                                                                    │
│  ┌──────────────────────┐        ┌──────────────────────────────┐                 │
│  │    HighlightEngine   │ -----> │  SyntaxRule Compiler (JSON)  │                 │
│  └──────────┬───────────┘        └──────────────────────────────┘                 │
│             │                                                                      │
│    ┌────────▼─────────┐       ┌──────────────────────┐                            │
│    │   TextAnalyzer   │       │   DocumentAnalyzer   │                            │
│    │   (Full Scan)    │       │   (Incremental)      │                            │
│    └────────┬─────────┘       └──────────┬───────────┘                            │
│             │                            │                                         │
│             │                   ┌────────▼─────────┐                               │
│             │                   │  Document Model  │                               │
│             │                   │ (Managed Text)   │                               │
│             │                   └──────────────────┘                               │
│             │                                                                      │
│             └──────────────┬───────────────────────┘                               │
│                            ▼                                                       │
│          ┌────────────────────────────────────────────────────────────┐            │
│          │ Regex + FSM Runtime (Oniguruma + State Machine)           │            │
│          └──────────────────────────┬─────────────────────────────────┘            │
│                                     ▼                                              │
│              Highlight Result + Scope/Indent Guide Analysis                        │
└────────────────────────────────────────────────────────────────────────────────────┘

Quick Start

C++ Usage

#include "highlight.h"
using namespace sweetline;

// 1. Create highlight engine
auto engine = std::make_shared<HighlightEngine>();

// 2. Compile syntax rules
auto rule = engine->compileSyntaxFromFile("syntaxes/java.json");

// 3. Create document object
auto document = std::make_shared<Document>("file:///example.java", R"(
public class HelloWorld {
    public static void main(String[] args) {
        System.out.println("Hello, World!");
    }
}
)");

// 4. Load document and analyze
auto analyzer = engine->loadDocument(document);
auto highlight = analyzer->analyze();

// 5. Iterate highlight results
for (size_t i = 0; i < highlight->lines.size(); i++) {
    auto& line = highlight->lines[i];
    for (auto& span : line.spans) {
        // span.range  - text range (line/column/index)
        // span.style_id - style ID (keyword=1, string=2, ...)
    }
}

Incremental Updates

// When the document is edited, only reanalyze the changed portion
TextRange change_range { {2, 4}, {2, 8} };
std::string new_text = "modified";
auto new_highlight = analyzer->analyzeIncremental(change_range, new_text);

// Read only the visible lines from the latest cached highlight result
LineRange visible_range {100, 60};
auto visible_slice = analyzer->getHighlightSlice(visible_range);

// Or combine patch + visible slice in one call
auto updated_slice = analyzer->analyzeIncrementalInLineRange(change_range, new_text, visible_range);

Use getHighlightSlice() after analyze() or analyzeIncremental() when the renderer only needs a visible window of lines.

Java 22 (FFM) Usage

import com.qiplat.sweetline.*;

try (HighlightEngine engine = new HighlightEngine(new HighlightConfig(true, false))) {
    engine.compileSyntaxFromFile("syntaxes/java.json");

    try (TextAnalyzer analyzer = engine.createAnalyzerByName("java")) {
        DocumentHighlight result = analyzer.analyzeText(sourceCode);
    }
}

Java 22 FFM wrapper is located at platform/Java22. Running code requires native access enabled (for example --enable-native-access=ALL-UNNAMED) and a resolvable SweetLine native library path.

Android Usage

// build.gradle
implementation 'com.qiplat:sweetline:1.2.0'

// Create engine
HighlightEngine engine = new HighlightEngine(new HighlightConfig());

// Compile syntax rules
engine.compileSyntaxFromJson(jsonString);

// Full analysis
TextAnalyzer analyzer = engine.createAnalyzerByName("java");
DocumentHighlight result = analyzer.analyzeText(sourceCode);

// Iterate results
for (LineHighlight line : result.lines) {
    for (TokenSpan span : line.spans) {
        // span.range, span.styleId
    }
}

WebAssembly Usage

import { sweetline } from './libsweetline.js';

// Create engine
const config = new sweetline.HighlightConfig();
const engine = new sweetline.HighlightEngine(config);

// Compile syntax rules
engine.compileSyntaxFromJson(jsonString);

// Analyze text
const analyzer = engine.createAnalyzerByName("javascript");
const highlight = analyzer.analyzeText(sourceCode);

// Iterate results
for (let i = 0; i < highlight.lines.size(); i++) {
    const line = highlight.lines.get(i);
    for (let j = 0; j < line.spans.size(); j++) {
        const span = line.spans.get(j);
        // span.range, span.styleId
    }
}

Custom Syntax Rules

SweetLine uses JSON to define syntax rules. Here is a simple example:

{
  "name": "myLanguage",
  "fileExtensions": [".mylang"],
  "variables": {
    "identifier": "[a-zA-Z_]\\w*"
  },
  "fragments": {
    "commonLiterals": [
      { "pattern": "\"(?:[^\"\\\\]|\\\\.)*\"", "style": "string" },
      { "pattern": "//[^\\n]*", "style": "comment" }
    ]
  },
  "states": {
    "default": [
      {
        "pattern": "\\b(if|else|while|return)\\b",
        "styles": [1, "keyword"]
      },
      { "include": "commonLiterals" }
    ]
  }
}

For complete syntax rule configuration, see the Syntax Rule Configuration Guide.

Documentation

| Document | Description | |----------|-------------| | Syntax Rule Configuration Guide | Detailed guide on writing JSON syntax rule files | | API Reference (Index) | API entry page and reading order | | Core API | Core concepts and C++ API | | C API | C interface for FFI integration | | Android API | Java/Kotlin API on Android | | Java 22 API | Java 22 FFM API | | .NET / WinForms API | C# API (P/Invoke wrapper) | | WebAssembly API | JavaScript/TypeScript API | | HarmonyOS API | ArkTS/NAPI API usage | | Build Guide | Multi-platform build commands and options | | Contributing Guide | How to participate in the project |

Built-in Language Support

| Language | File | Language | File | |----------|------|----------|------| | Java | java.json | Python | python.json | | C | c.json | C++ | c++.json | | C# | `csharp.j

SweetLine

Install / Use

README