syntect

syntect is a syntax highlighting library for Rust that uses Sublime Text syntax definitions. It aims to be a good solution for any Rust project that needs syntax highlighting, including deep integration with text editors written in Rust. It's used in production by at least two companies, and by many open source projects.

If you are writing a text editor (or something else needing highlighting) in Rust and this library doesn't fit your needs, I consider that a bug and you should file an issue or email me. I consider this project mostly complete, I still maintain it and review PRs, but it's not under heavy development.

Important Links

• API docs with examples: https://docs.rs/syntect
• Changelogs and upgrade notes for past releases

Getting Started

syntect is available on crates.io. You can add it to your Cargo.toml with the following command

cargo add syntect

After that take a look at the documentation and the examples.

If you've cloned this repository, be sure to run

git submodule update --init

to fetch all the required dependencies for running the tests.

Features/Goals

• [x] Work with many languages (accomplished through using existing grammar formats)
• [x] Highlight super quickly, faster than nearly all text editors
• [x] Include easy to use API for basic cases
• [x] API allows use in fancy text editors with piece tables and incremental re-highlighting and the like.
• [x] Expose internals of the parsing process so text editors can do things like cache parse states and use semantic info for code intelligence
• [x] High quality highlighting, supporting things like heredocs and complex syntaxes (like Rust's).
• [x] Include a compressed dump of all the default syntax definitions in the library binary so users don't have to manage a folder of syntaxes.
• [x] Well documented, I've tried to add a useful documentation comment to everything that isn't utterly self explanatory.
• [x] Built-in output to coloured HTML <pre> tags or 24-bit colour ANSI terminal escape sequences.
• [x] Nearly complete compatibility with Sublime Text 3, including lots of edge cases. Passes nearly all of Sublime's syntax tests, see issue 59.
• [x] Load up quickly, currently in around 23ms but could potentially be even faster.

Screenshots

There's currently an example program called syncat that prints one of the source files using hard-coded themes and syntaxes using 24-bit terminal escape sequences supported by many newer terminals. These screenshots don't look as good as they could for two reasons: first the sRGB colours aren't corrected properly, and second the Rust syntax definition uses some fancy labels that these themes don't have highlighting for.

Example Code

Prints highlighted lines of a string to the terminal. See the easy and html module docs for more basic use case examples.

use syntect::easy::HighlightLines;
use syntect::parsing::SyntaxSet;
use syntect::highlighting::{ThemeSet, Style};
use syntect::util::{as_24_bit_terminal_escaped, LinesWithEndings};

// Load these once at the start of your program
let ps = SyntaxSet::load_defaults_newlines();
let ts = ThemeSet::load_defaults();

let syntax = ps.find_syntax_by_extension("rs").unwrap();
let mut h = HighlightLines::new(syntax, &ts.themes["base16-ocean.dark"]);
let s = "pub struct Wow { hi: u64 }\nfn blah() -> u64 {}";
for line in LinesWithEndings::from(s) {
    let ranges: Vec<(Style, &str)> = h.highlight_line(line, &ps).unwrap();
    let escaped = as_24_bit_terminal_escaped(&ranges[..], true);
    print!("{}", escaped);
}

Performance

Currently syntect is one of the faster syntax highlighting engines, but not the fastest. The following perf features are done:

• [x] Pre-link references between languages (e.g <script> tags) so there are no tree traversal string lookups in the hot-path
• [x] Compact binary representation of scopes to allow quickly passing and copying them around
• [x] Determine if a scope is a prefix of another scope using bit manipulation in only a few instructions
• [x] Cache regex matches to reduce number of times oniguruma is asked to search a line
• [x] Accelerate scope lookups to reduce how much selector matching has to be done to highlight a list of scope operations
• [x] Lazily compile regexes so startup time isn't taken compiling a thousand regexes for Actionscript that nobody will use
• [ ] Optionally use the fancy-regex crate. Unfortunately this isn't yet faster than oniguruma on our benchmarks but it might be in the future.

The current perf numbers are below. These numbers may get better if more of the things above are implemented, but they're better than many other text editors. All measurements were taken on a mid 2012 15" retina Macbook Pro, my new 2019 Macbook takes about 70% of these times.

• Highlighting 9200 lines/247kb of jQuery 2.1 takes 600ms. For comparison:
  • Textmate 2, Spacemacs and Visual Studio Code all take around 2ish seconds (measured by hand with a stopwatch, hence approximate).
  • Atom takes 6 seconds
  • Sublime Text 3 dev build takes 98ms (highlighting only, takes ~200ms click to pixels), despite having a super fancy javascript syntax definition.
  • Vim is instantaneous but that isn't a fair comparison since vim's highlighting is far more basic than the other editors. Compare vim's grammar to Sublime's.
  • These comparisons aren't totally fair, except the one to Sublime Text since that is using the same theme and the same complex definition for ES6 syntax.
• Simple syntaxes are faster, JS is one of the most complex. It only takes 34ms to highlight a 1700 line 62kb XML file or 50,000 lines/sec.
• ~138ms to load and link all the syntax definitions in the default Sublime package set.
  • but only ~23ms to load and link all the syntax definitions from an internal pre-made binary dump with lazy regex compilation.
• ~1.9ms to parse and highlight the 30 line 791 character testdata/highlight_test.erb file. This works out to around 16,000 lines/second or 422 kilobytes/second.
• ~250ms end to end for syncat to start, load the definitions, highlight the test file and shut down. This is mostly spent loading.

Feature Flags

Syntect makes heavy use of cargo features, to support users who require only a subset of functionality. In particular, it is possible to use the highlighting component of syntect without the parser (for instance when hand-rolling a higher performance parser for a particular language), by adding default-features = false to the syntect entry in your Cargo.toml.

For more information on available features, see the features section in Cargo.toml.

Pure Rust fancy-regex mode, without onig

Since 4.0 syntect offers an alternative pure-rust regex engine based on the fancy-regex engine which extends the awesome regex crate with support for fancier regex features that Sublime syntaxes need like lookaheads.

The advantage of fancy-regex is that it does not require the onig crate which requires building and linking the Oniguruma C library. Many users experience difficulty building the onig crate, especially on Windows and Webassembly.

As far as our tests can tell this new engine is just as correct, but it hasn't been tested as extensively in production. It also currently seems to be about half the speed of the default Oniguruma engine, although further testing and optimization (perhaps by you!) may eventually see it surpass Oniguruma's speed and become the default.

To use the fancy-regex engine with syntect, add it to your Cargo.toml like so:

syntect = { version = "4.2", default-features = false, features = ["default-fancy"]}

If you want to run examples with the fancy-regex engine you can use a command line like the following:

cargo run --features default-fancy --no-default-features --release --example syncat testdata/highlight_test.erb

Due to the way Cargo features work, if any crate you depend on depends on syntect without enabling fancy-regex then you'll get the default onig mode.

Note: The fancy-regex engine is absurdly slow in debug mode, because the regex engine (the main hot spot of highlighting) is now in Rust instead of C that's always built with optimizations. Consider using release mode or onig when testing.

Caching

Because syntect's API exposes internal cacheable data structures, there is a caching strategy that text edito