Yozora

<header> <h1 align="center"> <a href="https://github.com/yozorajs/yozora#readme">Yozora</a> <div align="center"> <img alt="logo.png" src="./logo.png" width="400px" /> </div> </h1> <div align="center"> <a href="#license"> <img alt="License" src="https://img.shields.io/github/license/yozorajs/yozora" /> </a> <a href="https://github.com/yozorajs/yozora/tags"> <img alt="Package Version" src="https://img.shields.io/github/v/tag/yozorajs/yozora?include_prereleases&sort=semver" /> </a> <a href="https://github.com/yozorajs/yozora/search?l=typescript"> <img alt="Github Top Language" src="https://img.shields.io/github/languages/top/yozorajs/yozora" /> </a> <a href="https://github.com/nodejs/node"> <img alt="Node.js Version" src="https://img.shields.io/node/v/@yozora/core-tokenizer" /> </a> <a href="https://github.com/yozorajs/yozora/actions/workflows/ci.yml"> <img alt="CI Workflow" src="https://github.com/yozorajs/yozora/actions/workflows/ci.yml/badge.svg" /> </a> <a href="https://github.com/facebook/jest"> <img alt="Tested with Jest" src="https://img.shields.io/badge/tested_with-jest-9c465e.svg" /> </a> <a href="https://github.com/prettier/prettier"> <img alt="Code Style: prettier" src="https://img.shields.io/badge/code_style-prettier-ff69b4.svg?style=flat-square" /> </a> </div> </header> <br />

See [Yozora document][yozora-docs] (or https://yozorajs.github.io) for more details.

https://user-images.githubusercontent.com/42513619/129205123-6a1983c4-6a86-4c80-83d6-02bdbf70edbf.mp4

<br />

中文文档

🎉 Why named "yozora" ?

Yozora is the Roman sound of Japanese 「よぞら」, taken from the lyrics in『花鳥風月』 by the band 世界の終わり.

This project is a monorepo that aims to implement a highly extensible, pluggable Markdown parser. Based on the idea of middlewares, the core algorithm [@yozora/core-parser][] will schedule tokenizers (such as [@yozora/tokenizer-autolink][]) to complete the parsing tasks. More accurately, yozora is an algorithm to parse Markdown or its extended syntax contents into an abstract syntax tree (AST).

✨ Features

• 🔖 Fully support all the rules mentioned in the [GFM specification][gfm-spec], and has passed almost all test cases created based on the examples in the specification (except the one https://github.github.com/gfm/#example-653, as there is no plan to support native HTML tags in the [React Renderer][yozora-react], for the Yozora AST, so I'm a little lazy to do the tag filtering. If you need it, you can do the filtering by yourself).

  See [@yozora/parser-gfm] or [@yozora/parser-gfm-ex] for further information.

• 🚀 Robust.

  • All codes are written in Typescript, with the guarantee of strictly static type checking.

  • Eslint and Prettier to constrain coding styles to avoid error-prone problems such as hack syntax and shadow variables.

  • Tested with Jest, and passed a large number of test cases.

• 💚 Tidy: No third-party dependencies.

• ⚡️ Efficient.

  • The parsing complexity is the length of source contents multiplied by the number of tokenizers, which has reached the lower bound of theoretical complexity.

  • The parser API supports streaming read-in (using generators /iterators for input), and supports parsing while read-in (Only block-level data is supported yet).

  • Carefully handle the array creation / concat operations. To reused the array as much as possible during the entire matching phase, only use the array index to delineate the matching range. And a lot of strategies applied to reduce duplicated matching / parsing operations.

• 🩹 Compatibility, the parsed syntax tree is compatible with the one defined in [Mdast][mdast-homepage].

  Even if some data types are not compatible in the future, it is easy to traverse the AST for adaptation and modification through the API provided in [@yozora/ast-util][].

• 🎨 Extendibility, Yozora comes with a plug-in system, which allowed Yozora to schedule the tokenizers through an internal algorithms to complete the parsing tasks.

  • It's easy to create and integrate custom tokenizers.

  • All tokenizers can be mounted or unmounted freely.

    Some tokenizers of the data types that not mentioned in [GFM][gfm-spec] have been implemented in this repository, such as [@yozora/tokenizer-admonition][], [@yozora/tokenizer-footnote][], etc. All of them are built into [@yozora/parser][] in default, you can uninstall them at will, if you don't like it.

Usage

• [@yozora/parser][]: (Recommended) A Markdown parser with rich built-in tokenizers.

  import YozoraParser from '@yozora/parser'
  
  const parser = new YozoraParser()
  parser.parse('source content')
  

• [@yozora/parser-gfm][]: A Markdown parser that supports [GFM specification][gfm-spec]. Built-in tokenizers that supports all grammars mentioned in [GFM specification][gfm-spec] (excluding the extended grammar mentioned in the specification, such as [table][@yozora/tokenizer-table]).

  import GfmParser from '@yozora/parser-gfm'
  
  const parser = new GfmParser()
  parser.parse('github flavor markdown contents')
  

• [@yozora/parser-gfm-ex][]: A Markdown parser that supports [GFM specification][gfm-spec]. Built-in tokenizers that supports all grammars mentioned in [GFM specification][gfm-spec] (including the extended grammar mentioned in the specification, such as [table][@yozora/tokenizer-table]).

  import GfmExParser from '@yozora/parser-gfm-ex'
  
  const parser = new GfmExParser()
  parser.parse('github flavor markdown contents (with gfm extensions enabled)')
  

• Content AST into markup content

  import { DefaultMarkupWeaver } from '@yozora/markup-weaver'
  
  const weaver = new DefaultMarkupWeaver()
  weaver.weave({
    "type": "root",
    "children": [
      {
        "type": "paragraph",
        "children": [
          {
            "type": "text",
            "value": "emphasis: "
          },
          {
            "type": "strong",
            "children": [
              {
                "type": "text",
                "value": "foo \""
              },
              {
                "type": "emphasis",
                "children": [
                  {
                    "type": "text",
                    "value": "bar"
                  }
                ]
              },
              {
                "type": "text",
                "value": "\" foo"
              }
            ]
          }
        ]
      }
    ]
  })
  // => emphasis: **foo "*bar*" foo**
  

Overview

• Parsers

  | Parser | Description | | :------------------------ | :--------------------------------------------------------------------------------------- | | [@yozora/parser][] | A markdown parser with rich built-in tokenizers | | [@yozora/parser-gfm][] | A markdown parser with built-in tokenizers to fully support GFM (without GFM extensions) | | [@yozora/parser-gfm-ex][] | A markdown parser with built-in tokenizers to fully support GFM and GFM extensions |

• Weavers

  | Weaver | Description | | :------------------------ | :----------------------------- | | [@yozora/markup-weaver][] | Weave AST into markup content. |

• Tokenizers

  | Tokenizer | Description | | :---------------------------------------- | :----------------------------------------------------------------------------------------------------------------- | | [@yozora/tokenizer-admonition][] | Resolve admonitions | | [@yozora/tokenizer-autolink][] | Resolve [GFM Autolinks][gfm-autolink] | | [@yozora/tokenizer-autolink-extension][] | Resolve [GFM Autolinks (extension)][gfm-autolink-extension] | | [@yozora/tokenizer-blockquote][] | Resolve [GFM blockquotes][gfm-blockquote] | | [@yozora/tokenizer-break][] | Resolve [GFM hard line breaks][gfm-hard-line-break] and [GFM soft line breaks][gfm-soft-line-break] | | [@yozora/tokenizer-definition][] | Resolve [GFM link reference definitions][gfm-link-reference] | | [@yozora/tokenizer-delete][] | Resolve [GFM strikethrough (extension)][gfm-delete] | | [@yozora/tokenizer-ecma-import][] | Resolve ECMAScript import statements | | [@yozora/tokenizer-emphasis][] | Resolve [GFM emphasis and strong emphasis][gfm-emphasis] | | [@yozora/tokenizer-fenced-code][] | Resolve [GFM fenced code blocks][gfm-fenced-code] | | [@yozora/tokenizer-footnote][] | Resolve footnotes | | [@yozora/tokenizer-footnote-definition][] | Resolve footnote definitions