A dead simple parser package for Go

V2
Introduction
Tutorial
Tag syntax
Overview
Grammar syntax
Capturing
- Capturing boolean value
"Union" types
Custom parsing
Lexing
Options
Examples
Performance
Concurrency
Error reporting
Comments
Limitations
EBNF
Syntax/Railroad Diagrams

V2

This is version 2 of Participle.

It can be installed with:

$ go get github.com/alecthomas/participle/v2@latest

The latest version from v0 can be installed via:

$ go get github.com/alecthomas/participle@latest

<a name='Introduction'></a>Introduction

The goal of this package is to provide a simple, idiomatic and elegant way of defining parsers in Go.

Participle's method of defining grammars should be familiar to any Go programmer who has used the encoding/json package: struct field tags define what and how input is mapped to those same fields. This is not unusual for Go encoders, but is unusual for a parser.

Tutorial

A tutorial is available, walking through the creation of an .ini parser.

Tag syntax

Participle supports two forms of struct tag grammar syntax.

The easiest to read is when the grammar uses the entire struct tag content, eg.

Field string `@Ident @("," Ident)*`

However, this does not coexist well with other tags such as JSON, etc. and may cause issues with linters. If this is an issue then you can use the parser:"" tag format. In this case single quotes can be used to quote literals making the tags somewhat easier to write, eg.

Field string `parser:"@ident (',' Ident)*" json:"field"`

Overview

A grammar is an annotated Go structure used to both define the parser grammar, and be the AST output by the parser. As an example, following is the final INI parser from the tutorial.

type INI struct {
  Properties []*Property `@@*`
  Sections   []*Section  `@@*`
}

type Section struct {
  Identifier string      `"[" @Ident "]"`
  Properties []*Property `@@*`
}

type Property struct {
  Key   string `@Ident "="`
  Value *Value `@@`
}

type Value struct {
  String *string  `  @String`
  Float *float64  `| @Float`
  Int    *int     `| @Int`
}

Note: Participle also supports named struct tags (eg. <code>Hello string `parser:"@Ident"`</code>).

A parser is constructed from a grammar and a lexer:

parser, err := participle.Build[INI]()

Once constructed, the parser is applied to input to produce an AST:

ast, err := parser.ParseString("", "size = 10")
// ast == &INI{
//   Properties: []*Property{
//     {Key: "size", Value: &Value{Int: &10}},
//   },
// }

Grammar syntax

Participle grammars are defined as tagged Go structures. Participle will first look for tags in the form parser:"...". It will then fall back to using the entire tag body.

The grammar format is:

@<expr> Capture expression into the field.
@@ Recursively capture using the field’s own type.
<identifier> Match named lexer token.
( ... ) Group.
"..." or '...' Match the literal (note that the lexer must emit tokens matching this literal exactly).
"...":<identifier> Match the literal, specifying the exact lexer token type to match.
<expr> <expr> ... Match expressions.
<expr> | <expr> | ... Match one of the alternatives. Each alternative is tried in order, with backtracking.
~<expr> Match any token that is not the start of the expression (eg: @~";" matches anything but the ; character into the field).
(?= ... ) Positive lookahead group - requires the contents to match further input, without consuming it.
(?! ... ) Negative lookahead group - requires the contents not to match further input, without consuming it.

The following modifiers can be used after any expression:

* Expression can match zero or more times.
+ Expression must match one or more times.
? Expression can match zero or once.
! Require a non-empty match (this is useful with a sequence of optional matches eg. ("a"? "b"? "c"?)!).

Notes:

Each struct is a single production, with each field applied in sequence.
@<expr> is the mechanism for capturing matches into the field.
if a struct field is not keyed with "parser", the entire struct tag will be used as the grammar fragment. This allows the grammar syntax to remain clear and simple to maintain.

Capturing

Prefixing any expression in the grammar with @ will capture matching values for that expression into the corresponding field.

For example:

// The grammar definition.
type Grammar struct {
  Hello string `@Ident`
}

// The source text to parse.
source := "world"

// After parsing, the resulting AST.
result == &Grammar{
  Hello: "world",
}

For slice and string fields, each instance of @ will accumulate into the field (including repeated patterns). Accumulation into other types is not supported.

For integer and floating point types, a successful capture will be parsed with strconv.ParseInt() and strconv.ParseFloat() respectively.

A successful capture match into a bool field will set the field to true.

Tokens can also be captured directly into fields of type lexer.Token and []lexer.Token.

Custom control of how values are captured into fields can be achieved by a field type implementing the Capture interface (Capture(values []string) error).

Additionally, any field implementing the encoding.TextUnmarshaler interface will be capturable too. One caveat is that UnmarshalText() will be called once for each captured token, so eg. @(Ident Ident Ident) will be called three times.

Capturing boolean value

By default, a boolean field is used to indicate that a match occurred, which turns out to be much more useful and common in Participle than parsing true or false literals. For example, parsing a variable declaration with a trailing optional syntax:

type Var struct {
  Name string `"var" @Ident`
  Type string `":" @Ident`
  Optional bool `@"?"?`
}

In practice this gives more useful ASTs. If bool were to be parsed literally then you'd need to have some alternate type for Optional such as string or a custom type.

To capture literal boolean values such as true or false, implement the Capture interface like so:

type Boolean bool

func (b *Boolean) Capture(values []string) error {
	*b = values[0] == "true"
	return nil
}

type Value struct {
	Float  *float64 `  @Float`
	Int    *int     `| @Int`
	String *string  `| @String`
	Bool   *Boolean `| @("true" | "false")`
}

"Union" types

A very common pattern in parsers is "union" types, an example of which is shown above in the Value type. A common way of expressing this in Go is via a sealed interface, with each member of the union implementing this interface.

eg. this is how the Value type could be expressed in this way:

type Value interface { value() }

type Float struct { Value float64 `@Float` }
func (f Float) value() {}

type Int struct { Value int `@Int` }
func (f Int) value() {}

type String struct { Value string `@String` }
func (f String) value() {}

type Bool struct { Value Boolean `@("true" | "false")` }
func (f Bool) value() {}

Thanks to the efforts of Jacob Ryan McCollum, Participle now supports this pattern. Simply construct your parser with the Union[T](member...T) option, eg.

parser := participle.MustBuild[AST](participle.Union[Value](Float{}, Int{}, String{}, Bool{}))

Custom parsers may also be defined for union types with the ParseTypeWith option.

Custom parsing

There are three ways of defining custom parsers for nodes in the grammar:

Implement the Capture interface.
Implement the Parseable interface.
Use the ParseTypeWith option to specify a custom parser for union interface types.

Lexing

Participle relies on distinct lexing and parsing phases. The lexer takes raw bytes and produces tokens which the parser consumes. The parser transforms these tokens into Go values.

The default lexer, if one is not explicitly configured, is based on the Go text/scanner package and thus produces tokens for C/Go-like source code. This is surprisingly useful, but if you do require more control over lexing the included stateful participle/lexer lexer should cover most other cases. If that

Participle

Install / Use

README