inmemantlr

inmemantlr provides the functionality of ANTLR v4 through a simple API. Usually ANTLR requires the user to perform the grammar generation and compilation steps. Instead, inmemantlr does all of these steps automatically and in-memory while keeping all of the original ANTLR objects accessible through its GenericParser class which is serializable, and hence, can be reused at a later point in time or across different applications. inmemantlr can be used via an easy-to-use Java API or as command-line tool.

Moreover, you can easily generate a parse tree from a parsed file and convert it into various formats such as .dot, .xml or .json. A parse tree can be processed/translated by means of inmemantlr's ParseTreeProcessor class.

All of the above-mentioned inmemantlr features are illustrated by examples. inmemantlr is ready to use for all of the grammars-v4 grammars (for detailed examples please have a look at grammars-v4).

Status

Authors and major contributors

• Julian Thome, julian.thome@gmail.com, main author
• Alan Stewart (code style improvements, performance improvements, Tool customization)
• Radoslaw Cymer (bugfixes, code style improvements, typos in Javadoc)
• Nikolas Havrikov (code style, bugfixes, performance improvements)
• Calogero G. Zarba (feature improvements)
• dafei1288 (Maintainability)

TOC

Integration

API Usage Scenarios

• Get started
• Simple parsing
• Parse tree generation
• Parse tree serialization
• Parse tree pruning
• Parse tree processing
• Sequential parsing
• Non-combined grammars
• Lexer-only grammars
• Accessing ANTLR objects
• Parser serialization
• grammars-v4

Licence

Integration

Maven

inmemantlr is available on maven central and can be integrated by using the following dependency in the pom.xml file. Note, that the maven releases do not necessarily contain the newest changes that are available in the repository. The maven releases are kept in sync with the tagged releases. The API documentation for every release is avalable here. However, the content of this documentation, in particular the code examples and usage scenarios, is always aligned with the master branch of this repository. Hence, it might be that the latest inmemantlr features are not yet available through the maven package.

<dependency>
    <groupId>com.github.julianthome</groupId>
    <artifactId>inmemantlr-api</artifactId>
    <version>1.9.2</version>
</dependency>

API Usage Scenarios

The following code snippets shows an example how to use the API of inmemantlr; descriptions are provided as source code comments. For the sake of simplicity, exception handling is omitted for all of the following examples.

Get started

The code sample below shows how you could get started with inmemantlr. The class GenericParserToGo (available as of release 1.6) provides a very simple API that should be sufficient for most of the use cases: you only have to provide the ANTLR grammar file in conjunction with the file/string to parse, and a call to parse() (with the string and starting-rule as parameters) will return the corresponding parse tree.

File gfile = new File("Java.g4");
File cfile = new File("HelloWorld.java");
ParseTree pt = new GenericParserToGo(gfile).parse(cfile, "compilationUnit");

Simple parsing

// 1. load grammar
File f = new File("Java.g4");
GenericParser gp = new GenericParser(f);
// 2. load file content into string
String s = FileUtils.loadFileContent("HelloWorld.java");
// 3. set listener for checking parse tree elements. Here you could use any ParseTreeListener implementation. The default listener is used per default
gp.setListener(new DefaultListener());
// 4. compile Lexer and parser in-memory
gp.compile();
// 5. parse the string that represents the content of HelloWorld.java
ParserRuleContext ctx = gp.parse(s, "compilationUnit", GenericParser.CaseSensitiveType.NONE);

Parse tree generation

While providing access to the original ParseTree data-structure of ANTLR which can be obtained through the ParserRuleContext object, inmemantlr also provides it's own ParseTree data-structure and some ready-to-use utility classes which help with everyday tasks such as pruning, data-conversion, tree-traversal and translation.

If you would like to obtain the ParseTree from a parsed file, the following snippet could be of use:

File f = new File("Java.g4");
GenericParser gp = new GenericParser(f);
String s = FileUtils.loadFileContent("HelloWorld.java");

// this listener will create a parse tree from the java file
DefaultTreeListener dlist = new DefaultTreeListener();

gp.setListener(dlist);
gp.compile();

ParserRuleContext ctx = gp.parse(s, "compilationUnit", GenericParser.CaseSensitiveType.NONE);
// get access to the parse tree of inmemantlr
ParseTree pt = dlist.getParseTree();

Parse tree serialization

As depicted below, our ParseTree implementation can be serialized to various output formats such as .xml, .json or .dot.

String xml = parseTree.toXml();
String json = parseTree.toJson();
String dot = parseTree.toDot();

Serialization may be useful in case you would like to use parsing results across different applications or in case you would like get a quick and simple visualization of your parse tree by means of graphviz as in the example below.

<img src="https://github.com/julianthome/inmemantlr/blob/master/images/pt.png" alt="Example Parse tree" width="400px" align="second">

Parse tree pruning

In case you are just interested in particular nodes of the parse tree, it is possible to pass a lambda expression as parameter to the DefaultTreeListener as illustrated in the example below. Only those nodes remain in the parse tree for which the lambda expression returns true.

private Set<String> filter  = new HashSet<>(Arrays.asList(new String []{
        "alternation", "expr", "literal",
}));
// ...
dlist = new DefaultTreeListener(s -> filter.contains(s));
// ...

Parse tree processing

With inmemantlr, you can easily process or translate a given Parse tree by means of an ParseTreeProcessor. Note, that ANTLR can automatically generate visitors/listeners from a given grammar which you can obtain through the GenericParser member function getAllCompiledObjects. However, in case you would like to develop a simple application inmemantlr ParseTreeProcessor might be sufficient for your use case.

The following example illustrates how to process a simple parse tree that represents a mathematical expression. Given the grammar definition below, parsing the string '3+100' would yield this parse tree:

<img src="https://github.com/julianthome/inmemantlr/blob/master/images/simpleop.png" alt="ParseTree derived from simple expression '3+100'" width="200px" align="second">

grammar Ops;

Plus: '+';
Minus: '-';
Number: '-'?([0-9]|[1-9][0-9]+);

s: (expression)* EOF;
plus: Plus;
minus: Minus;
operation: plus | minus;
expression: operand operation operand;
operand: Number;

WS  :  [ \t\r\n]+ -> skip;

The following code example illustrates how to compute the result of a mathematical expression based on the above-mentioned grammar.

// ...
gp.compile();
// this example shows you how one could use inmemantlr for sequential parsing
ParseTree pt;
gp.parse("3+100");
pt = t.getParseTree();
// Process the Parse tree bottom-up starting from the leafs up to the root node
ParseTreeProcessor<String, String> processor = new ParseTreeProcessor<String, String>(pt) {
  @Override
  public String getResult() {
    // when all nodes have been processed, the result is available in the smap
    // value of the root node which is returned here
    return smap.get(pt.getRoot());
  }
  @Override
  protected void initialize() {
    // initialize smap - a data structure that keeps track of the intermediate
    // values for every node
    pt.getNodes().forEach(n -> smap.put(n, n.getLabel()));
  }
  // This operation is executed for each and every node in left to right and
  // bottom up order. Non-leaf nodes are processed only if all of their siblings
  // have been already processed
  @Override
  protected void process(ParseTreeNode n) {
    if(n.getRule().equals("expression")){
      int n0 = Integer.parseInt(smap.get(n.getChild(0)));
      int n1 = Integer.parseInt(smap.get(n.getChild(2)));
      int result = 0;
      switch(smap.get(n.getChild(1))) {
        case "+":
          result = n0 + n1;
        break;
        case "-":
          result = n0 - n1;
        break;
      }
      // store computation result of addition subtraction for current node
      smap.put(n, String.valueOf(result));
    } else {
      // when node is no expression NT, propate child no