Editor for Treebanks in CoNLL-U format and Front-End for dependency parser servers

This Software is a tool which facilitates the editing of syntactic relations and morphological features of files in CoNLL-U format (http://universaldependencies.org/format.html). It uses a Java-based server and a HTML/CSS/Javascript based front-end. The editor loads the CoNLL-U file and saves changes to disk (and performs a git commit if the file is under git version control).

The editor provides the following functionalities:

• editing words (forms, lemmas, upos, xpos, features, enhanced dependencies) (fast edit for UPOS and deprel)
• editing dependency relations
• indicates invalid UPOS, XPOS or dependency relations
• join/split/delete words (to correct tokenization errors)
• join/split sentences (to correct segmentation errors)
• undo/redo (partially)
• search: forms, lemmas, UPOS, XPOS, deprels, sentences IDs and comments, sequences of any of these, searching for subtrees, importing subtrees from current sentence, sd-parse support
• searching by source file line numbers
• edit non-CoNLL-U columns in a subset of CoNLL-U plus files
• create multiword tokens from existing words or add a MWT to contract two ore more existing words
• git support
• export of dependency graphs as svg or LaTeX (for the tikz-dependency package or the doc/deptree.sty class, see documentation)
• prohibits invalid (cyclic) trees
• Three edit modes: dependency trees, dependency «hedges» and a table edit mode
• mass editing: modify tokens if a (complex) condition is satisfied
• validation (using implications: if conditions1 true then conditions2 must be true)
• sentence metadata editing
• add highlight information on tokens and deprels which have yet to be validated (see highlighting)
• adding Translit= values to the MISC column (transliterating the FORM column) see section Transliteration
• finding similar or identical sentence in a list of CoNLL-U files, see section Find Similar Sentences
• configuring the UI on order to hide unneeded functionalities which otherwise clutter the UI

Current version: 2.33.0 (see change history)

ConlluEditor can also be used as front-end to display the results of dependency parsing in the same way as the editor.

• dependency tree/dependency hedge
• CoNLL-U/LaTeX/SD-Parse format

For more information see section Parser Front-End

In order to compare two files (e.g. a gold file with a predicted file) ConlluEditor provides

• a file compare mode

For more information see section File Comparison

Installation

Requirements

• Java jre >= 17.0
• Firefox (tested with version 60.9 on Windows 10, >= 63 on Linux and 69.0.1 on MacOS Sierra),
  • Chromium or Chrome (both tested with version 70 on Linux),
  • Edge (tested with version 44.17763.1.0 on Windows 10),
  • Opera (tested with version 63 on Linux),
  • Safari (tested with version 11.1 on MacOS Sierra)
• jquery 3.3.1 (https://code.jquery.com/jquery-3.3.1.min.js) and jquery-ui 1.14.1 (https://jqueryui.com)
• bootstrap 4.1.3 (https://github.com/twbs/bootstrap/releases/download/v4.1.3/bootstrap-4.1.3-dist.zip)
• popper.min.js and popper.min.js.map 1.14.6 (https://unpkg.com/popper.js/dist/umd/popper.min.js{.map}), needed by bootstrap
• sorttable.js (https://www.kryogenix.org/code/browser/sorttable/sorttable.js)
• on MacOS install in addition also: greadlink (brew install coreutils)

In order two compile the server, you also need

• Java jdk >= 17.0
• maven (tested with >= 3.3.9)

Docker

Alternatively, a recent version of Docker can be used to run the docker image (see section docker below)

Github Release

To avoid compilation, you can download the latest release, unzip the conllueditor-a.b.c.zip into a directory and start the server as described below.

License

• This software is under the 3-Clause BSD License

Compilation

• mvn install
  • if the tests fail after git pull try deleting target/test-classes and testoutput. If this does not solve the problem, run mvn install -DskipTests to avoid running the JUnit tests and drop me a note
  • individual tests can be run by mvn test -Dtest=TestConlluEditor#<testmethod>)
  • redirect stderr in order to see only maven messages (test reports are generated in target/surefire-reports/):
    • mvn install 2> stderr_output.txt
    • mvn test -Dtest=TestConlluEditor#<testmethod> 2> stderr_output.txt
• download javascript libraries (bootstrap, popper, jquery and jquery-ui), the script bin/installJQ.sh will do the job
• or change the corresponding lines in gui/index.html to load the libraries from a public server:

<script type="text/javascript" src="https://ajax.googleapis.com/ajax/libs/jquery/3.3.1/jquery.min.js">

• after having run bin/installJQ.sh the gui/lib directory must look like this

./gui/lib/bootstrap-4.1.3/js/bootstrap.min.js
./gui/lib/bootstrap-4.1.3/js/bootstrap.bundle.js.map
./gui/lib/bootstrap-4.1.3/js/bootstrap.js.map
./gui/lib/bootstrap-4.1.3/js/bootstrap.bundle.min.js.map
./gui/lib/bootstrap-4.1.3/js/bootstrap.bundle.js
./gui/lib/bootstrap-4.1.3/js/bootstrap.js
./gui/lib/bootstrap-4.1.3/js/bootstrap.bundle.min.js
./gui/lib/bootstrap-4.1.3/js/bootstrap.min.js.map
./gui/lib/bootstrap-4.1.3/css/bootstrap-grid.css.map
./gui/lib/bootstrap-4.1.3/css/bootstrap.min.css.map
./gui/lib/bootstrap-4.1.3/css/bootstrap-grid.min.css.map
./gui/lib/bootstrap-4.1.3/css/bootstrap.css.map
./gui/lib/bootstrap-4.1.3/css/bootstrap-reboot.min.css
./gui/lib/bootstrap-4.1.3/css/bootstrap-reboot.css.map
./gui/lib/bootstrap-4.1.3/css/bootstrap-grid.min.css
./gui/lib/bootstrap-4.1.3/css/bootstrap-grid.css
./gui/lib/bootstrap-4.1.3/css/bootstrap.min.css
./gui/lib/bootstrap-4.1.3/css/bootstrap-reboot.css
./gui/lib/bootstrap-4.1.3/css/bootstrap.css
./gui/lib/bootstrap-4.1.3/css/bootstrap-reboot.min.css.map
./gui/lib/jquery-3.3.1.min.js
./gui/lib/jquery-ui-1.14.1/jquery-ui.structure.min.css
./gui/lib/jquery-ui-1.14.1/external
./gui/lib/jquery-ui-1.14.1/external/jquery
./gui/lib/jquery-ui-1.14.1/external/jquery/jquery.js
./gui/lib/jquery-ui-1.14.1/jquery-ui.structure.css
./gui/lib/jquery-ui-1.14.1/package.json
./gui/lib/jquery-ui-1.14.1/jquery-ui.min.css
./gui/lib/jquery-ui-1.14.1/jquery-ui.js
./gui/lib/jquery-ui-1.14.1/jquery-ui.theme.min.css
./gui/lib/jquery-ui-1.14.1/jquery-ui.css
./gui/lib/jquery-ui-1.14.1/AUTHORS.txt
./gui/lib/jquery-ui-1.14.1/jquery-ui.theme.css
./gui/lib/jquery-ui-1.14.1/jquery-ui.min.js
./gui/lib/jquery-ui-1.14.1/index.html
./gui/lib/jquery-ui-1.14.1/images
./gui/lib/jquery-ui-1.14.1/images/ui-icons_cc0000_256x240.png
./gui/lib/jquery-ui-1.14.1/images/ui-icons_ffffff_256x240.png
./gui/lib/jquery-ui-1.14.1/images/ui-icons_777777_256x240.png
./gui/lib/jquery-ui-1.14.1/images/ui-icons_777620_256x240.png
./gui/lib/jquery-ui-1.14.1/images/ui-icons_444444_256x240.png
./gui/lib/jquery-ui-1.14.1/images/ui-icons_555555_256x240.png
./gui/lib/jquery-ui-1.14.1/LICENSE.txt
./gui/lib/popper.min.js
./gui/lib/popper.min.js.map

The server cannot be started without these libraries, an error message will show which files are missing.

Update ConlluEditor

to get bug corrections/new features, run

git pull
rm -rf target/test-classes
mvn install

Starting the server

On smaller machines, the memory management of the java VM (-Xmx... option) may have to be modified in bin/conlluedit.sh. The current value (-Xmx4g) is largely sufficient to load larger treebanks with up to 1,5M tokens.

ConlluEditor comes with a simple HTTP server:

bin/conlluedit.sh treebank.conllu 8888

Point your navigator to http://localhost:8888 .

Using docker

If you prefer a docker image, you can use the following

• get current docker image

docker pull jheinecke/conllueditor:latest

Run the image in a docker container from the directory where your .conllu-file (and other files like lists of UPOS, XPOS etc) reside and replace

• --user 1000:1000 with your uid and gid (you get the uid on Linux with the id-command)
• --env filename=<yourfile>.conllu with the filename you want to edit
• </absolute/path/to/datadir> with the directory where the .conllu-file and other files reside

docker run --rm -t -d \
  --name conllueditor -p 8888:5555 \
	--user 1000:1000 \
	-v </absolute/path/to/datadir>:/data \
	--env filename=<yourfile>.conllu \
	jheinecke/conllueditor:latest

Note that the option -d causes the docker container to run in the background and error messages will not appear on screen unless you run docker logs conllueditor. If you want to see any messages the server outputs to the terminal, omit the -d.

Other parameters (shown below in section other options) can be given with --env <optionname>=value e.g. --env UPOS=uposfile.txt, --env UPOS=cpos.ud, --env XPOS=xpos.txt, --env language=cy, --env include_unused=1, --env deprels=deprels.ud, --env features=feat_val.ud, --env shortcuts=multi-shortcuts.json, --env compare=file.conllu, or --env uiconfig=uiconfig.yml . However all files given, must reside in the </absolute/path/to/datadir> directory. Further files given in the uiconfig.yml must also reside in the </absolute/path/to/datadir> directory.

When finished, stop the docker container (it is removed automatically by the --rm option given):

docker stop conllueditor

Other options

• --uiconfig <file> yaml file to specify activated buttosn, display etc (see below in section UI configuration)
• --UPOS <file> comma separated list of files co