AlpacaTag

AlpacaTag is an active learning-based crowd annotation framework for sequence tagging, such as named-entity recognition (NER).

Website      Documenations      Paper      Poster

The UI framework of AlpacaTag is based on the awesome work of Doccano, while the distinctive advantages of AlpacaTag are three-fold:

• Active intelligent recommendation: dynamically suggesting annotations and sampling the most informative unlabeled instances with a back-end active learned model.

• Automatic crowd consolidation: enhancing real-time inter-annotator agreement by merging inconsistent labels from multiple annotators.

• Real-time model deployment: users can deploy their models in downstream systems while new annotations are being made.

• Overall Workflow

Documentations

Installation

Annotation Tutorial

Framework Customization

Model Server APIs

Citation

@InProceedings{acl19alpaca, 
     author = {Bill Yuchen Lin and Dongho Lee and Frank F. Xu and Ouyu Lan and Xiang Ren}, 
     title = {AlpacaTag: An Active Learning-based Crowd Annotation Framework for Sequence Tagging.}, 
     booktitle = {Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics (ACL), Demo Track},
     year = {2019} 
}

Demo

Back-end Model APIs

Crowd Consolidation

<img src="figures/consolidation.png" width="500">

Performance Study

<img src="figures/userevaluation.png" width="500">