rnnmorph

Important: please see https://github.com/natasha/slovnet#morphology-1

Morphological analyzer (POS tagger) for Russian and English languages based on neural networks and dictionary-lookup systems (pymorphy2, nltk).

Contacts

Telegram: @YallenGusev

Russian language, MorphoRuEval-2017 test dataset, accuracy

| Domain | Full tag | PoS tag | F.t. + lemma | Sentence f.t.| Sentence f.t.l. | |:-------------|:---------|:--------|:-------------|:-------------|:----------------| | Lenta (news) | 96.31% | 98.01% | 92.96% | 77.93% | 52.79% | | VK (social) | 95.20% | 98.04% | 92.06% | 74.30% | 60.56% | | JZ (lit.) | 95.87% | 98.71% | 90.45% | 73.10% | 43.15% | | All | 95.81%| 98.26% | N/A | 74.92% | N/A |

English language, UD EWT test, accuracy

| Dataset | Full tag | PoS tag | F.t. + lemma | Sentence f.t.| Sentence f.t.l. | |:-------------|:---------|:--------|:-------------|:-------------|:----------------| | UD EWT test | 91.57% | 94.10% | 87.02% | 63.17% | 50.99% |

Speed and memory consumption

Speed: from 200 to 600 words per second using CPU.

Memory consumption: about 500-600 MB for single-sentence predictions

Install

pip install rnnmorph

Usage

Example:

from rnnmorph.predictor import RNNMorphPredictor
predictor = RNNMorphPredictor(language="ru")
forms = predictor.predict(["мама", "мыла", "раму"])
print(forms[0].pos)
>>> NOUN
print(forms[0].tag)
>>> Case=Nom|Gender=Fem|Number=Sing
print(forms[0].normal_form)
>>> мама
print(forms[0].vector)
>>> [0 0 0 0 0 1 0 0 0 1 1 0 0 0 0 0 1 0 1 0 1 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 1 0 0 1 0 1 0 0 0 1 0 0 1]

Training

Simple model training:

Acknowledgements

Anastasyev D. G., Gusev I. O., Indenbom E. M., 2018, Improving Part-of-speech Tagging Via Multi-task Learning and Character-level Word Representations
Anastasyev D. G., Andrianov A. I., Indenbom E. M., 2017, Part-of-speech Tagging with Rich Language Description, презентация
Дорожка по морфологическому анализу "Диалога-2017"
Материалы дорожки
Morphine by kmike, CRF classifier for MorphoRuEval-2017 by kmike
Universal Dependencies
Tobias Horsmann and Torsten Zesch, 2017, Do LSTMs really work so well for PoS tagging? – A replication study
Barbara Plank, Anders Søgaard, Yoav Goldberg, 2016, Multilingual Part-of-Speech Tagging with Bidirectional Long Short-Term Memory Models and Auxiliary Loss

Rnnmorph

Install / Use

README