Zshot

<div align="center"> <img height="170x" src="https://ibm.github.io/zshot/img/graph.png" /> <h1>Zshot</h1> <p> <strong>Zero and Few shot named entity & relationships recognition</strong> </p> <p>

<a href="https://ibm.github.io/zshot/"><img alt="Tutorials" src="https://img.shields.io/badge/docs-tutorials-green" /></a> <a href="https://pypi.org/project/zshot/"><img src="https://img.shields.io/pypi/v/zshot" /></a> <a href="https://pypi.org/project/zshot/"><img src="https://img.shields.io/pypi/dm/zshot" /></a> <a href="https://github.com/IBM/zshot/actions/workflows/python-tests.yml"> <img alt="Build" src="https://github.com/IBM/zshot/actions/workflows/python-tests.yml/badge.svg" /></a> <a href="https://app.codecov.io/github/ibm/zshot"> <img alt="Build" src="https://codecov.io/github/ibm/zshot/branch/main/graph/badge.svg" /></a>

</p> </div>

Documentation: <a href="https://ibm.github.io/zshot/" target="_blank">https://ibm.github.io/zshot</a>

Source Code: <a href="https://github.com/IBM/zshot" target="_blank">https://github.com/IBM/zshot</a>

Paper: <a href="https://aclanthology.org/2023.acl-demo.34/" target="_blank">https://aclanthology.org/2023.acl-demo.34/</a>

Zshot is a highly customisable framework for performing Zero and Few shot named entity recognition.

Can be used to perform:

• Mentions extraction: Identify globally relevant mentions or mentions relevant for a given domain
• Wikification: The task of linking textual mentions to entities in Wikipedia
• Zero and Few Shot named entity recognition: using language description perform NER to generalize to unseen domains
• Zero and Few Shot named relationship recognition
• Visualization: Zero-shot NER and RE extraction

Requirements

• Python 3.6+

• <a href="https://spacy.io/" target="_blank"><code>spacy</code></a> - Zshot rely on <a href="https://spacy.io/" class="external-link" target="_blank">Spacy</a> for pipelining and visualization

• <a href="https://pytorch.org/get-started" target="_blank"><code>torch</code></a> - PyTorch is required to run pytorch models.

• <a href="https://huggingface.co/docs/transformers/index" target="_blank"><code>transformers</code></a> - Required for pre-trained language models.

• <a href="https://huggingface.co/docs/evaluate/index" target="_blank"><code>evaluate</code></a> - Required for evaluation.

• <a href="https://huggingface.co/docs/datasets/index" target="_blank"><code>datasets</code></a> - Required to evaluate over datasets (e.g.: OntoNotes).

Optional Dependencies

• <a href="https://github.com/flairNLP/flair" target="_blank"><code>flair</code></a> - Required if you want to use Flair mentions extractor and for TARS linker and TARS Mentions Extractor.
• <a href="https://github.com/facebookresearch/BLINK" target="_blank"><code>blink</code></a> - Required if you want to use Blink for linking to Wikipedia pages.
• <a href="https://github.com/urchade/GLiNER" target="_blank"><code>gliner</code></a> - Required if you want to use GLiNER Linker or GLiNER Mentions Extractor.
• <a href="https://github.com/SapienzaNLP/relik" target="_blank"><code>relik</code></a> - Required if you want to use Relik Linker.

Installation

<div class="termy">

$ pip install zshot

---> 100%

</div>

Examples

| Example | Notebook | |:------------------------------:|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | Installation and Visualization | | | Knowledge Extractor | | | Wikification | | | Custom Components | | | Evaluation | |

Zshot Approach

ZShot contains two different components, the mentions extractor and the linker.

Mentions Extractor

The mentions extractor will detect the possible entities (a.k.a. mentions), that will be then linked to a data source (e.g.: Wikidata) by the linker.

Currently, there are 7 different mentions extractors supported, SMXM, TARS, GLiNER, 2 based on SpaCy, and 2 that are based on Flair. The two different versions for SpaCy and Flair are similar, one is based on Named Entity Recognition and Classification (NERC) and the other one is based on the linguistics (i.e.: using Part Of the Speech tagging (PoS) and Dependency Parsing(DP)).

The NERC approach will use NERC models to detect all the entities that have to be linked. This approach depends on the model that is being used, and the entities the model has been trained on, so depending on the use case and the target entities it may be not the best approach, as the entities may be not recognized by the NERC model and thus won't be linked.

The linguistic approach relies on the idea that mentions will usually be a syntagma or a noun. Therefore, this approach detects nouns that are included in a syntagma and that act like objects, subjects, etc. This approach do not depend on the model (although the performance does), but a noun in a text should be always a noun, it doesn't depend on the dataset the model has been trained on.

Linker

The linker will link the detected entities to a existing set of labels. Some of the linkers, however, are end-to-end, i.e. they don't need the mentions extractor, as they detect and link the entities at the same time.

Again, there are 6 linkers available currently, 4 of them are end-to-end and 2 are not.

| Linker Name | end-to-end | Source Code | Paper | |:-----------:|:----------:|----------------------------------------------------------|--------------------------------------------------------------------| | Blink | X | Source Code | Paper | | GENRE | X | Source Code | Paper | | SMXM | ✓ | Source Code | Paper | | TARS | ✓ | Source Code | Paper | | GLINER | ✓ | Source Code | Paper | | RELIK | ✓ | Source Code | Paper |

Relations Extractor

The relations extractor will extract relations among different entities previously extracted by a linker..

Currently, the is only one Relation Extractor available:

• ZS-Bert
  • Paper
  • Source Code

Knowledge Extractor

The knowledge extractor will perform at the same time the extraction and classification of named entities and the extraction of relations among them. The pipeline with this component doesn't need any mentions extractor, linker or relation extractor to work.

Currently, there are only two Knowledge Extractor available:

• KnowGL

  • Rossiello et al. (AAAI 2023)
  • Mihindukulasooriya et al. (ISWC 2022)

• Relik

  • Paper
  • Source Code

How to use it

• Install requirements: pip install -r requirements.txt
• Install a spacy pipeline to use it for mentions extraction: python -m spacy download en_core_web_sm
• Create a file main.py with the pipeline configuration and entities definition (Wikipedia abstract are usually a good starting point for descriptions):

import spacy

from zshot import PipelineConfig, displacy
from zshot.linker import LinkerRegen
from zshot.mentions_extractor import MentionsExtractorSpacy
from zshot.utils.data_models import Entity

nlp = spacy.load("en_core_web_sm")
nlp_config = PipelineConfig(
    mentions_extractor=MentionsExtractorSpacy(),
    linker=LinkerRegen(),
    entities=[
        Entity(name="Paris",
               description="Paris is located in northern central France, in a north-bending arc of the river Seine"),
        Entity(name="IBM",
               description="International Business Machines Corporation (IBM) is an American multinational technology corporation headquartered in Armonk, New York"),
        Entity(name="New York", description="New York is a city in U.S. stat