Wiki Entity Summarization Benchmark (WikES)

This repository leverages the wiki-entity-summarization-preprocessor project to construct an Entity Summarization Graph based on a given set of nodes. The project tries to maintain the structure of the Wikidata knowledge graph by performing random walk sampling with a depth of K, starting from seed nodes after all the summary edges have been added to the result. It then checks if the expanded graph is a single weakly connected component. If not, it finds B paths to connect the components. The final result is a heterogeneous graph consisting of the seed nodes, their summary edges, (1..K)-hop neighbors of the seed nodes and their edges, and any intermediary nodes added to ensure graph connectivity. Each node and edge in the graph is enriched with metadata obtained from Wikidata and Wikipedia and predicate information, providing additional context and details about the entities and their relationships. <br/> <br/>

Loading the Datasets

Load Using wikes-toolkit

To load the dataset, we have introduced a toolkit that can be used to download, load, work, and evaluate 48 Wiki-Entity-Summarization datasets. The toolkit is available as a Python package and can be installed using pip:

pip install wikes-toolkit

A simple example of how to use the toolkit is as follows:

from wikes_toolkit import WikESToolkit, V1, WikESGraph

toolkit = WikESToolkit(save_path="./data")  # save_path is optional
G = toolkit.load_graph(
    WikESGraph,
    V1.WikiLitArt.SMALL,
    entity_formatter=lambda e: f"Entity({e.wikidata_label})",
    predicate_formatter=lambda p: f"Predicate({p.label})",
    triple_formatter=lambda
        t: f"({t.subject_entity.wikidata_label})-[{t.predicate.label}]-> ({t.object_entity.wikidata_label})"
)

root_nodes = G.root_entities()
nodes = G.entities()

Please refer to the Wiki-Entity-Summarization-Toolkit repository for more information.

Using mlcroissant

To load WikES datasets, you can use mlcorissant as well. You can find the metadata JSON files in the dataset details tabel. </br>

Here is an example of loading our dataset using mlcorissant:

from mlcroissant import Dataset


def print_first_item(record_name):
    for record in dataset.records(record_set=record_name):
        for key, val in record.items():
            if isinstance(val, bytes):
                val = str(val, "utf-8")
            print(f"{key}=[{val}]({type(val)})", end=", ")
        break
    print()


dataset = Dataset(
    jsonld="https://github.com/msorkhpar/wiki-entity-summarization/releases/download/1.0.5/WikiProFem-s.json")

print(dataset.metadata.record_sets)

print_first_item("entities")
print_first_item("root-entities")
print_first_item("predicates")
print_first_item("triples")
print_first_item("ground-truths")
""" The output of the above code:
wikes-dataset
[RecordSet(uuid="entities"), RecordSet(uuid="root-entities"), RecordSet(uuid="predicates"), RecordSet(uuid="triples"), RecordSet(uuid="ground-truths")]
id=[0](<class 'int'>), entity=[Q6387338](<class 'str'>), wikidata_label=[Ken Blackwell](<class 'str'>), wikidata_description=[American politician and activist](<class 'str'>), wikipedia_id=[769596](<class 'int'>), wikipedia_title=[Ken_Blackwell](<class 'str'>), 
entity=[9](<class 'int'>), category=[singer](<class 'str'>), 
id=[0](<class 'int'>), predicate=[P1344](<class 'str'>), predicate_label=[participant in](<class 'str'>), predicate_desc=[event in which a person or organization was/is a participant; inverse of P710 or P1923](<class 'str'>), 
subject=[1](<class 'int'>), predicate=[0](<class 'int'>), object=[778](<class 'int'>), 
root_entity=[9](<class 'int'>), subject=[9](<class 'int'>), predicate=[8](<class 'int'>), object=[31068](<class 'int'>), 
"""

Loading the Pre-processed Databases

As described in wiki-entity-summarization-preprocessor, we have imported en-wikidata items as a graph with their summaries into a Neo4j database using Wikipedia and Wikidata XML dump files. Additionally, all the other related metadata was imported into a Postgres database.

If you want to create your own dataset but do not want to run the pre-processor again, you can download and load the exported files from these two databases. Please refer to the release notes of the current version 1.0.0 ( enwiki-2023-05-1 and wikidata-wiki-2023-05-1).

• PostgreSQL-1.0.0 (wiki 2023-05-01)
• Neo4j-1.0.0 (wiki 2023-05-01)

Process Overview

1. Building the Summary Graph

• Create a summary graph where each seed node is expanded with its summary edges.

2. Expanding the Summary Graph

• Perform random walks starting from the seed nodes to mimic the structure of the Wikidata graph.
• Scale the number of walks based on the degree of the seed nodes.
• Add new edges to the graph from the random walk results.

3. Connecting Components

• Check if the expanded graph forms a single weakly connected component.
• If not, iteratively connect smaller components using the shortest paths until a single component is achieved.

4. Adding Metadata

• Enhance the final graph with additional metadata for each node and edge.
• Include labels, descriptions, and other relevant information from Wikidata, Wikipedia, and predicate information.

Pre-generated Datasets

We have generated datasets using A Brief History of Human Time project. These datasets contain different sets of seed nodes, categorized by various human arts and professions.

| dataset (variant, size, None/train/val/test) | #roots | #smmaries | #nodes | #edges | #labels | roots category distribution | Running Time(sec) | |---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|--------|-----------|--------|--------|---------|--------------------------------------------------------------------------------------------------------------------------------------------------------------|-------------------| | WikiLitArt-s </br>csv, graphml, croissant.json | 494 | 10416 | 85346 | 136950 | 547 | actor=150<br/> composer=35<br/> film=41<br/> novelist=24<br/> painter=59<br/> poet=39<br/> screenwriter=17<br/> singer=72<br/> writer=57 | 91.934 | | WikiLitArt-s-train </br>csv, graphml, croissant.json | 346 | 7234 | 61885 | 96497 | 508 | actor=105<br/> composer=24<br/> film=29<br/> novelist=17<br/> painter=42<br/> poet=27<br/> screenwriter=12<br/> singer=50<br/> writer=40 | 66.023 | | WikiLitArt-s-val </br>csv, graphml, croissant.json | 74 | 1572 | 14763 | 20795 | 340 | actor=23<br/> composer=5<br/> film=6<br/> novelist=4<br/> painter=9<br/> poet=6<br/> screenwriter=2<br/> singer=11<br/> writer=8 | 14.364 | | WikiLitArt-s-test </br>csv, graphml, croissant.json | 74 | 1626 | 15861 | 22029 | 350 | actor=22<br/> composer=6<br/> film=6<br/> novelist=3<br/> painter=8<br/> poet=6<br/> scree