🏖️ MEAformer: Multi-modal Entity Alignment Transformer for Meta Modality Hybrid

This paper introduces MEAformer, a multi-modal entity alignment transformer approach for meta modality hybrid, which dynamically predicts the mutual correlation coefficients among modalities for more fine-grained entity-level modality fusion and alignment.

<p align="center"><i><b>👆 Click to see the Video</b></i></p>

🔔 News

2024-03 Our paper NativE: Multi-modal Knowledge Graph Completion in the Wild [Repo] is accepted by SIGIR 2024 !.
2024-02 We preprint our Survey Knowledge Graphs Meet Multi-Modal Learning: A Comprehensive Survey [Repo].
2024-02 We release the [Repo] for our paper Unleashing the Power of Imbalanced Modality Information for Multi-modal Knowledge Graph Completion , COLING 2024
2024-02 We preprint our Paper ASGEA: Exploiting Logic Rules from Align-Subgraphs for Entity Alignment [Repo].
2024-01 Our paper Revisit and Outstrip Entity Alignment: A Perspective of Generative Models [Repo] is accepted by ICLR 2024 !
2023-07 We release the [Repo] for our paper: Rethinking Uncertainly Missing and Ambiguous Visual Modality in Multi-Modal Entity Alignment ! [Slide], ISWC 2023
2023-04 We release the complete code and data for MEAformer ! [Slide] [Vedio], ACM MM 2023

🔬 Dependencies

pip install -r requirement.txt

Details

Python (>= 3.7)
PyTorch (>= 1.6.0)
numpy (>= 1.19.2)
Transformers (== 4.21.3)
easydict (>= 1.10)
unidecode (>= 1.3.6)
tensorboard (>= 2.11.0)

🚀 Train

Quick start: Using script file (run.sh)

>> cd MEAformer
>> bash run.sh

Optional: Using the bash command

>> cd MEAformer
# -----------------------
# ---- non-iterative ----
# -----------------------
# ----  w/o surface  ---- 
# FBDB15K
>> bash run_meaformer.sh 1 FBDB15K norm 0.8 0 
>> bash run_meaformer.sh 1 FBDB15K norm 0.5 0 
>> bash run_meaformer.sh 1 FBDB15K norm 0.2 0 
# FBYG15K
>> bash run_meaformer.sh 1 FBYG15K norm 0.8 0 
>> bash run_meaformer.sh 1 FBYG15K norm 0.5 0 
>> bash run_meaformer.sh 1 FBYG15K norm 0.2 0 
# DBP15K
>> bash run_meaformer.sh 1 DBP15K zh_en 0.3 0 
>> bash run_meaformer.sh 1 DBP15K ja_en 0.3 0 
>> bash run_meaformer.sh 1 DBP15K fr_en 0.3 0
# ----  w/ surface  ---- 
# DBP15K
>> bash run_meaformer.sh 1 DBP15K zh_en 0.3 1 
>> bash run_meaformer.sh 1 DBP15K ja_en 0.3 1 
>> bash run_meaformer.sh 1 DBP15K fr_en 0.3 1
# -----------------------
# ------ iterative ------
# -----------------------
# ----  w/o surface  ---- 
# FBDB15K
>> bash run_meaformer_il.sh 1 FBDB15K norm 0.8 0 
>> bash run_meaformer_il.sh 1 FBDB15K norm 0.5 0 
>> bash run_meaformer_il.sh 1 FBDB15K norm 0.2 0 
# FBYG15K
>> bash run_meaformer_il.sh 1 FBYG15K norm 0.8 0 
>> bash run_meaformer_il.sh 1 FBYG15K norm 0.5 0 
>> bash run_meaformer_il.sh 1 FBYG15K norm 0.2 0 
# DBP15K
>> bash run_meaformer_il.sh 1 DBP15K zh_en 0.3 0 
>> bash run_meaformer_il.sh 1 DBP15K ja_en 0.3 0 
>> bash run_meaformer_il.sh 1 DBP15K fr_en 0.3 0
# ----  w/ surface  ---- 
# DBP15K
>> bash run_meaformer_il.sh 1 DBP15K zh_en 0.3 1 
>> bash run_meaformer_il.sh 1 DBP15K ja_en 0.3 1 
>> bash run_meaformer_il.sh 1 DBP15K fr_en 0.3 1

❗Tips: you can open the run_meaformer.sh or run_meaformer_il.sh file for parameter or training target modification.

🎯 Results

$\bf{H@1}$ Performance with the Settings: w/o surface & Non-iterative in UMAEA. We modified part of the MSNEA to involve not using the content of attribute values but only the attribute types themselves (See issues for details): | Method | $\bf{DBP15K_{ZH-EN}}$ | $\bf{DBP15K_{JA-EN}}$ | $\bf{DBP15K_{FR-EN}}$ | |:------------------:|:----------------:|:----------------:|:----------------:| | MSNEA | .609 | .541 | .557 | | EVA | .683 | .669 | .686 | | MCLEA | .726 | .719 | .719 | | MEAformer | .772 | .764 | .771 | | UMAEA | .800 | .801 | .818 |

📚 Dataset

❗NOTE: Download from GoogleDrive (1.26G) and unzip it to make those files satisfy the following file hierarchy:

ROOT
├── data
│   └── mmkg
└── code
    └── MEAformer

Case analysis Jupyter script: GoogleDrive (180M) base on the raw images of entities (need to be unzip). I hope this gives you a good understanding of this dataset.
[ Option ] The raw Relations & Attributes appeared in DBP15k and case from MEAformer can be downloaded from Huggingface (150M).
[ Option ] The raw images of entities appeared in DBP15k can be downloaded from Baidu Cloud Drive (50GB) with the pass code mmea. All images are saved as title-image pairs in dictionaries and can be accessed with the following code :

import pickle
zh_images = pickle.load(open("eva_image_resources/dbp15k/zh_dbp15k_link_img_dict_full.pkl",'rb'))
print(en_images["http://zh.dbpedia.org/resource/香港有線電視"].size)

Code Path

<details> <summary>👈 🔎 Click</summary>

MEAformer
├── config.py
├── main.py
├── requirement.txt
├── run_meaformer.sh
├── run_meaformer_il.sh
├── run.sh
├── model
│   ├── __init__.py
│   ├── layers.py
│   ├── MEAformer_loss.py
│   ├── MEAformer.py
│   ├── MEAformer_tools.py
│   └── Tool_model.py
├── src
│   ├── __init__.py
│   ├── distributed_utils.py
│   ├── data.py
│   └── utils.py
└── torchlight
    ├── __init__.py
    ├── logger.py
    ├── metric.py
    └── utils.py

</details>

Data Path