OmniBrainBench: A Comprehensive Multimodal Benchmark for Brain Imaging Analysis Across Multi-stage Clinical Tasks

<p align="center"> <img src="./assets/OmniBrainBench_icon.png" alt="" width="130" height="120"> </p> <!-- <i>The avatar is generated by DALLE-3.</i> -->

🤖 Homepage | 🤗 Dataset | 📖 Paper

Zhihao Peng^1* Cheng Wang^1* Shengyuan Liu^1* Zhiying Liang^2* Zanting Ye³ Min Jie Ju⁴ Peter YM Woo⁵ Yixuan Yuan^1✉ ,

¹Chinese University of Hong Kong   ²Sun Yat-sen Memorial Hospital, Sun Yat-sen University   ³School of Biomedical Engineering, Southern Medical University   ⁴Zhongshan Hospital, Fudan University   ⁵Department of Neurosurgery, Prince of Wales Hospital  

^* Equal Contributions. ^✉ Corresponding Author.

This repository is the official implementation of the paper OmniBrainBench: A Comprehensive Multimodal Benchmark for Brain Imaging Analysis Across Multi-stage Clinical Tasks.

✨ News

• [02/2026] Our OmniBrainBench is accepted by CVPR2026!
• [12/2025] We have released the evaluation code and dataset for OmniBrainBench.
• [11/2025] The manuscript can be found on arXiv.

🚀Overview

we introduce OmniBrainBench, the first comprehensive multimodal VQA benchmark specifically designed to assess the multimodal comprehension capabilities of MLLMs in brain imaging analysis with closed- and open-ended evaluations. OmniBrainBench comprises 15 distinct brain imaging modalities collected from 30 verified medical sources, yielding 9,527 validated VQA pairs and 31,706 images. It simulates clinical workflows and encompasses 15 multi-stage clinical tasks rigorously validated by a professional radiologist.

📊 Code

We provide a comprehensive evaluation of the following MLLMs on OmniBrainBench:

1. This project is built upon MedEvalKit. To get started:

Visit the MedEvalKit Repo for installation instructions. or you can run the following command for a quick start:

git clone https://github.com/CUHK-AIM-Group/OmniBrainBench.git
cd OmniBrainBench
pip install -r requirements.txt

2. You can evaluate the open-source model with the following command:

python eval.sh

And modify the model name in eval.sh to evaluate different models.

For closed-source models, please use api keys to access the models. You can refer to the following example for GPT-series models.

🎈Acknowledgements

Greatly appreciate the tremendous effort for the following projects!

• MedEvalKit
• EndoBench
• NOVA

Greatly appreciate all the authors of these datasets for their contributions to the field of medical image analysis.

📜 Citation

If you find this repository helpful in your research, please consider citing the following paper:

@misc{peng2025omnibrainbench,
      title={OmniBrainBench: A Comprehensive Multimodal Benchmark for Brain Imaging Analysis Across Multi-stage Clinical Tasks}, 
      author={Zhihao Peng and Cheng Wang and Shengyuan Liu and Zhiying Liang and Yixuan Yuan},
      year={2025},
      eprint={2511.00846},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2511.00846}, 
}