TumorChain

<h1 align = "center"> TumorChain: Interleaved Multimodal Chain-of-Thought Reasoning for Traceable Clinical Tumor Analysis </h1> <div align="center"> Sijing Li^1,2*, Zhongwei Qiu^2,3,1*, Jiang Liu¹, Wenqiao Zhang^1†, Tianwei Lin^1,2, <br> Yihan Xie¹, Jianxiang An¹, Boxiang Yun², Chenglin Yang¹, Jun Xiao¹, Guangyu Guo^2,1, <br> Jiawen Yao², Wei Liu², Yuan Gao², Ke Yan², Weiwei Cao², Zhilin Zheng² <br> Tony C. W. MOK², Kai Cao⁴, Yu Shi⁵, Jiuyu Zhang⁵, Jian Zhou⁶ <br> Beng Chin Ooi¹, Yingda Xia†², Ling Zhang² <br>

¹Zhejiang University ²DAMO Academy, Alibaba Group ³Hupan Lab ⁴Shanghai Institution of Pancreatic Disease ⁵Shengjing Hospital of China Medical University ⁶Sun Yat-sen University Cancer Center <br>

<a href='https://arxiv.org/abs/2603.05867'><img src='https://img.shields.io/badge/Paper-Arxiv-red'></a> <a href='https://github.com/alibaba-damo-academy/Tumorchain'><img src='https://img.shields.io/badge/DAMO-GitHub-green'></a>

</div>

🌟 Overview

Welcome to TumorChain!

Our goal is to advance clinical tumor analysis through reliable multimodal reasoning at scale. This project presents a cohesive three-part framework—Dataset, Benchmark, and Model—to enable safe, explainable, and reproducible tumor assessment in high-stakes settings.

<p align="center"> <img src="image/teaser.png" style="width:90%;vertical-align:middle;" /> </p>

:clap: Core Vision:

• Establish a closed-loop multimodal reasoning pipeline that standardizes the path from findings to impressions to pathology.
• Create high-quality benchmarks and reproducible evaluation protocols to enable cross-institution comparison and robust generalization.
• Deliver an interpretable, calibrated, and traceable multimodal framework that reduces hallucinations and supports real-world clinical decision-making.

:mailbox: Data collection and statistics

We introduce TumorCoT-1.5M — a large-scale dataset comprising 1.5 million Chain-of-Thought (CoT) labeled VQA prompts, paired with 3D CT scans, featuring stepwise reasoning and cross-modal alignments along the findings–impression–pathology trajectory.

<img src="image/agent.jpg" style="width:70%;vertical-align:middle;" /><img src="image/data.jpg" style="width:30%;vertical-align:middle;" />

:ferris_wheel: Model Architecture

TumorChain is a multi-modal, iterative interleaved reasoning framework for 3D CT tumor analysis that fuses a 3D vision encoder, organ segmentation model, auxiliary classification model, an MLP projector, and a large language model (LLM) to perform stepwise, evidence-grounded reasoning from findings to impressions to pathology, with traceable evidence and calibrated uncertainty.

<p align="center"> <img src="image/model.png" style="width:80%;vertical-align:middle;" /> </p>

🛠️ Getting Started

😊 We will release our task definitions, benchmarks, and evaluation protocols in the near future to advance safe, explainable, and reproducible multimodal reasoning for high-stakes tumor analysis. 🚀