MM4TSA

<div align="center"> <h1>Awesome Multi-Modalities For Time Series Analysis Papers (MM4TSA)</h1> </div> <p align="center"> <a href="https://awesome.re"><img src="https://awesome.re/badge.svg" alt="Awesome"></a> <img src="https://img.shields.io/badge/PRs-Welcome-green" alt="PRs Welcome"> <a href="https://arxiv.org/abs/2303.11835"> <img src="https://img.shields.io/badge/arXiv-2303.11835-b31b1b.svg" alt="arXiv"> </a> </p> <p align="center"> <strong><a href="https://mp.weixin.qq.com/s/DUgaWPuf0g5EdUlGulADWA">中文解读1</a></strong> <strong><a href="https://mp.weixin.qq.com/s/ytGc3c_ZaPpoz2pNsydH3w">中文解读2</a></strong> </p>

---

News

• 🔥 NeurIPS 2025 Paper Updated.
• 🔥 KDD 2025 Paper Updated.
• 🔥 ICML 2025 Paper Updated.
• 🔥 IJCAI 2025 Paper Updated.

---

Time series analysis (TSA) is a longstanding research topic in the data mining community and has wide real-world significance. Compared to "richer" modalities such as language and vision, which have recently experienced explosive development and are densely connected, the time-series modality remains relatively underexplored and isolated. We notice that many recent TSA works have formed a new research field, i.e., Multiple Modalities for TSA (MM4TSA). In general, these MM4TSA works follow a common motivation: how TSA can benefit from multiple modalities. This survey is the first to offer a comprehensive review and a detailed outlook for this emerging field. Specifically, we systematically discuss three benefits: (1) reusing foundation models of other modalities for efficient TSA, (2) multimodal extension for enhanced TSA, and (3) cross-modality interaction for advanced TSA. We further group the works by the introduced modality type, including text, images, audio, tables, and others, within each perspective. Finally, we identify the gaps with future opportunities, including the reused modalities selections, heterogeneous modality combinations, and unseen tasks generalizations, corresponding to the three benefits. We release this up-to-date GitHub repository that includes key papers and resources. More details please check our <a href="https://arxiv.org/abs/2503.11835"><strong>survey</strong></a>.

<div align="center"> <img src="https://github.com/AdityaLab/MM4TSA/blob/main/Survey_Logo_1_1.jpg" width="447"> <img src="https://github.com/AdityaLab/MM4TSA/blob/main/Survey_Logo_2.jpg" width="500"> </div>

Contributing

🚀 We will continue to update this repo. If you find it helpful, please Star it or Cite Our Survey.

🤝 Contributions are welcome! Please feel free to submit a Pull Request.

Citation

🤗 If you find this survey useful, please consider citing our paper. 🤗

@misc{liu2025timeseriesanalysisbenefit,
title={How Can Time Series Analysis Benefit From Multiple Modalities? A Survey and Outlook},
author={Haoxin Liu and Harshavardhan Kamarthi and Zhiyuan Zhao and Shangqing Xu and Shiyu Wang and Qingsong Wen and Tom Hartvigsen and Fei Wang and B. Aditya Prakash},
year={2025},
eprint={2503.11835},
archivePrefix={arXiv},
primaryClass={cs.LG},
url={[https://arxiv.org/abs/2503.11835}](https://arxiv.org/abs/2503.11835})
}

---

Table of Contents

• 1. Time2X and X2Time

  • 1.1 Text to Time Series

    • 1.1.1 Generation
    • 1.1.2 Retrieval

  • 1.2 Time Series to Text

    • 1.2.1 Explanation
    • 1.2.2 Captioning

  • 1.3 Text to Time + Time to Text

  • 1.4 Other Cross-Modality Works

  • 1.5 Domain Specific Applications

    • 1.5.1 Spatial-Temporal Data
    • 1.5.2 Medical Time Series
    • 1.5.3 Financial Time Series

  • 1.6 Gaps and Outlooks

• 2. Time+X

  • 2.1 Time Series + Text

  • 2.2 Time Series + Other Modalities

  • 2.3 Domain Specific Applications

    • 2.3.1 Spatial-Temporal Data
    • 2.3.2 Medical Time Series
    • 2.3.3 Financial Time Series

  • 2.4 Gaps and Outlooks

    • 2.4.1 Heterogeneous Modality Combinations
    • 2.4.2 Robust & Efficient Multimodal TS (Outlook)

  • 2.5 Datasets & Benchmarks (Multimodal)

• 3. TimeAsX

  • 3.1 Time Series as Text

  • 3.2 Time Series as Image

  • 3.3 Time Series as Other Modalities

    • 3.3.1 Tabular Data
    • 3.3.2 Audio Data
    • 3.3.3 Other Modalities

  • 3.4 Domain Specific Applications

    • 3.4.1 Spatial-Temporal Data
    • 3.4.2 Medical Time Series
    • 3.4.3 Financial Time Series

  • 3.5 Gaps and Outlooks

    • 3.5.1 Reuse Which Modality

• 4. Datasets for Multi-Modal Time Series Analysis

  • 4.1 General Datasets
  • 4.2 Financial Datasets
  • 4.3 Medical Datasets
  • 4.4 Spatial-Temporal Datasets

1. Time2X and X2Time

1.1 Text to Time Series

1.1.1 Generation

<a id="111-generation"></a>

| Title | Venue | | -------------------------------------------------------------------------------------------------------------------------- | --------- | | VerbalTS: Generating Time Series from Texts | ICML 2025 | | BRIDGE: Bootstrapping Text to Control Time-Series Generation via Multi-Agent Iterative Optimization and Diffusion Modeling | ICML 2025 | | T2S: High-resolution Time Series Generation with Text-to-Series Diffusion Models | IJCAI 2025 | | MedDiTPro: A Prompt-Guided Diffusion Transformer for Multimodal Longitudinal Medical Data Synthesis | KDD 2025 | | TarDiff: Target-Oriented Diffusion Guidance for Synthetic Electronic Health Record Time Series Generation | KDD 2025 | | Language Models Still Struggle to Zero-shot Reason about Time Series | EMNLP 2024 Findings | | DiffuSETS: 12-lead ECG Generation Conditioned on Clinical Text Reports and Patient-Specific Information | arXiv 25.01 | | ChatTS: Aligning Time Series with LLMs via Synthetic Data for Enhanced Understanding and Reasoning | arXiv 24.12 | | Forging Time Series with Language: A Large Language Model Approach to Synthetic Data Generation | NeurIPS 2025 |

1.1.2 Retrieval

<a id="112-retrieval"></a>

| Title | Venue | | ------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------- | | Evaluating Large Language Models on Time Series Feature Understanding: A Comprehensive Taxonomy and Benchmark | EMNLP 2024 | | TimeSeriesExam: A Time Series Understanding Exam | NeurIPS 2024 Workshop on Time Series in the Age of Large Models | | CLaSP: Learning Concepts for Time-Series Signals from Natural Language Supervision | arXiv 24.11 | | TRACE: Grounding Time Series in Context for Multimodal Embedding and Retrieval | NeurIPS 2025 |

1.2 Time Series to Text

1.2.1 Explanation

<a id="121-explanation"></a>

| Title | Venue | | ------------------------------------------------------------------------------------------ | ----------- | | Inferring Event Descriptions from Time Series with Language Models | arXiv 25.03 | | Explainable Multi-modal Time Series Prediction with LLM-in-the-Loop | NeurIPS 2025 | | Xforecast: Evaluating natural language explanations for time series forecasting | arXiv 24.10 | | Large language models can deliver accurate and interpretable time series anomaly detection | arXiv 24.05 |

1.2.2 Captioning

<a id="122-captioning"></a>

| Title | Venue | | ---------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------ | | Repr2Seq: A Data-to-Text Generation Model for Time Series | IJCNN 2023 | | Insight miner: A time series analysis dataset for cross-domain alignment with natural language | NeurIPS 2023 AI for Science Workshop |