Transformers in Time Series

<!--  -->

A professionally curated list of awesome resources (paper, code, data, etc.) on Transformers in Time Series, which is first work to comprehensively and systematically summarize the recent advances of Transformers for modeling time series data to the best of our knowledge.

We will continue to update this list with newest resources. If you found any missed resources (paper/code) or errors, please feel free to open an issue or make a pull request.

For general AI for Time Series (AI4TS) Papers, Tutorials, and Surveys at the Top AI Conferences and Journals, please check This Repo.

For general Recent AI Advances: Tutorials and Surveys in various areas (DL, ML, DM, CV, NLP, Speech, etc.) at the Top AI Conferences and Journals, please check This Repo.

Survey paper

Transformers in Time Series: A Survey (IJCAI'23 Survey Track)

Qingsong Wen, Tian Zhou, Chaoli Zhang, Weiqi Chen, Ziqing Ma, Junchi Yan and Liang Sun.

If you find this repository helpful for your work, please kindly cite our survey paper.

@inproceedings{wen2023transformers,
  title={Transformers in time series: A survey},
  author={Wen, Qingsong and Zhou, Tian and Zhang, Chaoli and Chen, Weiqi and Ma, Ziqing and Yan, Junchi and Sun, Liang},
  booktitle={International Joint Conference on Artificial Intelligence(IJCAI)},
  year={2023}
}

Taxonomy of Transformers for time series modeling

<!--  -->

<img src="TS_Xformer_V2.jpg" width=700 align=middle> <br />

Application Domains of Time Series Transformers

[official code]

Transformers in Forecasting

Time Series Forecasting

• CARD: Channel Aligned Robust Blend Transformer for Time Series Forecasting, in ICLR 2024. [paper] [official code]
• Pathformer: Multi-scale Transformers with Adaptive Pathways for Time Series Forecasting, in ICLR 2024. [paper] [official code]
• GAFormer: Enhancing Timeseries Transformers Through Group-Aware Embeddings, in ICLR 2024. [paper]
• Transformer-Modulated Diffusion Models for Probabilistic Multivariate Time Series Forecasting, in ICLR 2024. [paper]
• iTransformer: Inverted Transformers Are Effective for Time Series Forecasting, in ICLR 2024. [paper]
• Considering Nonstationary within Multivariate Time Series with Variational Hierarchical Transformer for Forecasting, in AAAI 2024. [paper]
• Latent Diffusion Transformer for Probabilistic Time Series Forecasting, in AAAI 2024. [paper]
• BasisFormer: Attention-based Time Series Forecasting with Learnable and Interpretable Basis, in NeurIPS 2023. [paper]
• ContiFormer: Continuous-Time Transformer for Irregular Time Series Modeling, in NeurIPS 2023. [paper]
• A Time Series is Worth 64 Words: Long-term Forecasting with Transformers, in ICLR 2023. [paper] [code]
• Crossformer: Transformer Utilizing Cross-Dimension Dependency for Multivariate Time Series Forecasting, in ICLR 2023. [paper]
• Scaleformer: Iterative Multi-scale Refining Transformers for Time Series Forecasting, in ICLR 2023. [paper]
• Non-stationary Transformers: Rethinking the Stationarity in Time Series Forecasting, in NeurIPS 2022. [paper]
• Learning to Rotate: Quaternion Transformer for Complicated Periodical Time Series Forecasting”, in KDD 2022. [paper]
• FEDformer: Frequency Enhanced Decomposed Transformer for Long-term Series Forecasting, in ICML 2022. [paper] [official code]
• TACTiS: Transformer-Attentional Copulas for Time Series, in ICML 2022. [paper]
• Pyraformer: Low-Complexity Pyramidal Attention for Long-Range Time Series Modeling and Forecasting, in ICLR 2022. [paper] [official code]
• Autoformer: Decomposition transformers with auto-correlation for long-term series forecasting, in NeurIPS 2021. [paper] [official code]
• Informer: Beyond efficient transformer for long sequence time-series forecasting, in AAAI 2021. [paper] [official code] [dataset]
• Temporal fusion transformers for interpretable multi-horizon time series forecasting, in International Journal of Forecasting 2021. [paper] [code]
• Probabilistic Transformer For Time Series Analysis, in NeurIPS 2021. [paper]
• Deep Transformer Models for Time Series Forecasting: The Influenza Prevalence Case, in arXiv 2020. [paper]
• Adversarial sparse transformer for time series forecasting, in NeurIPS 2020. [paper] [code]
• Enhancing the locality and breaking the memory bottleneck of transformer on time series forecasting, in NeurIPS 2019. [paper] [code]
• SSDNet: State Space Decomposition Neural Network for Time Series Forecasting, in ICDM 2021, [paper]
• From Known to Unknown: Knowledge-guided Transformer for Time-Series Sales Forecasting in Alibaba, in arXiv 2021. [paper]
• TCCT: Tightly-coupled convolutional transformer on time series forecasting, in Neurocomputing 2022. [paper]
• Triformer: Triangular, Variable-Specific Attentions for Long Sequence Multivariate Time Series Forecasting, in IJCAI 2022. [paper]

Spatio-Temporal Forecasting

• AirFormer: Predicting Nationwide Air Quality in China with Transformers, in AAAI 2023. [paper] [official code]
• Earthformer: Exploring Space-Time Transformers for Earth System Forecasting, in NeurIPS 2022. [paper] [official code]
• Bidirectional Spatial-Temporal Adaptive Transformer for Urban Traffic Flow Forecasting, in TNNLS 2022. [paper]
• Spatio-temporal graph transformer networks for pedestrian trajectory prediction, in ECCV 2020. [paper] [official code]
• Spatial-temporal transformer networks for traffic flow forecasting, in arXiv 2020. [paper] [official code]
• Traffic transformer: Capturing the continuity and periodicity of time series for traffic forecasting, in Transactions in GIS 2022. [paper]

Event Irregular Time Series Modeling

• Time Series as Images: Vision Transformer for Irregularly Sampled Time Series，in NeurIPS 2023. [paper]
• ContiFormer: Continuous-Time Transformer for Irregular Time Series Modeling，in NeurIPS 2023. [paper]
• HYPRO: A Hybridly Normalized Probabilistic Model for Long-Horizon Prediction of Event Sequences，in NeurIPS 2022. [paper] [official code]
• Transformer Embeddings of Irregularly Spaced Events and Their Participants, in ICLR 2022. [paper] [[official cod