A Survey on Self-Supervised Graph Foundation Models: Knowledge-Based Perspective

This is an extensive and continuously updated compilation of self-supervised GFM literature categorized by the knowledge-based taxonomy, proposed by our TKDE paper :page_facing_up:A Survey on Self-Supervised Graph Foundation Models: Knowledge-Based Perspective arXiv (full) | IEEE Xplore. Here every pretext of each paper is listed and briefly explained. You can find all pretexts and their corresponding papers with detailed metadata below, including additional pretexts and literature not listed in our paper.

A kind reminder: to search for a certain paper, type the title or the abbreviation of the proposed method (recommended) into the browser search bar (Ctrl + F). :warning:Some papers fall under multiple sections.

News

• [7 May 2025]: Full version (v3) of our paper uploaded to arXiv! Check now! :fire:
• [3 May 2025]: Our paper is accepted in TKDE! :fire::fire::fire: Final version coming soon!
• [3 May 2025]: Updated papers in AAAI'25, NAACL'25 and more. Rearranged "Graph Language Models" to be more detailed and accurate.
• [8 Feb 2025]: Updated papers in ICLR'25, WWW'25 and more.
• [5 Dec 2024]: Updated papers in WSDM'25, LoG'24 and more.
• [4 Oct 2024]: Updated papers in CIKM'24 and NeurIPS'24.
• [2 Sept 2024]: Updated papers in IJCAI'24, SIGIR'24, and KDD'24.
• [1 Aug 2024]: We have a huge update (v2) thanks to the joining of Dr. Yixin Su! :fire:
• [1 Aug 2024]: Updated papers in ICDE'24 and MM'24.
• [24 Mar 2024]: Our survey has uploaded to arXiv!

Contents

• Relevant surveys, benchmarks & empirical studies

• Microscopic pre-training

  • Node features
  • Node properties
  • Links

• Mesoscopic pre-training

  • Context
  • Long-range similarities
  • Motifs
  • Clusters

• Macroscopic pre-training

  • Global structure
  • Manifolds

• Multi-task pre-training

• Downstream tuning

• Graph language models

Relevant surveys, benchmarks & empirical studies

Note: :spider_web: graph-related; :robot: LLM-related; :books: survey; :bar_chart: benchmark; :microscope: empirical study

| Paper | Venue | | ------------------------------------------------------------ | --------------------------------- | | Pre-trained Models for Natural Language Processing: A Survey:books: | SCTS'20 | | Self-supervised Learning on Graphs: Deep Insights and New Direction:spider_web::microscope: | arXiv:2006 | | Pretrained Language Models for Text Generation: A Survey:robot::books: | IJCAI'21 | | An Empirical Study of Graph Contrastive Learning:spider_web::microscope: | NeurIPS'21 | | Self-supervised Learning: Generative or Contrastive:spider_web::books: | TKDE'21 | | Self-supervised Learning on Graphs: Contrastive, Generative, or Predictive:spider_web::books: | TKDE'21 | | A Survey on Contrastive Self-Supervised Learning:books: | Technologies'21 | | Pre-Trained Models: Past, Present and Future:robot::books: | AI Open'21 | | On the Opportunities and Risks of Foundation Models:robot::books: | arXiv:2108 | | A Survey of Pretrained Language Models:robot::books: | KSEM'22 | | Contrastive Self-Supervised Learning: A Survey on Different Architectures:books: | ICAI'22 | | Graph Self-Supervised Learning: A Survey:spider_web::books::bar_chart: | TKDE'22 | | Self-Supervised Learning of Graph Neural Networks: A Unified Review:spider_web::books: | TPAMI'22 | | A Survey of Pretraining on Graphs: Taxonomy, Methods, and Applications:spider_web::books: | arXiv:2202 | | A Survey on Masked Autoencoder for Self-supervised Learning in Vision and Beyond:books: | arXiv:2208 | | A Systematic Survey of Chemical Pre-trained Models:spider_web::books: | IJCAI'23 | | Can Language Models Solve Graph Problems in Natural Language?:spider_web::robot::microscope: | NeurIPS'23 | | Graph Meets LLMs: Towards Large Graph Models:spider_web::robot::books: | NeurIPS Workshop (GLFrontiers)'23 | | Beyond Text: A Deep Dive into Large Language Models’ Ability on Understanding Graph Data​:spider_web::robot::microscope: | NeurIPS Workshop (GLFrontiers)'23 | | Self-supervised Learning: A Succinct Review:books: | Arch. Comput. Methods Eng.'23 | | Self-Supervised Learning for Recommender Systems: A Survey:books: | TKDE'23 | | To Compress or Not to Compress - Self-Supervised Learning and Information Theory: A Review:books: | arXiv:2304 | | GPT4Graph: Can Large Language Models Understand Graph Structured Data? An Empirical Evaluation and Benchmarking:spider_web::robot::bar_chart::microscope: | arXiv:2305 | | Evaluating Large Language Models on Graphs: Performance Insights and Comparative Analysis:spider_web::robot::microscope: | arXiv:2308 | | Graph Prompt Learning: A Comprehensive Survey and Beyond:spider_web::robot::books: | arXiv:2311 | | Talk like a Graph: Encoding Graphs for Large Language Models:spider_web::robot::microscope: | ICLR'24 | | Which Modality should I use - Text, Motif, or Image? : Understanding Graphs with Large Language Models:spider_web::robot::bar_chart: | NAACL Findings'24 | | A Survey of Graph Meets Large Language Model: Progress and Future Directions:spider_web::robot::books: | IJCAI'24 | | Position: Graph Foundation Models are Already Here:spider_web::robot: | ICML'24 | | VisionGraph: Leveraging Large Multimodal Models for Graph Theory Problems in Visual Context:spider_web::robot::bar_chart: | ICML'24 | | A Survey of Large Language Models for Graphs:spider_web::robot::books: | KDD'24 | | LLM4DyG: Can LLMs Solve Spatial-Temporal Problems on Dynamic Graphs?:spider_web::robot::bar_chart::microscope: | KDD'24 | | Investigating Instruction Tuning Large Language Models on Graphs:spider_web::robot::bar_chart: | COLM'24 | | Do Neural Scaling Laws Exist on Graph Self-Supervised Learning?:spider_web::bar_chart::microscope: | LoG'24 | | ProG: A Graph Prompt Learning Benchmark:spider_web::bar_chart: | NeurIPS'24 | | GLBench: A Comprehensive Benchmark for Graph with Large Language Models:spider_web::robot::bar_chart: | NeurIPS'24 | | Text-space Graph Foundation Models: Comprehensive Benchmarks and New Insights:spider_web::robot::bar_chart::microscope: | NeurIPS'24 | | Can Large Language Models Analyze Graphs like Professionals? A Benchmark and Dataset:spider_web::robot::bar_chart: | NeurIPS'24 | | DTGB: A Comprehensive Benchmark for Dynamic Text-Attributed Graphs:spider_web::robot::bar_chart: | NeurIPS'24 | | Exploring the Potential of Large Language Models (LLMs) in Learning on Graphs:spider_web::robot::microscope: | KDD Explor. Newsl.'24 | | Integrating Graphs with Large Language Models: Methods and Prospects:spider_web::robot: | IEEE Intell. Syst.'24 | | A Comprehensive Survey on Pretrained Foundation Models: A History from BERT to ChatGPT:spider_web::robot::microscope: | IJMLC'24 | | [Large Language Models on Graphs: A Comprehensive Survey](https://arxiv