Awesome KV Cache Optimization

<h1 align="center">Awesome KV Cache Optimization</h1> <div align="center">

</div> <div align="center">

[<a href="https://aclanthology.org/2026.findings-acl.1916/">ACL Anthology</a>] [<a href="https://arxiv.org/abs/2607.08057">arXiv</a>] [<a href="https://jjiantong.github.io/Awesome-KV-Cache-Optimization/">Website</a>] [<a href="https://www.linkedin.com/feed/update/urn:li:activity:7384388868407529472/">LinkedIn</a>] [<a href="https://mp.weixin.qq.com/s/466l3Dh4xD4LRLR8vSOc8Q">机器之心</a>]

</div>

This repository is for our survey paper (ACL 2026 Findings):

Towards Efficient Large Language Model Serving: A Survey on System-Aware KV Cache Optimization
*Jiantong Jiang¹, Peiyu Yang^1*, Rui Zhang², Feng Liu¹
¹The University of Melbourne, ²Huazhong University of Science and Technology

---

This repository aims to record papers of system-aware, serving-time, KV-centric optimization methods that improve system metrics without retraining or architecture modification (which we call this scope sKis). We systematize recent advances via a system behavior-oriented taxonomy, organizing existing efforts into three behavioral dimensions:
🔷 Temporal — when is KV cache accessed or computed?
🔷 Spatial — where is KV cache placed and migrated?
🔷 Structural — how is KV cache represented and managed?

🧠 Grounded in this taxonomy, we analyze cross-behavior co-design affinity and behavior–objective effects, revealing overlooked regions and concrete open challenges.

<p align="center"> <img src="assets/awesome-cover.png" width="90%" style="align:center;"/> </p>

News

• 2026-06: 🔥 51 new papers were added.
• 2026-04: 🎉 Our survey was accepted to ACL 2026.
• 2026-01: 📰 Our survey was featured by 机器之心.
• 2025-11: ⭐ 100-star milestone! Thanks to the amazing support from the community!
• 2025-10: 📝 The survey was available on TechRxiv.

Contributing

The repository is still under active development and will be updated regularly.

🙋 If you would like to include your paper in this survey and repository, please feel free to submit a pull request. You can generate the markdown row for each paper by filling in the first part of generate.py and running python generate.py. Alternatively, you can open an issue with the paper's title and a brief summary highlighting its key techniques. You can also contact us via email.

🙋🏻‍♀️ Please let us know if you find out a mistake or have any suggestions! We greatly appreciate your feedback regarding this repository or survey!

🌟 If you find this resource helpful for your work, please consider giving us a star and citing our research.

---

<a name="readme-index"></a>

Quick Index

• Temporal — Execution & Scheduling
  • KV-Centric Scheduling (KVS)
  • Pipelining & Overlapping (OVLP)
  • Hardware-Aware Execution (HAE)
• Spatial — Placement & Migration
  • Memory Hierarchy KV Orchestration (MHO)
  • Compute Device KV Orchestration (CDO)
• Structural — Representation & Retention
  • KV Cache Compression (KVCC) (including quantization, low-rank approximation, and structural compression, and codec-based compression)
  • KV Cache Retention Management (KVRM) (including allocation, reuse, and eviction)
• Cross-behavior Co-design Affinity
• Behavior-objective Effects

<p align="center"> <img src="assets/literature.png" width = "95%" alt="" align=center /> </p>

---

Temporal — Execution & Scheduling

These methods act on when KV data is executed, computed, or scheduled to improve latency and throughput. We divide these methods into three categories: KV-centric scheduling, pipelining & overlapping, and hardware-aware execution.

KV-Centric Scheduling

KV-centric scheduling methods explicitly integrate KV characteristics into runtime scheduling decisions.

|Paper|Type|Code| | -- | -- | -- | | <br> AlignedServe: Orchestrating Prefix-aware Batching to Build a High-throughput and Computing-efficient LLM Serving System [Link] <br> Fengyao Bai, Hongbin Zhang, Zhitao Chen, Jiangsu Du, Zhiguang Chen, Yutong Lu | Prefix-aware token scheduling | | | <br> CoDec: Prefix-Shared Decoding Kernel for LLMs [Link] <br> Zhibin Wang, Rui Ning, Chao Fang, Zhonghui Zhang, Xi Lin, Shaobo Ma, Mo Zhou, Xue Li, Zhongfeng Wang, Chengying Huan, Rong Gu, Kun Yang, Guihai Chen, Sheng Zhong, Chen Tian | Shared KV access scheduling across requests | <br> <br> codec | | <br> HotPrefix: Hotness-Aware KV Cache Scheduling for Efficient Prefix Sharing in LLM Inference Systems [Link] <br> Yuhang Li, Rong Gu, Chengying Huan, Zhibin Wang, Renjie Yao, Chen Tian, Guihai Chen | Hotness-aware KV cache scheduling for prefix sharing; Also belongs to KV cache retention management | | | <br> PRISM: Fast Online LLM Serving via Scheduling-Memory Co-design [Link] <br> Xingyu Qu, Tianhao Lin, Yiqi Li, Zhiyu Chen, Sheng Wang | KV reuse-aware request scheduling; Also belongs to allocation & reuse (structural) | | | <br> DASH-KV: Accelerating Long-Context LLM Inference via Asymmetric KV Cache Hashing [Link] <br> Jinyu Guo, Zhihan Zhang, Jiehui Xie, Md. Tamim Iqbal, Dongshen Han, Lik-Hang Lee, Sung-Ho Bae, Jie Zou, Yang Yang, Chaoning Zhang | Hash-based token-level attention compute scheduling | | | <br> TokenSelect: Efficient Long-Context Inference and Length Extrapolation for LLMs via Dynamic Token-Level KV Cache Selection [Link] <br> Wei Wu, Zhuoshi Pan, Kun Fu, Chao Wang, Liyi Chen, Yunchu Bai, Tianfu Wang, Zheng Wang, Hui Xiong | Token-level attention compute scheduling | <br> <br> TokenSelect| | <br> TinyServe: Query-Aware Cache Selection for Efficient LLM Serving [Link] <br> Dong Liu, Yanxuan Yu | Query-aware page-level KV selection for sparse attention computation; Also belongs to memory hierarchy KV orchestration (spatial) | | | <br> TokenCake: A KV-Cache-centric Serving Framework for LLM-based Multi-Agent Applications [Link] <br> Zhuohang Bian, Feiyang Wu, Zhuoran Li, Teng Ma, Youwei Zhuo | Agent-aware KV scheduling under tool-call stalls; Also belongs to memory hierarchy KV orchestration | | | <br> RefreshKV: Updating Small KV Cache During Long-form Generation [Link] <br> Fangyuan Xu, Tanya Goyal, Eunsol Choi| Token-level attention compute scheduling | <br> <br> RefreshKV| | <br> RocketKV: Accelerating Long-Context LLM Inference via Two-Stage KV Cache Compression [Link] <br> Payman Behnam, Yaosheng Fu, Ritchie Zhao, Po-An Tsai, Zhiding Yu, Alexey Tumanov| Token-level attention compute scheduling | <br> <br> RocketKV| | ![Award](https://img.shie