Marconi: Prefix Caching for the Era of Hybrid LLMs

This repository contains the source code implementation of the MLSys '25 paper Marconi: Prefix Caching for the Era of Hybrid LLMs.

Getting Started

Marconi is implemented in Python. We have tested Marconi on Ubuntu 22.04 with Python 3.11.9.

Detailed instructions on how to reproduce the main results from our MLSys paper are in artifact_evaluation.md.

References

@article{pan2024marconi,
  title={Marconi: Prefix Caching for the Era of Hybrid LLMs},
  author={Pan, Rui and Wang, Zhuang and Jia, Zhen and Karakus, Can and Zancato, Luca and Dao, Tri and Netravali, Ravi and Wang, Yida},
  journal={arXiv preprint arXiv:2411.19379},
  year={2024}
}