Loong

<img src="assets/logo.png" alt="Loong" width="120" align="left"><div align="center"><h1>  Loong: Benchmarking Long-Context LLMs with Extended Multi-Doc QA</h1></div>

<p align="center" style="font-size:200%"> <img alt="GitHub" src="https://img.shields.io/github/license/MozerWang/Loong.svg?color=blue&style=flat-square"> <img alt="GitHub top language" src="https://img.shields.io/github/languages/top/MozerWang/Loong"> <img alt="GitHub last commit" src="https://img.shields.io/github/last-commit/MozerWang/Loong"> </p> <p align="center"> <font size=6>📃</font> <a target="_self" href="https://arxiv.org/abs/2406.17419"> <img style="height:14pt" src="https://img.shields.io/badge/-Paper-red?style=flat&logo=arxiv"></a> <font size=6>•</font> <font size=6>💻</font> <a target="_self" href="https://github.com/MozerWang/Loong"> <img style="height:14pt" src="https://img.shields.io/badge/-Code-pink?style=flat&logo=github"></a> <font size=6>•</font> <font size=6>🤗</font> <a target="_self" href="https://modelscope.cn/datasets/iic/Loong"> <img style="height:14pt" src="https://img.shields.io/badge/-🤗%20Dataset-red?style=flat"></a> </p>

👀Overview

This repository contains code for our paper Leave No Document Behind: Benchmarking Long-Context LLMs with Extended Multi-Doc QA. We propose a novel long-context benchmark, 🐉 Loong, aligning with realistic scenarios through extended multi-document question answering (QA). Loong typically consists of 11 documents per test instance on average, spanning three real-world scenarios in English and Chinese: (1) Financial Reports, (2) Legal Cases, and (3) Academic Papers. Meanwhile, Loong introduces new evaluation tasks from the perspectives of Spotlight Locating, Comparison, Clustering, and Chain of Reasoning, to facilitate a more realistic and comprehensive evaluation of long-context understanding. Furthermore, Loong features inputs of varying lengths (e.g., 10K-50K, 50K-100K, 100K-200K, beyond 200K) and evaluation tasks of diverse difficulty, enabling fine-grained assessment of LLMs across different context lengths and task complexities.

Please find more details of this work in our paper.

Showcase of four evaluation tasks in Loong (<di>...</di> marks the content of the i-th document). (a) Spotlight Locating: Locate the evidence. (b) Comparison: Locate and compare the evidence. (c) Clustering: Locate and cluster the evidence into groups. (d) Chain of Reasoning: Locate and reasoning along a logical chain.

📰News

[2024-09-20] 📰Our paper has been accepted to the EMNLP Main Conference.

[2024-07-30] 🤖The performance of phi-3, llama-3.1-8B, gpt-4o-mini on Loong are updated.

[2024-07-03] 🔥The code and benchmark are releasing. If you encounter any issues, please feel free to contact us.

[2024-06-25] 👨‍💻The code is currently being refined, and we plan to release the evaluation code and benchmark within the next one or two weeks. If you encounter any issues, please feel free to contact me at wangminzheng2023@ia.ac.cn.

🏆Leaderboard

<table> <thead> <tr> <th>Models</th> <th>Claimed Length</th> <th colspan="2" style="text-align: center;">Spotlight Locating</th> <th colspan="2" style="text-align: center;">Comparison</th> <th colspan="2" style="text-align: center;">Clustering</th> <th colspan="2" style="text-align: center;">Chain of Reason</th> <th colspan="2" style="text-align: center;">Overall</th> </tr> </thead> <tbody> <tr> <td><a href="https://ai.google.dev/gemini-api/docs/models/gemini#:~:text=Gemini-,Gemini%201.5%20Pro%20(Preview%20only),-Text%20and%20images">Gemini-1.5-pro</a></td> <td style="text-align: center;">1000K</td> <td style="text-align: center;">75.02</td><td style="text-align: center;">0.56</td> <td style="text-align: center;">49.94</td><td style="text-align: center;">0.27</td> <td style="text-align: center;">44.10</td><td style="text-align: center;">0.09</td> <td style="text-align: center;">64.97</td><td style="text-align: center;">0.37</td> <td style="text-align: center;">55.37</td><td style="text-align: center;">0.27</td> </tr> <tr style="background-color:#f0f0f0;"> <td><a href="https://platform.openai.com/docs/models/gpt-4o">GPT-4o</a></td> <td style="text-align: center;">128K</td> <td style="text-align: center;">73.95</td><td style="text-align: center;">0.62</td> <td style="text-align: center;">50.50</td><td style="text-align: center;">0.28</td> <td style="text-align: center;">44.29</td><td style="text-align: center;">0.09</td> <td style="text-align: center;">57.95</td><td style="text-align: center;">0.28</td> <td style="text-align: center;">53.47</td><td style="text-align: center;">0.26</td> </tr> <tr> <td><a href="https://docs.anthropic.com/en/docs/intro-to-claude#claude-3-5-family">Claude3.5-Sonnet</a></td> <td style="text-align: center;">200K</td> <td style="text-align: center;">58.45</td><td style="text-align: center;">0.49</td> <td style="text-align: center;">54.21</td><td style="text-align: center;">0.35</td> <td style="text-align: center;">45.77</td><td style="text-align: center;">0.07</td> <td style="text-align: center;">43.92</td><td style="text-align: center;">0.25</td> <td style="text-align: center;">48.85</td><td style="text-align: center;">0.23</td> </tr> <tr style="background-color:#f0f0f0;"> <td><a href="https://docs.anthropic.com/en/docs/intro-to-claude#claude-3-family">Claude3-Haiku</a></td> <td style="text-align: center;">200K</td> <td style="text-align: center;">68.68</td><td style="text-align: center;">0.59</td> <td style="text-align: center;">42.10</td><td style="text-align: center;">0.21</td> <td style="text-align: center;">35.04</td><td style="text-align: center;">0.02</td> <td style="text-align: center;">47.59</td><td style="text-align: center;">0.17</td> <td style="text-align: center;">44.88</td><td style="text-align: center;">0.19</td> </tr> <tr> <td><a href="https://huggingface.co/Qwen/Qwen2-72B-Instruct">Qwen2-72B-Instruct</a></td> <td style="text-align: center;">128K</td> <td style="text-align: center;">54.17</td><td style="text-align: center;">0.36</td> <td style="text-align: center;">42.38</td><td style="text-align: center;">0.20</td> <td style="text-align: center;">36.71</td><td style="text-align: center;">0.04</td> <td style="text-align: center;">47.76</td><td style="text-align: center;">0.18</td> <td style="text-align: center;">43.29</td><td style="text-align: center;">0.15</td> </tr> <tr style="background-color:#f0f0f0;"> <td><a href="https://platform.openai.com/docs/models/gpt-4o-mini">GPT-4o-mini</a></td> <td style="text-align: center;">128K</td> <td style="text-align: center;">53.12</td><td style="text-align: center;">0.41</td> <td style="text-align: center;">44.27</td><td style="text-align: center;">0.20</td> <td style="text-align: center;">32.58</td><td style="text-align: center;">0.04</td> <td style="text-align: center;">52.34</td><td style="text-align: center;">0.23</td> <td style="text-align: center;">42.95</td><td style="text-align: center;">0.18</td> </tr> <tr> <td><a href="https://huggingface.co/THUDM/glm-4-9b-chat-1m">GLM4-9B-Chat</a></td> <td style="text-align: center;">1000K</td> <td style="text-align: center;">57.35</td><td style="text-align: center;">0.47</td> <td style="text-align: center;">40.38</td><td style="text-align: center;">0.20</td> <td style="text-align: center;">28.52</td><td style="text-align: center;">0.02</td> <td style="text-align: center;">39.94</td><td style="text-align: center;">0.16</td> <td style="text-align: center;">38.31</td><td style="text-align: center;">0.16</td> </tr> <tr style="background-color:#f0f0f0;"> <td><a href="https://kimi.moonshot.cn/">Kimi-Chat</a></td> <td style="text-align: center;">200K</td> <td style="text-align: center;">60.98</td><td style="text-align: center;">0.50</td> <td style="text-align: center;">34.74</td><td style="text-align: center;">0.13</td> <td style="text-align: center;">28.76</td><td style="text-align: center;">0.04</td> <td style="text-align: center;">38.52</td><td style="text-align: center;">0.15</td> <td style="text-align: center;">37.49</td><td style="text-align: center;">0.16</td> </tr> <tr> <td><a href="https://huggingface.co/meta-llama/Meta-Llama-3.1-8B-Instruct">Llama-3.1-8B-Instruct</a></td> <td style="text-align: center;">128K</td> <td style="text-align: center;">59.96</td><td style="text-align: center;">0.46</td> <td style="text-align: center;">35.73</td><td style="text-align: center;">0.18</td> <td style="text-align: center;">27.83</td><td style="text-align: center;">0.01</td> <td style="text-align: center;">35.59</td><td style="text-align: center;">0.14</td> <td style="text-align: center;">36.31</td><td style="text-align: center;">0.14</td> </tr> <tr style="background-color:#f0f0f0;"> <td><a href="https://huggingface.co/microsoft/Phi-3-small-128k-instruct">Phi-3-small</a></td> <td style="text-align: center;">128K</td> <td style="text-align: center;">29.23</td><td style="text-align: center;">0.10</td> <td style="text-align: center;">20.12</td><td style="text-align: center;">0.06</td> <td style="text-align: center;">17.53</td><td style="text-align: center;">0.00</td> <td style="text-align: center;">14.36</td><td style="text-align: center;">0.01</td> <td style="text-align: center;">19.03</td><td style="text-align: center;">0.03</td> </tr> <tr>