<img src="resources/title.png" style="width: 85%;" id="title-icon"> 🌐 <a href="https://cevalbenchmark.com/" target="_blank">Website</a> • 🤗 <a href="https://huggingface.co/datasets/ceval/ceval-exam" target="_blank">Hugging Face</a> • ⏬ <a href="#data" target="_blank">Data</a> • 📃 <a href="https://arxiv.org/abs/2305.08322" target="_blank">Paper</a> 📖 <a href="resources/tutorial.md" target="_blank">Tutorial (ZH)</a> <a href="https://github.com/SJTU-LIT/ceval/blob/main/README_zh.md"> 中文</a> | <a href="https://github.com/SJTU-LIT/ceval/blob/main/README.md">English

C-Eval is a comprehensive Chinese evaluation suite for foundation models. It consists of 13948 multi-choice questions spanning 52 diverse disciplines and four difficulty levels, as shown below. Please visit our website or check our paper for more details.

We hope C-Eval could help developers track the progress and analyze the important strengths/shortcomings of their models.

📝 Why C-Eval? How did we build it? (in Chinese)

News

[2025.7.27] We have released the complete C-Eval test set to the community!
[2023.10.26] C-Eval has been accepted to NeurIPS 2023 🎉🎉🎉
[2023.07.17] C-Eval now has been added into lm-evaluation-harness 🚀🚀🚀 Please refer to Use Through Evaluation Harness for details.

Leaderboard
C-Eval Hard Leaderboard
Results On Validation Split
Data
How to Evaluate on C-Eval
Licenses
Citation

Leaderboard

Below are zero-shot and five-shot accuracies from the models that we evaluate in the initial release, please visit our official Leaderboard for up-to-date models and their detailed results on each subject. We note that zero-shot performance is better than five-shot for many instruction-tuned models.

Zero-shot

| Model | STEM | Social Science | Humanities | Other | Average | | ------------------- | :--: | :------------: | :--------: | :---: | :-----: | | GPT-4 | 65.2 | 74.7 | 62.5 | 64.7 | 66.4 | | ChatGPT | 49.0 | 58.0 | 48.8 | 50.4 | 51.0 | | Claude-v1.3 | 48.5 | 58.6 | 47.3 | 50.1 | 50.5 | | Bloomz-mt-176B | 39.1 | 53.0 | 47.7 | 42.7 | 44.3 | | GLM-130B | 36.7 | 55.8 | 47.7 | 43.0 | 44.0 | | Claude-instant-v1.0 | 38.6 | 47.6 | 39.5 | 39.0 | 40.6 | | ChatGLM-6B | 33.3 | 48.3 | 41.3 | 38.0 | 38.9 | | LLaMA-65B | 32.6 | 41.2 | 34.1 | 33.0 | 34.7 | | MOSS | 31.6 | 37.0 | 33.4 | 32.1 | 33.1 | | Chinese-Alpaca-13B | 27.4 | 39.2 | 32.5 | 28.0 | 30.9 | | Chinese-LLaMA-13B | 28.8 | 32.9 | 29.7 | 28.0 | 29.6 |

Five-shot

| Model | STEM | Social Science | Humanities | Other | Average | | ------------------- | :--: | :------------: | :--------: | :---: | :-----: | | GPT-4 | 67.1 | 77.6 | 64.5 | 67.8 | 68.7 | | ChatGPT | 52.9 | 61.8 | 50.9 | 53.6 | 54.4 | | Claude-v1.3 | 51.9 | 61.7 | 52.1 | 53.7 | 54.2 | | Claude-instant-v1.0 | 43.1 | 53.8 | 44.2 | 45.4 | 45.9 | | GLM-130B | 34.8 | 48.7 | 43.3 | 39.8 | 40.3 | | Bloomz-mt-176B | 35.3 | 45.1 | 40.5 | 38.5 | 39.0 | | LLaMA-65B | 37.8 | 45.6 | 36.1 | 37.1 | 38.8 | | ChatGLM-6B | 30.4 | 39.6 | 37.4 | 34.5 | 34.5 | | Chinese LLaMA-13B | 31.6 | 37.2 | 33.6 | 32.8 | 33.3 | | MOSS | 28.6 | 36.8 | 31.0 | 30.3 | 31.1 | | Chinese Alpaca-13B | 26.0 | 27.2 | 27.8 | 26.4 | 26.7 |

C-Eval Hard Leaderboard

We select 8 challenging math, physics, and chemistry subjects from C-Eval to form a separate benchmark, C-Eval Hard, which includes advanced mathematics, discrete mathematics, probability and statistics, college chemistry, college physics, high school mathematics, high school chemistry, and high school physics. These subjects often involve with complex LaTeX equations and require non-trivial reasoning abilities to solve. Zero-shot and five-shot accuracies are shown below.

| Model | Zero-shot | Five-shot | | ------------------- | :-------: | :-------: | | GPT-4 | 53.3 | 54.9 | | Claude-v1.3 | 37.6 | 39.0 | | ChatGPT | 36.7 | 41.4 | | Claude-instant-v1.0 | 32.1 | 35.5 | | Bloomz-mt | 30.8 | 30.4 | | GLM-130B | 30.7 | 30.3 | | LLaMA-65B | 29.8 | 31.7 | | ChatGLM-6B | 29.2 | 23.1 | | MOSS | 28.4 | 24.0 | | Chinese-LLaMA-13B | 27.5 | 27.3 | | Chinese-Alpaca-13B | 24.4 | 27.1 |

Results On Validation Split

Since we do not publicly release the labels for our test split, we provide the zero-shot and five-shot average accuracy on the validation split as a reference for developers. The validation split comprises a total of 1346 questions. We report the average answer-only accuracy across all subjects in table below. The average validation accuracy closely mirrors the average test accuracy as presented in Leaderboard.

| Model | Zero-shot | Five-shot | | ------------------- | :-------: | :-------: | | GPT-4 | 66.7 | 69.9 | | Claude-v1.3 | 52.1 | 55.5 | | ChatGPT | 50.8 | 53.5 | | Bloomz-mt | 45.9 | 38.0 | | GLM-130B | 44.2 | 40.8 | | Claude-instant-v1.0 | 43.2 | 47.4 | | ChatGLM-6B | 39.7 | 37.1 | | LLaMA-65B | 38.6 | 39.8 | | MOSS | 35.1 | 28.9 | | Chinese-Alpaca-13B | 32.0 | 27.2 | | Chinese-LLaMA-13B | 29.4 | 33.1 |

Data

Download

Method 1: Download the zip file (you can also simply open the following link with the browser):

wget https://huggingface.co/datasets/ceval/ceval-exam/resolve/main/ceval-exam.zip

then unzip it and you may load the data with pandas:

import os
import pandas as pd

File_Dir="ceval-exam"
test_df=pd.read_csv(os.path.join(File_Dir,"test","computer_network_test.csv"))

Method 2: Directly load the dataset using Hugging Face datasets:

from datasets import load_dataset
dataset=load_dataset(r"ceval/ceval-exam",name="computer_network")

print(dataset['val'][0])
# {'id': 0, 'question': '使用位填充方法，以01111110为位首flag，数据为011011111111111111110010，求问传送时要添加几个0____', 'A': '1', 'B': '2', 'C': '3', 'D': '4', 'answer': 'C', 'explanation': ''}

Notes

To facilitate usage, we have organized the subject name handlers and English/Chinese names corresponding to 52 subjects. Please refer to subject_mapping.json for details. The format is:

# the dict key is the subject handler, and the dict value is (English name, Chinese name, category) tuple 
{
    "computer_network": [
        "Computer Network",
        "计算机网络",
        "STEM"
    ],
    ...
    "filename":[
        "English Name",
        "Chinese Name"
        "Supercatagory Label(STEM, Social Science, Humanities or Other)"
    ]
}

Each subject consists of three splits: dev, val, and test. The dev set per subject consists of five exemplars with explanations for few-shot evaluation. The val set is intended to be used for hyperparameter tuning. And the test set is for model evaluation. Labels on the test split are not released, users are required to submit their results to automatically obtain test accuracy. How to submit?

Below is a dev example from computer network:

id: 1
question: 25 °C时，将pH=2的强酸溶液与pH=13的强碱溶液混合，所得混合液的pH=11，则强酸溶液与强碱溶液 的体积比是(忽略混合后溶液的体积变化)____
A: 11:1
B: 9:1
C: 1:11
D: 1:9
answer: B
explanation: 
1. pH=13的强碱溶液中c(OH-)=0.1mol/L, pH=2的强酸溶液中c(H+)=0.01mol/L，酸碱混合后pH=11，即c(OH-)=0.001mol/L。
2. 设强酸和强碱溶液的体积分别为x和y，则：c(OH-)=(0.1y-0.01x)/(x+y)=0.001，解得x:y=9:1。

How to Evaluate on C-Eval

Normally you can directly take the model's generations and extract the answer token (i.e. A,B,C,D) from it with simple regular expressions. In few-shot evaluation, the model usually follows the given template thus this is easy. Sometimes, however, especially in zero-shot evaluation for models without experiencing instruction tuning, the model may not follow the instruction well to give a well-formatted generation, in this case we recommend computing the probability of "A", "B", "C", "D" and take the most likely one as the answer -- this is a constrained decoding approach and was used in the official MMLU test code. Such a probability approach is not applicable for chain-of-thought settings. [More detailed evaluation tutor

Ceval

Install / Use

README