First-Order Error Matters: Accurate Compensation for Quantized Large Language Models.

FOEM has been accepted at AAAI 2026.

We have completed the integration with GPTQModel.

Parts of this repository are now outdated, but we keep it available for developers who wish to debug or experiment with the algorithm.

The code snippets and results below are all obtained using GPTQModel.

Quant

from datasets import load_dataset
from gptqmodel import GPTQModel, QuantizeConfig, FOEMConfig

size = "8B"
model_id = f"Qwen/Qwen3-{size}"
quant_path = f"models/gptqmodel/Qwen3-{size}-foem-4bit"

calibration_dataset = load_dataset(
    "allenai/c4",
    data_files="en/c4-train.00001-of-01024.json.gz",
    split="train"
  ).select(range(256))["text"]

quant_config = QuantizeConfig(bits=4, group_size=128, foem=FOEMConfig(alpha=0, beta=0.2, device="auto"))

model = GPTQModel.load(model_id, quant_config)

model.quantize(calibration_dataset, batch_size=4)

model.save(quant_path)

Eval

lm-eval --model vllm --model_args pretrained=models/gptqmodel/Qwen3-8B-foem-4bit,tensor_parallel_size=1,dtype=auto,gpu_memory_utilization=0.6 --tasks wikitext --batch_size auto

Result

Note: The PPL evaluation on WikiText using lm-eval differs from that reported in our original paper.

| Model | Method | Bits | Hyperparameters | Wikitext PPL | | ---------- | ---------------- | ---- | -------------------- | ------------ | | Qwen3-0.6B | GPTQ | 4 | \ | 30.0372 | | | GPTAQ | 4 | alpha=0.25 | 30.5776 | | | FOEM (w/o GPTAQ) | 4 | alpha=0, beta=0.2 | 29.6199 | | | FOEM (w/ GPTAQ) | 4 | alpha=0.25, beta=0.2 | 29.3823 | | Qwen3-8B | GPTQ | 4 | \ | 12.5488 | | | GPTAQ | 4 | alpha=0.25 | 12.7152 | | | FOEM (w/o GPTAQ) | 4 | alpha=0, beta=0.2 | 12.5128 | | | FOEM (w/ GPTAQ) | 4 | alpha=0.25, beta=0.2 | 12.6172 |

Citation

If you find this work useful, please cite:

@inproceedings{zheng2026first,
  title={First-order error matters: Accurate compensation for quantized large language models},
  author={Zheng, Xingyu and Qin, Haotong and Li, Yuye and Chu, Haoran and Wang, Jiakai and Guo, Jinyang and Magno, Michele and Liu, Xianglong},
  booktitle={Proceedings of the AAAI Conference on Artificial Intelligence},
  volume={40},
  number={34},
  pages={28883--28891},
  year={2026}
}