Auto Round

<div align="center"> <p align="center"> <img src="docs/imgs/AutoRound.png" alt="AutoRound Overview" width="20%"> </p> <h3> Advanced Quantization Algorithm for LLMs</h3>

<a href="https://huggingface.co/Intel"> <img alt="Model Checkpoints" src="https://img.shields.io/badge/%F0%9F%A4%97%20HF-Models-F57C00"> </a>

    English | 简体中文

User Guide | 用户指南  

---

<div align="left">

🚀 What is AutoRound?

AutoRound is an advanced quantization toolkit designed for Large Language Models (LLMs) and Vision-Language Models (VLMs). It achieves high accuracy at ultra-low bit widths (2–4 bits) with minimal tuning by leveraging sign-gradient descent and providing broad hardware compatibility. See our papers SignRoundV1 and SignRoundV2 for more details. For usage instructions, please refer to the User Guide.

<p align="center"> <img src="docs/imgs/autoround_overview.png" alt="AutoRound Overview" width="80%"> </p>

🆕 What's New

• [2026/07] torch.compile is enabled by default except on Windows to accelerate quantization. Minor numerical differences compared with the non-compiled path are expected due to compiler optimizations. To disable it, pass enable_torch_compile=False to the Python API or use --disable_torch_compile on the CLI.

• [2026/06] AutoScheme has been refined to improve accuracy for gguf format. See AutoScheme Accuracy for details. This enhancement incurs additional tuning cost.

• [2026/06] AutoRound is now fully integrated into vLLM-Omni: vLLM blog

• [2026/05] We provide free devices for calibration-free quantization; please visit Intel Low Bit Open LLM Leaderboard for more details.

• [2026/05] Model free quantization is available, auto-round-rtn will now default to using the model-free approach: Doc.

• [2026/03] Block-wise FP8 quantization is available and rtn mode is recommended. auto-round-rtn --scheme FP8_BLOCK.

• [2026/03] MTP layer quantization has been supported in this PR

• [2025/12] The SignRoundV2 paper is available. Turn on enable_alg_ext and use the AutoScheme API for mixed-precision quantization to reproduce the results: Paper, Notes for evaluating LLaMA models.

• [2025/11] AutoRound has landed in LLM-Compressor: Usage, vLLM blog, RedHat blog, X post, Intel blog, Linkedin, 微信, 知乎.

• [2025/11] An enhanced GGUF quantization algorithm is available via --enable_alg_ext: Accuracy.

• [2025/10] AutoRound has been integrated into SGLang: Usage, LMSYS Blog, X post, Intel blog, Linkedin.

• [2025/10] A mixed precision algorithm is available to generate schemes in minutes: Usage, Accuracy.

• [2025/09] MXFP4 and NVFP4 dtypes is available: Accuracy.

• [2025/08] An improved INT2 algorithm is available via --enable_alg_ext: Accuracy

• [2025/07] GGUF format is supported: Usage.

• [2025/05] AutoRound has been integrated into vLLM: Usage, Medium blog, 小红书.

• [2025/05] AutoRound has been integrated into Transformers: Blog.

• [2025/03] The INT2-mixed DeepSeek-R1 model (~200GB) retains 97.9% accuracy: Model.

✨ Key Features

✅ Superior Accuracy Delivers strong performance even at 2–3 bits example models, with leading results at 4 bits benchmark.

✅ Ecosystem Integration Seamlessly works with Transformers, vLLM, SGLang and more.

✅ Multiple Formats Export Support AutoRound, AutoAWQ, AutoGPTQ, and GGUF for maximum compatibility. Details are shown in export formats

✅ Fast Mixed Bits/Dtypes Scheme Generation Automatically configure in minutes, with about 1.1X-1.5X the model’s BF16 RAM size as overhead. Accuracy results and user guide.

✅ Optimized Round-to-Nearest Mode Use --iters 0 for fast quantization with some accuracy drop for 4 bits. Details are shown in opt_rtn mode

✅ Affordable Quantization Cost Quantize 7B models in about 10 minutes on a single GPU. Details are shown in quantization costs

✅ 10+ VLMs Support Out-of-the-box quantization for 10+ vision-language models example models, support matrix

✅ Multiple Recipes Choose from auto-round-best, auto-round, auto-round-light, auto-round-opt-rtn (optimized RTN) and auto-round-rtn (pure RTN, fastest baseline) to suit your needs. Details are shown in quantization recipes

✅ Advanced Utilities Includes multiple gpus quantization, multiple calibration datasets and support for 10+ runtime backends.

✅ Beyond weight only quantization. We are actively expanding support for additional datatypes such as MXFP, NVFP, W8A8, and more.

Installation

Install from pypi

# CPU(Xeon)/GPU(CUDA)
pip install auto-round

# CPU(Xeon)/GPU(CUDA) nightly
pip install auto-round-nightly

# HPU(Gaudi)
# install inside the hpu docker container, e.g. vault.habana.ai/gaudi-docker/1.23.0/ubuntu24.04/habanalabs/pytorch-installer-2.9.0:latest  
pip install auto-round-hpu

# XPU(Intel GPU)
pip install torch --index-url https://download.pytorch.org/whl/xpu
pip install auto-round

<details> <summary>Build from Source</summary>

# CPU(Xeon)/GPU(CUDA)
pip install .

# HPU(Gaudi)
python setup.py install hpu

# XPU(Intel GPU)
pip install torch --index-url https://download.pytorch.org/whl/xpu
pip install .

</details>

Model Quantization (CPU/Intel GPU/Gaudi/CUDA)

If you encounter issues during quantization, try using pure RTN mode with iters=0, disable_opt_rtn=True. Additionally, using group_size=32 or mixed bits is recommended for better results.

CLI Usage

The full list of supported arguments is provided by calling auto-round -h on the terminal.

ModelScope is supported for model downloads, simply set AR_USE_MODELSCOPE=1.

auto-round \
    --model Qwen/Qwen3-0.6B \
    --scheme "W4A16" \
    --format "auto_round" \
    --output_dir ./tmp_autoround

We offer another two recipes, auto-round-best and auto-round-light, designed for optimal accuracy and improved speed, respectively. Details are as follows.

<details> <summary>Other Recipes</summary>

``