Ipex Llm

Accelerate local LLM inference and finetuning (LLaMA, Mistral, ChatGLM, Qwen, DeepSeek, Mixtral, Gemma, Phi, MiniCPM, Qwen-VL, MiniCPM-V, etc.) on Intel XPU (e.g., local PC with iGPU and NPU, discrete GPU such as Arc, Flex and Max); seamlessly integrate with llama.cpp, Ollama, HuggingFace, LangChain, LlamaIndex, vLLM, DeepSpeed, Axolotl, etc.

Generate Convert Improve

Install / Use

npx skills add intel/ipex-llm

Installs into whichever agent you are using.

About this skill

Quality Score

0/100

README

THIS PROJECT IS ARCHIVED

Intel will not provide or guarantee development of or support for this project, including but not limited to, maintenance, bug fixes, new releases or updates.
Patches to this project are no longer accepted by Intel.
This project has been identified as having known security issues.

💫 Intel® LLM Library for PyTorch*

< English | <a href='./README.zh-CN.md'>中文</a> >

IPEX-LLM is an LLM acceleration library for Intel GPU (e.g., local PC with iGPU, discrete GPU such as Arc, Flex and Max), NPU and CPU [^1].

[!NOTE]

IPEX-LLM provides seamless integration with llama.cpp, Ollama, vLLM, HuggingFace transformers, LangChain, LlamaIndex, Text-Generation-WebUI, DeepSpeed-AutoTP, FastChat, Axolotl, HuggingFace PEFT, HuggingFace TRL, AutoGen, ModeScope, etc.

70+ models have been optimized/verified on ipex-llm (e.g., Llama, Phi, Mistral, Mixtral, DeepSeek, Qwen, ChatGLM, MiniCPM, Qwen-VL, MiniCPM-V and more), with state-of-art LLM optimizations, XPU acceleration and low-bit (FP8/FP6/FP4/INT4) support; see the complete list here.

Latest Update 🔥

[2025/05] You can now run DeepSeek V3/R1 671B and Qwen3MoE 235B models with just 1 or 2 Intel Arc GPU (such as A770 or B580) using FlashMoE in ipex-llm.
[2025/04] We released ipex-llm 2.2.0, which includes Ollama Portable Zip and llama.cpp Portable Zip.
⚠️ Warning (for llama.cpp Portable Zip)
mmap-based model loading in llama.cpp may leak data via side-channels in multi-tenant or shared-host environments.
To disable mmap, add:
```
--no-mmap
```
[2025/04] We added support of PyTorch 2.6 for Intel GPU.
[2025/03] We added support for Gemma3 model in the latest llama.cpp Portable Zip.
[2025/03] We can now run DeepSeek-R1-671B-Q4_K_M with 1 or 2 Arc A770 on Xeon using the latest llama.cpp Portable Zip.
[2025/02] We added support of llama.cpp Portable Zip for Intel GPU (both Windows and Linux) and NPU (Windows only).
[2025/02] We added support of Ollama Portable Zip to directly run Ollama on Intel GPU for both Windows and Linux (without the need of manual installations).
[2025/02] We added support for running vLLM 0.6.6 on Intel Arc GPUs.
[2025/01] We added the guide for running ipex-llm on Intel Arc B580 GPU.
[2025/01] We added support for running Ollama 0.5.4 on Intel GPU.
[2024/12] We added both Python and C++ support for Intel Core Ultra NPU (including 100H, 200V, 200K and 200H series).

<details><summary>More updates</summary>

[2024/11] We added support for running vLLM 0.6.2 on Intel Arc GPUs.
[2024/07] We added support for running Microsoft's GraphRAG using local LLM on Intel GPU; see the quickstart guide here.
[2024/07] We added extensive support for Large Multimodal Models, including StableDiffusion, Phi-3-Vision, Qwen-VL, and more.
[2024/07] We added FP6 support on Intel GPU.
[2024/06] We added experimental NPU support for Intel Core Ultra processors; see the examples here.
[2024/06] We added extensive support of pipeline parallel inference, which makes it easy to run large-sized LLM using 2 or more Intel GPUs (such as Arc).
[2024/06] We added support for running RAGFlow with ipex-llm on Intel GPU.
[2024/05] ipex-llm now supports Axolotl for LLM finetuning on Intel GPU; see the quickstart here.
[2024/05] You can now easily run ipex-llm inference, serving and finetuning using the Docker images.
[2024/05] You can now install ipex-llm on Windows using just "one command".
[2024/04] You can now run Open WebUI on Intel GPU using ipex-llm; see the quickstart here.
[2024/04] You can now run Llama 3 on Intel GPU using llama.cpp and ollama with ipex-llm; see the quickstart here.
[2024/04] ipex-llm now supports Llama 3 on both Intel GPU and CPU.
[2024/04] ipex-llm now provides C++ interface, which can be used as an accelerated backend for running llama.cpp and ollama on Intel GPU.
[2024/03] bigdl-llm has now become ipex-llm (see the migration guide here); you may find the original BigDL project here.
[2024/02] ipex-llm now supports directly loading model from ModelScope (魔搭).
[2024/02] ipex-llm added initial INT2 support (based on llama.cpp IQ2 mechanism), which makes it possible to run large-sized LLM (e.g., Mixtral-8x7B) on Intel GPU with 16GB VRAM.
[2024/02] Users can now use ipex-llm through Text-Generation-WebUI GUI.
[2024/02] ipex-llm now supports Self-Speculative Decoding, which in practice brings ~30% speedup for FP16 and BF16 inference latency on Intel GPU and CPU respectively.
[2024/02] ipex-llm now supports a comprehensive list of LLM finetuning on Intel GPU (including LoRA, QLoRA, DPO, QA-LoRA and ReLoRA).
[2024/01] Using ipex-llm QLoRA, we managed to finetune LLaMA2-7B in 21 minutes and LLaMA2-70B in 3.14 hours on 8 Intel Max 1550 GPU for Standford-Alpaca (see the blog here).
[2023/12] ipex-llm now supports ReLoRA (see "ReLoRA: High-Rank Training Through Low-Rank Updates").
[2023/12] ipex-llm now supports Mixtral-8x7B on both Intel GPU and CPU.
[2023/12] ipex-llm now supports QA-LoRA (see "QA-LoRA: Quantization-Aware Low-Rank Adaptation of Large Language Models").
[2023/12] ipex-llm now supports FP8 and FP4 inference on Intel GPU.
[2023/11] Initial support for directly loading GGUF, AWQ and GPTQ models into ipex-llm is available.
[2023/11] ipex-llm now supports [vL