intel/ipex-llm

THIS PROJECT IS ARCHIVED

Intel will not provide or guarantee development of or support for this project, including but not limited to, maintenance, bug fixes, new releases or updates.
Patches to this project are no longer accepted by Intel.
This project has been identified as having known security issues.

💫 Intel® LLM Library for PyTorch*

< English | 中文 >

IPEX-LLM is an LLM acceleration library for Intel GPU (e.g., local PC with iGPU, discrete GPU such as Arc, Flex and Max), NPU and CPU ¹.

[!NOTE]

• IPEX-LLM provides seamless integration with llama.cpp, Ollama, vLLM, HuggingFace transformers, LangChain, LlamaIndex, Text-Generation-WebUI, DeepSpeed-AutoTP, FastChat, Axolotl, HuggingFace PEFT, HuggingFace TRL, AutoGen, ModeScope, etc.
• 70+ models have been optimized/verified on ipex-llm (e.g., Llama, Phi, Mistral, Mixtral, DeepSeek, Qwen, ChatGLM, MiniCPM, Qwen-VL, MiniCPM-V and more), with state-of-art LLM optimizations, XPU acceleration and low-bit (FP8/FP6/FP4/INT4) support; see the complete list here.

Latest Update 🔥

• [2025/05] You can now run DeepSeek V3/R1 671B and Qwen3MoE 235B models with just 1 or 2 Intel Arc GPU (such as A770 or B580) using FlashMoE in ipex-llm.
• [2025/04] We released ipex-llm 2.2.0, which includes Ollama Portable Zip and llama.cpp Portable Zip.

  ⚠️ Warning (for llama.cpp Portable Zip)
  mmap-based model loading in llama.cpp may leak data via side-channels in multi-tenant or shared-host environments.
  To disable mmap, add:

  --no-mmap
  

• [2025/04] We added support of PyTorch 2.6 for Intel GPU.
• [2025/03] We added support for Gemma3 model in the latest llama.cpp Portable Zip.
• [2025/03] We can now run DeepSeek-R1-671B-Q4_K_M with 1 or 2 Arc A770 on Xeon using the latest llama.cpp Portable Zip.
• [2025/02] We added support of llama.cpp Portable Zip for Intel GPU (both Windows and Linux) and NPU (Windows only).
• [2025/02] We added support of Ollama Portable Zip to directly run Ollama on Intel GPU for both Windows and Linux (without the need of manual installations).
• [2025/02] We added support for running vLLM 0.6.6 on Intel Arc GPUs.
• [2025/01] We added the guide for running ipex-llm on Intel Arc B580 GPU.
• [2025/01] We added support for running Ollama 0.5.4 on Intel GPU.
• [2024/12] We added both Python and C++ support for Intel Core Ultra NPU (including 100H, 200V, 200K and 200H series).

More updates

• [2024/11] We added support for running vLLM 0.6.2 on Intel Arc GPUs.
• [2024/07] We added support for running Microsoft's GraphRAG using local LLM on Intel GPU; see the quickstart guide here.
• [2024/07] We added extensive support for Large Multimodal Models, including StableDiffusion, Phi-3-Vision, Qwen-VL, and more.
• [2024/07] We added FP6 support on Intel GPU.
• [2024/06] We added experimental NPU support for Intel Core Ultra processors; see the examples here.
• [2024/06] We added extensive support of pipeline parallel inference, which makes it easy to run large-sized LLM using 2 or more Intel GPUs (such as Arc).
• [2024/06] We added support for running RAGFlow with ipex-llm on Intel GPU.
• [2024/05] ipex-llm now supports Axolotl for LLM finetuning on Intel GPU; see the quickstart here.
• [2024/05] You can now easily run ipex-llm inference, serving and finetuning using the Docker images.
• [2024/05] You can now install ipex-llm on Windows using just "one command".
• [2024/04] You can now run Open WebUI on Intel GPU using ipex-llm; see the quickstart here.
• [2024/04] You can now run Llama 3 on Intel GPU using llama.cpp and ollama with ipex-llm; see the quickstart here.
• [2024/04] ipex-llm now supports Llama 3 on both Intel GPU and CPU.
• [2024/04] ipex-llm now provides C++ interface, which can be used as an accelerated backend for running llama.cpp and ollama on Intel GPU.
• [2024/03] bigdl-llm has now become ipex-llm (see the migration guide here); you may find the original BigDL project here.
• [2024/02] ipex-llm now supports directly loading model from ModelScope (魔搭).
• [2024/02] ipex-llm added initial INT2 support (based on llama.cpp IQ2 mechanism), which makes it possible to run large-sized LLM (e.g., Mixtral-8x7B) on Intel GPU with 16GB VRAM.
• [2024/02] Users can now use ipex-llm through Text-Generation-WebUI GUI.
• [2024/02] ipex-llm now supports Self-Speculative Decoding, which in practice brings ~30% speedup for FP16 and BF16 inference latency on Intel GPU and CPU respectively.
• [2024/02] ipex-llm now supports a comprehensive list of LLM finetuning on Intel GPU (including LoRA, QLoRA, DPO, QA-LoRA and ReLoRA).
• [2024/01] Using ipex-llm QLoRA, we managed to finetune LLaMA2-7B in 21 minutes and LLaMA2-70B in 3.14 hours on 8 Intel Max 1550 GPU for Standford-Alpaca (see the blog here).
• [2023/12] ipex-llm now supports ReLoRA (see "ReLoRA: High-Rank Training Through Low-Rank Updates").
• [2023/12] ipex-llm now supports Mixtral-8x7B on both Intel GPU and CPU.
• [2023/12] ipex-llm now supports QA-LoRA (see "QA-LoRA: Quantization-Aware Low-Rank Adaptation of Large Language Models").
• [2023/12] ipex-llm now supports FP8 and FP4 inference on Intel GPU.
• [2023/11] Initial support for directly loading GGUF, AWQ and GPTQ models into ipex-llm is available.
• [2023/11] ipex-llm now supports vLLM continuous batching on both Intel GPU and CPU.
• [2023/10] ipex-llm now supports QLoRA finetuning on both Intel GPU and CPU.
• [2023/10] ipex-llm now supports FastChat serving on on both Intel CPU and GPU.
• [2023/09] ipex-llm now supports Intel GPU (including iGPU, Arc, Flex and MAX).
• [2023/09] ipex-llm tutorial is released.

ipex-llm Demo

See demos of running local LLMs on Intel Core Ultra iGPU, Intel Core Ultra NPU, single-card Arc GPU, or multi-card Arc GPUs using ipex-llm below.

ipex-llm Performance

See the Token Generation Speed on Intel Core Ultra and Intel Arc GPU below¹ (and refer to [2][3][4] for more details).

You may follow the Benchmarking Guide to run ipex-llm performance benchmark yourself.

Model Accuracy

Please see the Perplexity result below (tested on Wikitext dataset using the script here).

ipex-llm Quickstart

Use

• Ollama: running Ollama on Intel GPU without the need of manual installations
• llama.cpp: running llama.cpp on Intel GPU without the need of manual installations
• Arc B580: running ipex-llm on Intel Arc B580 GPU for Ollama, llama.cpp, PyTorch, HuggingFace, etc.
• NPU: running ipex-llm on Intel NPU in both Python/C++ or llama.cpp API.
• PyTorch/HuggingFace: running PyTorch, HuggingFace, LangChain, LlamaIndex, etc. (using Python interface of ipex-llm) on Intel GPU for Windows and Linux
• vLLM: running ipex-llm in vLLM on both Intel GPU and CPU
• FastChat: running ipex-llm in FastChat serving on on both Intel GPU and CPU
• Serving on multiple Intel GPUs: running ipex-llm serving on multiple Intel GPUs by leveraging DeepSpeed AutoTP and FastAPI
• Text-Generation-WebUI: running ipex-llm in oobabooga WebUI
• Axolotl: running ipex-llm in Axolotl for LLM finetuning
• Benchmarking: running (latency and throughput) benchmarks for ipex-llm on Intel CPU and GPU

Docker

• GPU Inference in C++: running llama.cpp, ollama, etc., with ipex-llm on Intel GPU
• GPU Inference in Python : running HuggingFace transformers, LangChain, LlamaIndex, ModelScope, etc. with ipex-llm on Intel GPU
• vLLM on GPU: running vLLM serving with ipex-llm on Intel GPU
• vLLM on CPU: running vLLM serving with ipex-llm on Intel CPU
• FastChat on GPU: running FastChat serving with ipex-llm on Intel GPU
• VSCode on GPU: running and developing ipex-llm applications in Python using VSCode on Intel GPU

Applications

• GraphRAG: running Microsoft's GraphRAG using local LLM with ipex-llm
• RAGFlow: running RAGFlow (an open-source RAG engine) with ipex-llm
• LangChain-Chatchat: running LangChain-Chatchat (Knowledge Base QA using RAG pipeline) with ipex-llm
• Coding copilot: running Continue (coding copilot in VSCode) with ipex-llm
• Open WebUI: running Open WebUI with ipex-llm
• PrivateGPT: running PrivateGPT to interact with documents with ipex-llm
• Dify platform: running ipex-llm in Dify(production-ready LLM app development platform)

Install

• Windows GPU: installing ipex-llm on Windows with Intel GPU
• Linux GPU: installing ipex-llm on Linux with Intel GPU
• For more details, please refer to the full installation guide

Code Examples

• Low bit inference

  • INT4 inference: INT4 LLM inference on Intel GPU and CPU
  • FP8/FP6/FP4 inference: FP8, FP6 and FP4 LLM inference on Intel GPU
  • INT8 inference: INT8 LLM inference on Intel GPU and CPU
  • INT2 inference: INT2 LLM inference (based on llama.cpp IQ2 mechanism) on Intel GPU

• FP16/BF16 inference

  • FP16 LLM inference on Intel GPU, with possible self-speculative decoding optimization
  • BF16 LLM inference on Intel CPU, with possible self-speculative decoding optimization

• Distributed inference

  • Pipeline Parallel inference on Intel GPU
  • DeepSpeed AutoTP inference on Intel GPU

• Save and load

  • Low-bit models: saving and loading ipex-llm low-bit models (INT4/FP4/FP6/INT8/FP8/FP16/etc.)
  • GGUF: directly loading GGUF models into ipex-llm
  • AWQ: directly loading AWQ models into ipex-llm
  • GPTQ: directly loading GPTQ models into ipex-llm

• Finetuning

  • LLM finetuning on Intel GPU, including LoRA, QLoRA, DPO, QA-LoRA and ReLoRA
  • QLoRA finetuning on Intel CPU

• Integration with community libraries

  • HuggingFace transformers
  • Standard PyTorch model
  • LangChain
  • LlamaIndex
  • DeepSpeed-AutoTP
  • Axolotl
  • HuggingFace PEFT
  • HuggingFace TRL
  • AutoGen
  • ModeScope
• Tutorials

API Doc

• HuggingFace Transformers-style API (Auto Classes)
• API for arbitrary PyTorch Model

FAQ

• FAQ & Trouble Shooting

Verified Models

Over 70 models have been optimized/verified on ipex-llm, including LLaMA/LLaMA2, Mistral, Mixtral, Gemma, LLaVA, Whisper, ChatGLM2/ChatGLM3, Baichuan/Baichuan2, Qwen/Qwen-1.5, InternLM and more; see the list below.

Get Support

• Please report a bug or raise a feature request by opening a Github Issue
• Please report a vulnerability by opening a draft GitHub Security Advisory

Footnotes

1. Performance varies by use, configuration and other factors. ipex-llm may not optimize to the same degree for non-Intel products. Learn more at www.Intel.com/PerformanceIndex. ↩ ↩²