Intel Neural Compressor: LLM Quantization Toolkit
Intel Neural Compressor is Intel's open-source Python library for quantizing and compressing deep learning models on PyTorch, TensorFlow, and JAX. The README covers quantization, SmoothQuant, and mixed precision, plus newer FP8, MXFP4, and NVFP4 support, much still experimental. Reach for it when deploying on Intel Gaudi or Xeon; skip it if NVIDIA GPU is your target, since that path gets only limited testing per the README.
What is Intel Neural Compressor?
Intel Neural Compressor is an open-source Python library from Intel that applies compression techniques — quantization, mixed precision, and related methods — to models in PyTorch, TensorFlow, or JAX. It integrates with the AutoRound project to quantize LLMs and VLMs such as LLaMA, Qwen, DeepSeek, and Flux, and treats Intel's own accelerators and CPUs as its primary hardware target, with AMD, ARM, and NVIDIA marked limited-testing.
Core Compression Techniques and LLM Support
- ✓Six named techniques anchor the library: weight-only quantization, SmoothQuant, quantization-aware training, mixed precision, and both static and dynamic quantization, per the README's own description of what it provides.
- ✓Low-bit format coverage down to INT4, MXFP4, and NVFP4, alongside INT8, FP8, and MXFP8 — six formats named together in the repo's GitHub description.
- ✓LLM and VLM quantization runs through a separate integration, AutoRound (intel/auto-round), with the README naming LLaMA, Qwen, DeepSeek, Flux, and FramePack as models it covers.
- ✓Three framework-specific PyPI packages — neural-compressor-jax, neural-compressor-tf, and neural-compressor-pt — each pair the extension API with that framework's own dependency.
- ✓An Auto Tune module sits under 'Other Modules' in the docs tree, separate from the named quantization technique pages.
- ✓The changelog logs frequent, dated additions: FP8 support for Keras/JAX and FP8 KV-cache/attention static quantization via AutoRound both landed in 2026/03, still marked experimental.
- ✓GitHub's topic labels extend the picture beyond the README body text — pruning, sparsity, knowledge-distillation, post-training-quantization, sparsegpt, awq, and gptq all appear as topics on the repo.
Optimizing Models for Diverse Hardware
- •Quantizing an LLM or VLM — the README names LLaMA, Qwen, DeepSeek, Flux, and FramePack — for lower-precision inference before deploying it on Intel accelerator hardware.
- •Running FP8 or weight-only quantization specifically on Intel Gaudi2, following the README's own worked example built around Habana's docker image.
- •Targeting production inference on Xeon Scalable, Xeon CPU Max, Core Ultra, or Data Center GPU Flex/Max hardware, where the README says testing is most extensive.
- •Experimenting with newer low-bit formats — NVFP4, MXFP8, MXFP4 — that the changelog marks experimental, for teams comfortable with pre-production tooling.
- •Moving a compression workflow across frameworks without switching tools, since PyTorch, TensorFlow, and JAX models all go through the same library.
Installation and Framework Integration
Installation is framework-specific and happens in two steps per the README. First comes the deep learning framework itself: for PyTorch, that means intel_extension_for_pytorch — separate builds exist for CPU and Intel GPU — or, for Gaudi/HPU, a Habana docker image with PyTorch already installed; other platforms use a standard PyTorch install. Second, pip install the matching Neural Compressor package: neural-compressor-pt for PyTorch, neural-compressor-tf for TensorFlow, or neural-compressor-jax for JAX — each bundles the framework extension API together with that framework's dependency. The README points to a separate Installation Guide and FAQ document for deeper install scenarios rather than covering every case inline.
First Steps with Quantization
The README's own quickstart is a Gaudi2 FP8 example: import FP8Config, prepare, and convert from neural_compressor.torch.quantization, wrap a torchvision ResNet-18 with FP8Config(fp8_config="E4M3") through prepare(), run a calibration forward pass, then call convert() to get the quantized model back. A second example covers weight-only LLM loading — load() with format="huggingface" and device="hpu" pulls a GPTQ-format checkpoint (the README uses TheBloke/Llama-2-7B-GPTQ) and converts it to HPU format on first load, caching the result as hpu_model.safetensors so later loads are faster. Both examples assume a Habana Gaudi container with the habana runtime enabled, and the README recommends setting PT_HPU_LAZY_MODE=1, since most low-precision functions don't yet support the newer lazy-mode-off default.
Strengths
- ✓A genuinely wide technique catalog in one library — static and dynamic quantization, SmoothQuant, weight-only quantization, quantization-aware training, mixed precision, plus low-bit formats down to INT4, MXFP4, and NVFP4 — instead of assembling separate tools per method.
- ✓Built and maintained by the hardware vendor itself, so Gaudi, Xeon, and Core Ultra support isn't a community best-effort add-on.
- ✓LLM and VLM quantization is a first-class path through the AutoRound integration, with named model coverage (LLaMA, Qwen, DeepSeek, Flux, FramePack) instead of a vague 'should work' claim.
- ✓Framework coverage across PyTorch, TensorFlow, and JAX means one library can follow a model through more than one training stack.
- ✓Apache-2.0 licensing, plus a published arXiv paper on FP8 inference on Gaudi backing the technique choices rather than just marketing copy.
Current Scope and Experimental Features
- △Several of the newest capabilities — FP8 for Keras/JAX, FP8 KV-cache/attention quantization via AutoRound, NVFP4, and MXFP8/MXFP4 — are labeled experimental in the README's own changelog, not production-ready.
- △Extensive hardware testing is scoped to Intel's own accelerators and CPUs; the README calls AMD CPU, ARM CPU, and NVIDIA GPU support 'limited testing' by comparison.
- △Both flagship quickstart examples in the README — FP8 quantization and weight-only LLM loading — target Intel Gaudi2 through a Habana docker image, so replicating that exact walkthrough elsewhere means piecing docs together instead of following one path end to end.
- △Install scenarios beyond the three PyPI packages aren't documented in the README itself — it defers to a separate Installation Guide and FAQ page.
- △There's a version-compatibility coupling worth tracking: the README notes a version-mapping table between Neural Compressor and the Gaudi software stack, so upgrading one without checking the other risks a mismatch.
Exploring Other Model Compression Tools
Common Questions About Neural Compressor
Intel Neural Compressor supports PyTorch, TensorFlow, and JAX per the README, with separate PyPI packages — neural-compressor-pt, neural-compressor-tf, neural-compressor-jax — for each. The repo's GitHub description also lists ONNX Runtime among its target frameworks.
Intel Neural Compressor covers static and dynamic quantization, SmoothQuant, weight-only quantization, quantization-aware training, and mixed precision, plus low-bit formats down to INT4, MXFP4, and NVFP4, according to the README and repo description.
Intel Neural Compressor gets its most extensive testing on Intel Gaudi accelerators, Core Ultra processors, Xeon Scalable and Xeon CPU Max series, and Data Center GPU Flex/Max — the README calls AMD CPU, ARM CPU, and NVIDIA GPU support 'limited testing' by comparison.
Intel Neural Compressor is built for LLM and VLM quantization, integrating with Intel's AutoRound project to compress models the README names directly — LLaMA, Qwen, DeepSeek, Flux, and FramePack — across multiple low-precision formats.
Intel Neural Compressor is released under the Apache-2.0 license, per its GitHub repository.
Intel Neural Compressor's README FP8 example imports FP8Config, prepare, and convert from neural_compressor.torch.quantization, wraps a model with FP8Config(fp8_config="E4M3") via prepare(), runs a calibration pass, then calls convert() — demonstrated on Intel Gaudi2 in the README.
The problem it solves
Model compression tooling tends to be scattered by framework and by hardware target — a PyTorch quantization recipe tuned for an NVIDIA GPU rarely transfers cleanly to an Intel Gaudi accelerator or a TensorFlow deployment. Intel Neural Compressor's README frames its reason for existing around that fragmentation: one library covering static and dynamic quantization, SmoothQuant, weight-only quantization, quantization-aware training, mixed precision, and now several low-bit LLM formats, built specifically for Intel's own accelerator and CPU lineup rather than being hardware-agnostic in name only.
Who should try it — and who should skip
Reach for Intel Neural Compressor if you're deploying deep learning models on Intel hardware — Gaudi accelerators, Xeon Scalable or Xeon CPU Max, Core Ultra, or Data Center GPU Flex/Max — and want quantization or mixed-precision tooling maintained by the chip vendor, especially for LLM and VLM workloads through the AutoRound integration. Skip it if NVIDIA GPU is your primary deployment target, since the README calls that path only limited-tested, or if you just need basic 8-bit quantization inside an existing Transformers pipeline — bitsandbytes or Transformers' own quantization configs are a lighter lift.
Related repositories
Curious whether neural-compressor is right for you?
Let ChatGPT, Claude, or Perplexity look into it — click below and see what AI actually says about neural-compressor.
