zhihu/ZhiLight is a C++ project with 908 stars in the AI Tools space. A highly optimized LLM inference acceleration engine for Llama and its variants.
Snapshot summary built from the project's own GitHub metadata — there's no written TopGit review yet. The page will update automatically when a full review is published.
WHY NO REVIEW YET
TopGit writes full reviews for the most-starred, most-requested repositories. This page is a snapshot until then — see the READ ME tab for the original README in full.
✨ ZhiLight ✨is a highly optimized LLM inference engine developed by Zhihu and ModelBest Inc. The "Zhi" in its name stands for Zhihu. ZhiLight can accelerate the inference of models like Llama and its variants, especially on PCIe-based GPUs. Compared to mainstream open-source inference engines, for example, vllm, it has significant performance advantages.
News 🔥
[2025/05] Support DeepSeek-V3/DeepSeek-R1 FP8 block-quantized model.
[2025/02] Support DeepSeek-V3/DeepSeek-R1 AWQ/GPTQ quantized model.
[2025/01] Support DeepSeek-VL2 multimodal models: deepseek-vl2-tiny, deepseek-vl2-small, deepseek-vl2.
[2025/01] Support LLaMA3 models: Llama-3.1-8B, Llama-3.1-70B, Llama-3.2-1B, Llama-3.2-3B, Llama-3.3-70B.
Roadmap
see Roadmap
🎉🎉 Main Features
Asynchronous OpenAI compatible interface adapted from vllm
Custom defined tensor and unified global memory management
🔥 Encode and all-reduce overlap, we named "dual streams"
Support Int8-quantized all-reduce to further reduce all-reduce cost.
Host all-reduce based on SIMD instructions
Optimized fused kernels, qkv, residual & layernorm etc.
🔥 Fused batch attention for decoding based on tensor core instructions
Support TP and PP on one node, TP is recommended
Support dynamic batch
Support flash attention prefill
Support chunked prefill
Support prefix cache
Support Native INT8/SmoothQuant/FP8/AWQ/GPTQ quantization
Support Marlin kernel for GPTQ
Support MoE, DeepseekV2 MoE and DeepseekV2 MLA
Support Llama/Llama2, Mixtral, Qwen2 series and similar models
🔧 Basic Usage
# Concurrently compile the wheel package, and turn off the unit test
CMAKE_BUILD_PARALLEL_LEVEL=32 TESTING=0 python setup.py bdist_wheel
# Compile with ninja backend
CMAKE_GENERATER="Ninja" python setup.py bdist_wheel
# Install directly
cd ./ZhiLight && pip install -e .
# Start OpenAI compatible server
python -m zhilight.server.openai.entrypoints.api_server [options]
✈️ Docker Image
ZhiLight only depends on the CUDA runtime, cuBLAS, NCCL, and a few Python packages in requirements.txt. You can use the image below for running or building it. You can also directly refer to docker/Dockerfile.
We conducted performance reviews on various mainstream NVIDIA GPUs with different model sizes and precisions. For dense models ranging from 2B to 110B parameters on PCIe devices, ZhiLight demonstrates significant performance advantages compared to mainstream open-source inference engines.
Test Description:
Test purpose is to demonstrate performance, applicable scenarios and limitations
Test metrics include:
QPS: Queries Per Second
TTFT (Time To First Token): First token generation latency
TPOT (Time Per Output Token): Generation latency per output token
Test environments include:
AD102 PCIe : Consumer-grade GPU for experimental research
The most recent commit recorded on zhihu/ZhiLight was 5 months ago, based on the GitHub push timestamp. The repository has 104 forks — one of the better signals of community interest.
How does zhihu/ZhiLight compare to other AI Tools projects?
zhihu/ZhiLight is tracked by TopGit in the AI Tools category, with 908 GitHub stars and written in C++. Browse the AI Tools topic page on TopGit to compare it against similar projects by stars and activity.
How many stars does zhihu/ZhiLight have?
zhihu/ZhiLight has 908 GitHub stars — refresh the page for the live number, or check github.com/zhihu/ZhiLight. TopGit mirrors GitHub's count but does not claim minute-by-minute accuracy.
Is zhihu/ZhiLight open source?
Yes — zhihu/ZhiLight ships under the Apache-2.0 license, which makes its source code freely readable (and, depending on license terms, forkable and reusable). Source: github.com/zhihu/ZhiLight.
What is zhihu/ZhiLight?
zhihu/ZhiLight (zhihu/ZhiLight) is a C++ project on GitHub. From the project's own README: A highly optimized LLM inference acceleration engine for Llama and its variants.
Where do I read more about zhihu/ZhiLight?
This TopGit page is a snapshot — the READ ME tab shows the project's own README content (links stripped, images preserved). The GitHub repository at github.com/zhihu/ZhiLight is the definitive source.
Read full README in the tab above.
Is ZhiLight worth your time?
ChatGPT, Claude and Perplexity can all read this page. Ask one of them what it makes of ZhiLight.
/ZhiLight