DwarfStar: Local DeepSeek V4 & GLM Inference
/ds4
DwarfStar is a narrow, fast-moving inference engine built specifically to run DeepSeek V4 Flash, GLM 5.2, and DeepSeek V4 PRO on Metal, CUDA, and ROCm hardware you already own. Reach for it if you have a 128GB+ Mac, a DGX Spark, or a Strix Halo box and want these exact models running fast; skip it if you need a general GGUF loader for arbitrary models, since DwarfStar refuses those by design.
What is DwarfStar?
Written in C, DwarfStar is a small native inference engine. It's optimized first for DeepSeek V4 Flash, with support extended to GLM 5.2, and DeepSeek V4 PRO for very high-memory machines. The engine operates using Metal, NVIDIA CUDA, and AMD ROCm. In contrast to a general GGUF runner, its design is intentionally constrained: it exclusively loads GGUF files for which it provides download scripts.
Key Features and Optimizations
- ✓Multi-backend support: Metal (Macs with 96GB or more RAM), NVIDIA CUDA (including multi-GPU systems and DGX Spark), and ROCm (Strix Halo systems like the Framework Desktop).
- ✓SSD streaming keeps non-routed weights resident while routed MoE experts are cached in memory and loaded from the GGUF file on cache misses, so RAM-constrained machines can still run the models.
- ✓Two-machine tensor parallelism over RDMA lets two MacBook M5 Max or M3 Ultra machines run 4-bit DeepSeek Flash or GLM 5.2 together.
- ✓Pipeline parallelism splits transformer layers across machines over TCP to sum RAM for models too large for one box, and can also speed up prefill by processing micro-batches on different layers concurrently.
- ✓DSpark speculative decoding, by reading hidden states from the main model, puts forward up to five future tokens. The main Flash model then verifies these, committing only the accepted prefix.
- ✓Asymmetrical 2-bit quantization keeps routed MoE experts at IQ2_XXS/Q2_K while leaving shared experts, projections, and routing untouched, per the README.
- ✓ds4-server supports micro-batching of decoding and generation for multi-user serving on older CUDA cards.
Practical Applications of DwarfStar
- •Run DeepSeek V4 Flash or GLM 5.2 directly on a MacBook, DGX Spark, or Strix Halo box you already own, using SSD streaming when RAM is tight.
- •Turn a server with older Ada Lovelace CUDA cards — no longer supported for new models by vLLM, per the README — into a multi-user LLM server; the README reports a tested 8xL40S setup reaching 120 t/s aggregated generation and 2000 t/s prefill.
- •For the purpose of executing 4-bit DeepSeek Flash or GLM 5.2, employing tensor parallelism, two MacBook M5 Max or M3 Ultra machines can be pooled via RDMA.
- •Combine multiple machines' RAM with pipeline parallelism to run models that don't fit on a single system, such as the full 4-bit Flash quant split across two 128GB MacBooks.
Getting Started: Setup and Build
Build from source with make; there's no package-manager install path documented. Pick the target for your hardware — make for macOS Metal, make cuda-spark for a DGX Spark or GB10, make cuda-generic for other local CUDA GPUs, make strix-halo for AMD Strix Halo ROCm, or make cpu for a CPU-only diagnostics build. Then download a model with the bundled script, for example ./download_model.sh ds4f-q2 for 96/128GB RAM machines, which pulls from Hugging Face into ./gguf/ and points ./ds4flash.gguf at it. Run ./ds4 --help or ./ds4-server --help for the full flag list.
Running Models with DwarfStar
Point the CLI at a downloaded GGUF: ./ds4 -m ./ds4flash.gguf runs the default model, or pass -m to pick another file from ./gguf/. For DSpark speculative decoding on Flash 0731 models, download the support checkpoint with ./download_model.sh ds4f-dspark, then run ./ds4 -m ds4flash.gguf --mtp gguf/DeepSeek-V4-Flash-DSpark-support-0731.gguf --dspark --temp 0. On RAM-constrained machines, add --ssd-streaming, optionally with --ssd-streaming-cache-experts to set the routed-expert cache size explicitly. The ds4-server binary runs the same models behind an HTTP server for multi-user or programmatic access.
Performance and Speed Benchmarks
- ✓On a MacBook Pro M5 Max with 128GB RAM over Metal, the README's own benchmark shows 790.18 t/s prefill and 39.35 t/s generation at a 2048-token context, dropping to 398.50 t/s prefill and 27.64 t/s generation at 65536 tokens.
- ✓On a DGX Spark GB10 with 128GB RAM over CUDA, the same benchmark reports 825.76 t/s prefill and 18.05 t/s generation at 2048 tokens, still 822.98 t/s prefill at 65536 tokens.
- ✓The asymmetric 2-bit quantization is described in the README as verified to behave well, work under coding agents, and call tools reliably, rather than shipped untested.
- ✓SSD streaming turns 'not enough RAM' into 'slower, but working.' That's a real option, not a bragging point.
Current Status and Known Limitations
- △The project is explicitly beta: the README calls it 'very fast changing' and says instabilities are still possible even after a pre-release QA run.
- △It's genuinely not a general GGUF loader — arbitrary GGUF files won't have the tensor layout, quantization mix, or metadata DwarfStar expects, so you're locked into the specific DeepSeek V4 and GLM 5.2 builds it ships download scripts for.
- △DSpark speculative decoding is still experimental and opt-in; the README notes low-yield prompts can end up no faster, or even slower, than plain decoding.
- △Distributed generation across pipeline-parallel machines is slower than a single process, per the README, since each token must finish the full route before the next one starts — only prefill benefits from the extra machines.
- △According to the README, significant aid in building the codebase came from GPT 5.5, 5.6, and Claude Fable. It also makes it clear that such AI-developed code presents an unacceptable condition for those not wishing to include it in their technical infrastructure.
DwarfStar vs. General GGUF Loaders
Frequently Asked Questions
DwarfStar is optimized first for DeepSeek V4 Flash, also supports GLM 5.2, and can run DeepSeek V4 PRO on very high-memory machines. It's not a general GGUF loader, so only the specific GGUF builds it ships download scripts for will work correctly.
DwarfStar operates using Metal (targeting Macs having 96GB or more RAM primarily), NVIDIA CUDA, which supports multi-GPU setups and DGX Spark, and ROCm functions on Strix Halo systems, for instance the Framework Desktop.
DwarfStar's SSD streaming mode handles this: it keeps non-routed weights resident while routed MoE experts are cached in memory and streamed from the GGUF file on cache misses, so smaller machines can still run larger models, just at reduced speed.
Not yet, by the project's own description — the README calls DwarfStar's current status 'beta quality' and 'very fast changing,' and says instabilities are still possible even after each release's QA run.
DwarfStar is released under the MIT license, per its GitHub repository.
DwarfStar's README discloses that GPT 5.5, 5.6, and Claude Fable provided strong developmental assistance, with humans leading the ideas, testing, and debugging, and states this software isn't for anyone unhappy with AI-developed code.
The problem it solves
Running DeepSeek V4 Flash or GLM 5.2 locally usually means adapting a general-purpose GGUF runner. One then hopes the routed-MoE quantization, KV cache, with tool-calling path all behave together. DwarfStar avoids such a gamble by developing loading, prompt display, tool invocation, KV state management, its HTTP server, along with the coding agent, all specifically for these two model families. These are tested as a single unit rather than a patchwork of flags.
Who should try it — and who should skip
Try DwarfStar if you own a 128GB+ Mac, a DGX Spark, a Strix Halo desktop, or a multi-GPU CUDA box and specifically want to run DeepSeek V4 Flash, GLM 5.2, or DeepSeek V4 PRO fast on that hardware — the download scripts, quant choices, and speed benchmarks are all built around those exact combinations. Skip it if you want one tool for many different open models, need a stable production API today, or aren't comfortable running software the README says was built with heavy AI coding assistance.
Related repositories
Want a second opinion on ds4?
Ask an AI that can read this page — one click and you get its take on ds4.