DwarfStar: Local DeepSeek V4 & GLM Inference
DwarfStar is a narrow, fast-moving inference engine built specifically to run DeepSeek V4 Flash, GLM 5.2, and DeepSeek V4 PRO on Metal, CUDA, and ROCm hardware you already own. Reach for it if you have a 128GB+ Mac, a DGX Spark, or a Strix Halo box and want these exact models running fast; skip it if you need a general GGUF loader for arbitrary models, since DwarfStar refuses those by design.
What is DwarfStar?
DwarfStar, a compact native inference engine coded in C, is primarily optimized for DeepSeek V4 Flash. It additionally offers support for GLM 5.2, and for DeepSeek V4 PRO when deployed on very high-memory machines. Compatibility extends to Metal, NVIDIA CUDA, and AMD ROCm platforms. Unlike a typical GGUF runner, its scope is intentionally limited: it exclusively loads the GGUF files for which it provides download scripts.
Key Features and Optimizations
- ✓Multi-backend support: Metal (Macs with 96GB or more RAM), NVIDIA CUDA (including multi-GPU systems and DGX Spark), and ROCm (Strix Halo systems like the Framework Desktop).
- ✓SSD streaming keeps non-routed weights resident while routed MoE experts are cached in memory and loaded from the GGUF file on cache misses, so RAM-constrained machines can still run the models.
- ✓Two-machine tensor parallelism over RDMA lets two MacBook M5 Max or M3 Ultra machines run 4-bit DeepSeek Flash or GLM 5.2 together.
- ✓Pipeline parallelism splits transformer layers across machines over TCP to sum RAM for models too large for one box, and can also speed up prefill by processing micro-batches on different layers concurrently.
- ✓DSpark speculative decoding, which accesses hidden states from the main model, proposes up to five future tokens that the main Flash model verifies, prior to committing only the accepted prefix.
- ✓Asymmetrical 2-bit quantization keeps routed MoE experts at IQ2_XXS/Q2_K while leaving shared experts, projections, and routing untouched, per the README.
- ✓ds4-server supports micro-batching of decoding and generation for multi-user serving on older CUDA cards.
Practical Applications of DwarfStar
- •Run DeepSeek V4 Flash or GLM 5.2 directly on a MacBook, DGX Spark, or Strix Halo box you already own, using SSD streaming when RAM is tight.
- •Turn a server with older Ada Lovelace CUDA cards — no longer supported for new models by vLLM, per the README — into a multi-user LLM server; the README reports a tested 8xL40S setup reaching 120 t/s aggregated generation and 2000 t/s prefill.
- •Using RDMA, two MacBook M5 Max or M3 Ultra machines can be pooled together to execute 4-bit DeepSeek Flash or GLM 5.2 via tensor parallelism.
- •Combine multiple machines' RAM with pipeline parallelism to run models that don't fit on a single system, such as the full 4-bit Flash quant split across two 128GB MacBooks.
Getting Started: Setup and Build
Build from source with make; there's no package-manager install path documented. Pick the target for your hardware — make for macOS Metal, make cuda-spark for a DGX Spark or GB10, make cuda-generic for other local CUDA GPUs, make strix-halo for AMD Strix Halo ROCm, or make cpu for a CPU-only diagnostics build. Then download a model with the bundled script, for example ./download_model.sh ds4f-q2 for 96/128GB RAM machines, which pulls from Hugging Face into ./gguf/ and points ./ds4flash.gguf at it. Run ./ds4 --help or ./ds4-server --help for the full flag list.
Running Models with DwarfStar
Point the CLI at a downloaded GGUF: ./ds4 -m ./ds4flash.gguf runs the default model, or pass -m to pick another file from ./gguf/. For DSpark speculative decoding on Flash 0731 models, download the support checkpoint with ./download_model.sh ds4f-dspark, then run ./ds4 -m ds4flash.gguf --mtp gguf/DeepSeek-V4-Flash-DSpark-support-0731.gguf --dspark --temp 0. On RAM-constrained machines, add --ssd-streaming, optionally with --ssd-streaming-cache-experts to set the routed-expert cache size explicitly. The ds4-server binary runs the same models behind an HTTP server for multi-user or programmatic access.
Performance and Speed Benchmarks
- ✓On a MacBook Pro M5 Max with 128GB RAM over Metal, the README's own benchmark shows 790.18 t/s prefill and 39.35 t/s generation at a 2048-token context, dropping to 398.50 t/s prefill and 27.64 t/s generation at 65536 tokens.
- ✓On a DGX Spark GB10 with 128GB RAM over CUDA, the same benchmark reports 825.76 t/s prefill and 18.05 t/s generation at 2048 tokens, still 822.98 t/s prefill at 65536 tokens.
- ✓The asymmetric 2-bit quantization is described in the README as verified to behave well, work under coding agents, and call tools reliably, rather than shipped untested.
- ✓SSD streaming turns 'not enough RAM' into 'slower, but working.' That's a real option, not a bragging point.
Current Status and Known Limitations
- △The project is explicitly beta: the README calls it 'very fast changing' and says instabilities are still possible even after a pre-release QA run.
- △It's genuinely not a general GGUF loader — arbitrary GGUF files won't have the tensor layout, quantization mix, or metadata DwarfStar expects, so you're locked into the specific DeepSeek V4 and GLM 5.2 builds it ships download scripts for.
- △DSpark speculative decoding is still experimental and opt-in; the README notes low-yield prompts can end up no faster, or even slower, than plain decoding.
- △Distributed generation across pipeline-parallel machines is slower than a single process, per the README, since each token must finish the full route before the next one starts — only prefill benefits from the extra machines.
- △The README discloses the codebase was built, assisted strongly by GPT 5.5, 5.6, and Claude Fable, and states plainly that this is a dealbreaker for anyone who doesn't want AI-developed code in their stack.
DwarfStar vs. General GGUF Loaders
Frequently Asked Questions
DwarfStar is optimized first for DeepSeek V4 Flash, also supports GLM 5.2, and can run DeepSeek V4 PRO on very high-memory machines. It's not a general GGUF loader, so only the specific GGUF builds it ships download scripts for will work correctly.
DwarfStar runs on Metal (Macs with 96GB or more RAM as the primary target), NVIDIA CUDA including multi-GPU setups and DGX Spark, and ROCm functions on Strix Halo systems, for example the Framework Desktop.
DwarfStar's SSD streaming mode handles this: it keeps non-routed weights resident while routed MoE experts are cached in memory and streamed from the GGUF file on cache misses, so smaller machines can still run larger models, just at reduced speed.
Not yet, by the project's own description — the README calls DwarfStar's current status 'beta quality' and 'very fast changing,' and says instabilities are still possible even after each release's QA run.
DwarfStar is released under the MIT license, per its GitHub repository.
The README for DwarfStar indicates that GPT 5.5, 5.6, and Claude Fable significantly aided its creation, while human input guided the concepts, testing, and debugging efforts. It also declares that individuals who dislike AI-generated code should not use this software.
The problem it solves
To run DeepSeek V4 Flash or GLM 5.2 on a local machine often requires modifying a generic GGUF runner, with the expectation that routed-MoE quantization, KV cache, and tool-calling path will all function harmoniously. DwarfStar circumvents that risk by creating dedicated systems for loading, for prompt rendering, for tool calls, for KV state, for the HTTP server, and for the coding agent. These are designed specifically for these two model families and are tested as a single unit, rather than a collection of disparate flags.
Who should try it — and who should skip
Try DwarfStar if you own a 128GB+ Mac, a DGX Spark, a Strix Halo desktop, or a multi-GPU CUDA box and specifically want to run DeepSeek V4 Flash, GLM 5.2, or DeepSeek V4 PRO fast on that hardware — the download scripts, quant choices, and speed benchmarks are all built around those exact combinations. Skip it if you want one tool for many different open models, need a stable production API today, or aren't comfortable running software the README says was built with heavy AI coding assistance.