Coding & agents New
DeepSeek open-sources DeepInfer, the inference software behind its own production service
Published:
DeepSeek published DeepInfer on GitHub under the Apache 2.0 licence and announced it on 20 October. The repository describes it as the high-performance inference framework that powers DeepSeek's large-scale production services. Inference software runs a finished model on chips and turns requests into answers, so it largely decides how fast and how cheaply a model can be served.
This release supports one model, DeepSeek-V4.1-Flash, on Nvidia SM100 GPUs and Huawei Ascend 950 NPUs. Listed features include optimised kernels such as FlashMLA, DeepGEMM and DeepEP, attention and mixture-of-experts parallelism, draft-token decoding with confidence-scheduled verification, batch-invariant inference, deterministic sampling and structured outputs. The batch-invariant and deterministic modes work on Nvidia hardware only.
The public code covers offline inference. DeepSeek says a complete online service also needs a KV cache system, prefill and decode disaggregation and load balancing across data-parallel replicas, and none of these is included. The README says DeepInfer was originally built primarily on vLLM, with additional inspiration from SGLang and TensorRT-LLM.
Caveat. No outside party has yet benchmarked DeepInfer's efficiency, Quartz reports, citing Bloomberg. Throughput and revenue figures circulating in the Hacker News thread were quoted from a third-party post and were not verified for this edition.
Sources