# DeepSeek DeepInfer: Hardware Requirements and What the Open Release Includes DeepSeek has opened the inference framework behind V4.1 Flash, but the first release leaves important serving infrastructure for operators to build. By Paras Madan | 2026-10-20 DeepSeek released DeepInfer on October 20, opening the inference framework behind its V4.1 Flash production services. The code gives infrastructure teams a closer look at how DeepSeek runs its models, although the public release provides offline inference rather than a complete hosted API service. That boundary is the first thing I would check before planning a deployment. Open weights let you obtain a model. An inference engine determines how efficiently that model runs, while a reliable service also needs caching, request routing and operational controls around it. ## What DeepInfer includes DeepSeek's [announcement](https://x.com/deepseek_ai/status/2112476496283775135) describes the framework as its in-house production stack. The [repository](https://github.com/deepseek-ai/DeepInfer) makes the release scope more precise: it supports DeepSeek-V4.1-Flash, with the October 17 checkpoint used by the default offline example. DeepInfer combines optimized kernels, distributed execution, speculative decoding, asynchronous scheduling and structured outputs. The code is available under Apache 2.0. DeepSeek acknowledges that the framework was originally built primarily on vLLM, with additional inspiration from SGLang and TensorRT-LLM. **The useful change is access to the implementation choices behind a specific production model.** Developers can examine how those choices fit together instead of trying to infer the whole serving design from model weights and API behavior. ## Hardware and software requirements The supported hardware list is narrow: NVIDIA SM100 GPUs and Huawei Ascend 950 NPUs. The documented setup requires Linux, GCC 13 or newer and Python 3.12, with CUDA 13 for NVIDIA or CANN 9.2 for Ascend. This limits who can run the framework directly today. A developer with a consumer GPU should not read the release as a new desktop inference option. Other teams may still find individual techniques useful, but adapting them to different hardware would be separate engineering work. The published performance setup also deserves attention. DeepSeek measures prefill on four SM100 GPUs and decode on 64, each within a single scale-up domain. Those measurements should not be presented as expected performance on an arbitrary cluster, or as a universal cost advantage over another engine. ## Where the inference optimizations help Several features address different parts of the same request. DSpark uses draft tokens and confidence-scheduled verification to accelerate decoding. Asynchronous scheduling overlaps CPU batch preparation with device execution, reducing the time hardware spends waiting for the next batch. Sliding-window-attention bounded replay reduces decoder prefill work by retaining and replaying a limited suffix. The retained portion must still cover the attention window and the permitted backtracking length. This is an architecture-specific optimization, so copying the setting into another model would miss the reason it works. For testing and debugging, DeepInfer also documents batch-invariant inference and deterministic sampling on NVIDIA. These modes have configuration requirements, including disabling variable-length speculative decoding for deterministic sampling. Teams should verify those settings before expecting identical outputs across repeated runs. ## What is missing from a full service The repository explicitly excludes a complete KV-cache system, the deployment of separate vision-encoding, prefill and decode stages, and load balancing across data-parallel replicas. It notes that most engine functionality needed for that disaggregation is already implemented. **Running an offline example does not establish that a production endpoint is ready.** An operator still needs to decide how requests enter the system, where cached state lives, how failures are handled and how work is distributed under changing traffic. This is particularly important when estimating savings. Faster execution can improve unit economics, but engineering time, utilization and reliability also affect the cost of serving a useful response. The release gives teams more material to measure; it does not settle their deployment economics for them. ## How deepseek-recipe fits alongside it The accompanying [deepseek-recipe](https://github.com/deepseek-ai/deepseek-recipe) project handles another boundary: converting API requests into DeepSeek conversations, prompts and token IDs, then parsing generated output back into responses. Its Rust libraries and Python bindings support Messages, Chat Completions and Responses formats. Recipe still leaves model inference, tool execution and HTTP transport to the caller. Its example servers use mock inference. Treating protocol compatibility as a working model backend would therefore be a mistake. For teams building their own deployment, these two releases make different layers inspectable. DeepInfer exposes execution; recipe exposes formatting and parsing. The next meaningful test is whether an operator can connect those layers, reproduce correct behavior and sustain the service quality their workload requires. I'm Paras. I work at the intersection of AI, engineering and distribution. If you're building an AI product, I help turn technical releases and product decisions into clear stories developers and founders read and share. [Book a 30-minute call](https://cal.com/paras-madan/30min).