DeepSeek has fully open-sourced its inference framework, which is compatible with NVIDIA and Huawei Ascend chips, supports a daily throughput of 155 trillion tokens on a single cluster, and generates a revenue of 38 USD per card per hour.

DeepSeek has fully open-sourced its inference framework DeepInfer and the Token codec component deepseek-recipe, which are compatible with both NVIDIA and Huawei Ascend platforms. Based on this framework, the daily throughput of a single DeepSeek V4.1 Flash cluster reaches 155 trillion Tokens, with revenue of 38 US dollars per card per hour calculated at off-peak electricity prices. Through technologies such as Prefill-Decode separation, KVCache reuse and speculative decoding, DeepSeek aims to improve inference efficiency and reduce computing power costs and energy consumption.

DeepSeek further opens up the large model inference infrastructure.

Following the open-sourcing of large models, the TileLang compilation tool and a series of inference operator libraries, DeepSeek announced the full open-sourcing of its internal inference framework DeepInfer, and simultaneously open-sourced the Token codec component deepseek-recipe, aiming to help developers and enterprises optimize large-scale model deployment and reduce the computing power cost and energy consumption of AI services.

DeepSeek introduced that based on DeepInfer, the team has implemented a high-throughput, low-latency large-scale inference service for DeepSeek V4.1 Flash. At present, the service throughput of a single cluster reaches 155 trillion Tokens per day, and the revenue per card per hour is 38 US dollars calculated at off-peak electricity prices.

The inference framework is the key infrastructure connecting the capabilities of large models and end users. With the growth of call volume of AI applications, the cost of inference services not only depends on the model itself, but is also affected by links such as computing, communication, task scheduling and cache management. How to improve the effective throughput while meeting the latency requirements has become an important link for cost reduction and efficiency improvement of large-scale AI services.

Compatible with NVIDIA and Huawei Ascend, optimizing computing power configuration for different inference stages

DeepInfer integrates high-performance operators and a deeply optimized inference engine, supports distributed inference, speculative decoding, SWA Bounded Replay, structured output and other capabilities, and can run on NVIDIA and Huawei Ascend platforms. In the future, the inference code of DeepSeek's new models and subsequent performance optimizations will continue to be open-sourced in this framework.

Facing the large-scale AI call demand, DeepSeek focuses the optimization of the inference system on throughput and latency. The system adopts separated deployment of Prefill and Decode: the former is responsible for processing input prompts, and the latter is responsible for generating Tokens step by step. Since the computing loads of the two stages are different, separate deployment can independently configure resources to optimize the first Token generation time and the subsequent Token generation interval respectively.

In terms of hardware organization, DeepSeek uses common 8-card machines to undertake Prefill tasks, and adopts 64-to-128-card supernodes to undertake Decode tasks. At the same time, the system dynamically adjusts the instance ratio of the two stages through automatic scaling to reduce resource mismatch caused by traffic changes; in MoE model inference, it reduces the load difference between different computing cards through expert parallelism and dynamic load balancing.

KVCache can be retained for up to 7 days to reduce redundant computing overhead

Cache reuse is another key design for DeepSeek to reduce inference costs. With the increase of long-context dialogue and AI Agent applications, the same prefix may appear repeatedly in different requests. If these contents are reprocessed every time, additional computing overhead will be generated.

DeepSeek V4.1 Flash divides KVCache into two parts: Block KVCache and SWA KVCache, and adopts different storage strategies. Among them, Block KVCache is stored in the independent 3FS distributed file system, with a related storage capacity of about 100 PiB. According to the data disclosed by DeepSeek, the KVCache retention time for Agent traffic reaches 3 days, and that for chatbot traffic reaches 7 days.

When subsequent requests hit the existing cache, the system can reuse the previous calculation results to reduce redundant Prefill calculation. DeepSeek disclosed that the peak aggregated read bandwidth of the 3FS storage cluster reaches 10 TiB/s, and the peak aggregated write bandwidth is about 850 GiB/s, to meet the large-scale cache read and write requirements.

For SWA KVCache, which is mainly reused in the short term within active sessions, DeepSeek chooses to keep it in the machine memory without persistent storage. Online data shows that the retention time of this part of the cache is about 180 seconds, and the hit rate exceeds 98%.

For the case of SWA KVCache miss, DeepSeek adopts the SWA Bounded Replay mechanism to recalculate only the latest 128 Tokens, and limits the attention window to the recalculation interval to reduce the additional computing cost caused by cache miss.

Speculative decoding improves throughput, the codec component is open-sourced simultaneously

In addition to cache reuse, DeepSeek also improves inference efficiency through speculative decoding and asynchronous scheduling.

DeepSeek adopts the DSpark speculative decoding framework, which dynamically determines the verification range according to the expected acceptance probability of candidate Tokens, so as to avoid wasting verification computing power on candidate results with low acceptance probability. DeepSeek disclosed that in online deployment, compared with the method of verifying 5 Tokens fixedly, variable-length verification increases throughput by 6%.

In the scheduling link, through the asynchronous cooperation between CPU and GPU, the system makes operations such as request scheduling, forward calculation and sampling overlap as much as possible to reduce device waiting time. In addition, DeepInfer supports structured output, which can control the model generation results according to constraints such as JSON Schema and regular expressions to meet the output format requirements of specific applications.

DeepSeek also simultaneously open-sourced the production-grade Token codec component deepseek-recipe. This component supports mainstream interfaces such as Chat Completions API, Responses API and Messages API, and improves the processing efficiency of long texts through mechanisms such as special Token boundary recognition, parallel encoding of ordinary text segments and streaming output parsing, ensuring that the API processing logic of third-party deployment is consistent with the official service.

However, DeepSeek has not opened all components of the inference system. Since the KVCache management and load balancing parts are deeply coupled with the internal infrastructure, the relevant code has not been fully open-sourced at present.

From models, operator libraries to large-scale inference frameworks, DeepSeek has further expanded the coverage of the open-source technology stack this time. For developers and enterprises, this is expected to lower the technical threshold for building private and public cloud AI services; for the large model industry, how to improve inference throughput, reuse calculation results and adapt to different hardware platforms is also becoming an important direction to control the cost of AI services.

This article is from "Wallstreetcn", author: Bu Shuqing, published with authorization from 36Kr.