DeepSeek open-source inference framework, some people have calculated that the gross profit margin of AI is 82%

I didn't expect this to be open-sourced either.

On the afternoon of October 20, DeepSeek open-sourced DeepInfer, the inference framework it uses internally, along with the supporting token codec component deepseek-recipe.

DeepInfer is the current foundation supporting DeepSeek's large-scale inference services. It is released under the Apache 2.0 license this time, and currently supports DeepSeek-V4.1-Flash.

This release quickly drew widespread attention. Yesterday, all public discussions centered on how DeepSeek now enables everyone to quickly deploy their own AI models. But when people rethink about it today, a question pops up: wait a minute, how come large model inference is so profitable?

DeepSeek disclosed that based on this system, the throughput of a single cluster serving per day reaches 155 trillion tokens, calculated at the off-peak API price, the overall revenue hits $38 per GPU per hour.

What does this figure mean? Some people have calculated that the gross profit margin reaches 82%: assuming the API you sell has a sufficiently high utilization rate, and you rent GPUs at a rate of $7 per B300 per hour, there is still a considerable profit margin after deducting only the rent cost.

Some people on Hacker News described it directly like this: a nuclear bomb. The cold hard numbers tell you that there is no bubble in the AI industry.

DeepInfer supplements important data for current discussions on AI service pricing, and for the first time lets us understand the engineering capabilities required to run models efficiently.

For ordinary users, calling a large model only means sending a request and waiting for a response. On the service side, it is necessary to arrange a large number of requests to share chips, read model parameters, mobilize computing resources distributed on different devices, and control queuing and generation latency.

You will find that even when the models are the same DeepSeek model, the official API and third-party deployed instances do not perform the same. This is because for the same model, different inference implementations may lead to different numbers of concurrent users that each GPU can serve, and different waiting times for users. Inference frameworks are designed to solve such problems, aiming to improve inference efficiency, and achieve the optimal level of effective throughput per unit of computing power and the response speed perceived by users.

No one is likely to do a better job in this area than DeepSeek itself.

DeepInfer is mainly oriented to large-scale deployment, supports NVIDIA SM100 GPUs and Huawei Ascend 950 NPUs, integrates optimized operators, distributed parallelism, speculative decoding, asynchronous scheduling and other capabilities, and also supports restricting output according to rules such as JSON Schema. DeepInfer was initially developed mainly based on vLLM, and also referenced SGLang and TensorRT-LLM.

At the same time, deepseek-recipe is responsible for converting API requests in different formats into prompt words and token sequences used by the model, and then restoring the output to responses in corresponding formats. It supports interface formats such as Chat Completions, Responses, and Messages, covering processing of thought content, tool calls, and streaming output.

This is of great significance for third-party deployment: developers can reuse the official request processing and codec logic, reducing the chance of behavior differences after the same set of applications is migrated to other servers. Whether a model can be smoothly connected to the existing Agent workflow often depends on these details.

The currently open-sourced repository provides core capabilities for offline inference and supporting large-scale services. Of course, the complete online service still needs supporting components such as KV Cache system, visual encoding, Prefill/Decode separated deployment, and cross-replica load balancing. Third parties have obtained an important engine, but system integration still needs to be completed during implementation.

The common goal of all these deployment efforts is to make each GPU do less redundant calculation and serve more requests.

According to the official deployment materials, DeepSeek's optimization ideas include organizing resources by inference stage, reusing context calculation results, and improving the effective output of each decoding.

When a large model processes a request, it first processes the input context (namely Prefill), and then generates responses step by step (namely Decode). Separating the deployment of the two stages can allocate resources respectively, avoiding different types of loads from crowding together. This helps to optimize both the waiting time for the first token and the subsequent generation speed at the same time.

Caching is especially suitable for long conversations and Agents. After an agent calls a tool, the next round of requests often carries the previous system prompts, tool descriptions and historical records again. If the intermediate calculation results of the existing context can be reused, the overhead of repeatedly processing the same content can be reduced.

A prominent point is the retention time of KV Cache. The official disclosed that Agent traffic is retained for 3 days, and chat traffic is retained for 7 days. These caches are stored in an independent 3FS storage cluster, and all Prefill machines can read them. By combining multi-day retention, shared storage and automatic matching, DeepSeek makes the long context that has been calculated continue to play a role in more subsequent requests.

Other companies including Anthropic, OpenAI and Google mostly set their cache retention time between 5 minutes and 24 hours.

Another example is DSpark speculative decoding: generate candidate tokens first, and then hand them over to the main model for verification. DeepSeek further arranges verification according to the expected acceptance rate of candidates, reducing the calculation spent on low-yield candidates. According to the official deployment Q&A estimation, near the listed production working points, dynamic length verification can increase the throughput by about 4-7% compared with the fixed verification of 5 candidate tokens.

When these optimizations add up, what changes is the amount of service that each GPU can deliver. Its selling price is low but it is not losing money at all. As long as the computing power is not idle, the more computing GPUs you buy, the more profit you make.

Regarding the discussion around token costs, according to the current official price list, the APIs of DeepSeek-V4.1-Flash and V4.1 Pro are as follows.

To compare service revenue and cost, we need to consider the composition of input, output and cache hit at the same time. After all, the selling price of cache-hit input is only one-fiftieth of that of cache-miss input. The good news is that DeepSeek itself has given the $38 figure.

In addition to the $7 for renting GPUs, even if other service expenses are added, there is still a considerable gap between API selling price and cost. This shows that efficient inference can achieve both low pricing and considerable profit margins. No wonder cloud service vendors are borrowing money to build data centers. On Hacker News, people have divergent discussions: some people calculate a higher profit margin based on lower GPU costs, while others emphasize that the investment in the entire life cycle of the model must be calculated.

However, the previous statement of "recovering all equipment costs within 10 months" now has a factual basis.

On the other hand, The Information disclosed last month that DeepSeek's annualized revenue reached about 1 billion US dollars, roughly equivalent to 2.74 million US dollars per day, which exactly corresponds to 3000 GPUs if compared with the revenue of $38 per GPU per hour.

DeepInfer provides a code starting point for optimized deployment. For most users, more sufficient competition in the inference field is expected to bring lower calling costs and better response experience.

I wonder in the future, will people "not waste Liang's time on training AGI", or will they wait for the launch of V4.1 Pro and crash DeepSeek's servers again?

References:

https://zhuanlan.zhihu.com/p/2095560295930508953

https://news.ycombinator.com/item?id=50163277

https://www.theinformation.com/articles/deepseeks-annualized-revenue-hits-1-billion-startup-finalizes-7-5-billion-fundraising

https://x.com/Yuchenj_UW/status/2112591074409742733

This article is from the WeChat official account "Machine Heart" (ID: almosthuman2014), author: Machine Heart focusing on DeepSeek, editor: ZENAN, published with authorization from 36Kr.