DeepSeek's open source release is incredibly well-received, vLLM and SGLang have overtaken its performance overnight, pushing the competition for inference speed to a whole new level.

It has to be the legendary "Yuan Shen" Teacher D, what a kind act!

Its self-developed inference engine DeepInfer has been released for less than a week, and the open source community has submitted their own optimized implementations one after another.

It's truly the case that when a whale falls, all things thrive. The first batch of open source results have been freshly released:

Aspire to learn from DeepSeek, become DeepSeek, and surpass DeepSeek. (just kidding)

Mature inference frameworks including vLLM and SGLang have taken the lead to roll out their T0 level upgrade packages —

Under their respective publicly disclosed test configurations, their performance has approached or even partially exceeded DeepInfer.

Pretty impressive! DeepInfer was initially built on vLLM, and its design also drew inspiration from SGLang's ideas.

Now that DeepSeek has made its polished implementation public, the two teams have absorbed the relevant achievements and continued to make optimizations.

These teams have truly brought the open source ecosystem to a whole new level of prosperity.

There are also a number of personal developer projects inspired by DeepInfer going online one after another, covering scenarios ranging from hardware adaptation, inference acceleration to deployment optimization with diverse features.

Yet their performance improvements are surprisingly consistent: all of them have become far more competitive!

After DeepInfer was open sourced, the entire community has been fully invigorated.

The inference performance race has entered a white-hot stage, It’s showtime~

The Open Source Community Is Making Full Use of DeepInfer

Let's start with vLLM.

The latest results it delivered have gone far beyond "catching up with DeepInfer" and extended to full rack deployment scenarios.

By integrating DeepSeek's open source kernel and making a large number of optimizations for the prefill and decoding phases, the current performance curve of vLLM has reached the same level as DeepInfer.

Under the same decoding configuration (131K context, 3 draft tokens per round), the throughput of vLLM measured on GB200 is only 6% behind DeepInfer when the batch size ranges from 8 to 256.

For the prefill phase, the throughput of new tokens has also increased by 35.6% ~ 43.9%.

Then expand the deployment scale to the NVIDIA GB300 NVL72 rack.

With prefill-decoding separation and expanded expert parallelism, vLLM achieves a per-card throughput of up to 954,000 tokens/s in the SemiAnalysis AgentX test, which is 7.9 times the previous single-node result.

They have actually squeezed out further performance improvements from the same DeepSeek implementation...

Where do these improvements come from? The team has optimized almost every possible detail without missing anything.

For example, when processing inputs, sparse attention needs to filter out content worthy of attention from a batch of candidates, which involves top-k selection.

The previous workflow would first convert the BF16 scores output by DeepGEMM to FP32 before performing the filtering operation.

The new implementation lets DeepSelect process BF16 data directly, eliminating that intermediate conversion step.

Removing one computation step seems to be a minor change, but in the corresponding operator test, the top-k processing speed can reach about 3.7 times of the original level.

The vLLM team also developed a new CPU communication method, using recursive doubling and other methods to exchange information to reduce the number of synchronization rounds required.

For the small-batch decoding scenarios commonly used by Agents, vLLM also provides dedicated support for DSpark draft generation, and uses the full CUDA Graph to execute the dedicated decoding path.

The answer submitted immediately afterwards by SGLang goes even further, covering both the input and output phases.

SGLang tested prefill performance on 4 B300 GPUs and 4 GB300 GPUs respectively, and its maximum throughput is about 4% higher than DeepInfer.

During the decoding test, 64 GB300 GPUs were also used for the test.

When there are 256 requests per GPU, DeepInfer takes 31.19ms per step while SGLang takes 28.88ms, reducing the per-step latency by about 7.4%. When there are 8 requests per GPU, the performance of the two is basically at the same level.

In specific implementations, SGLang first integrated all Mega kernels (MegaMoE, MegaAttention, Mega-mHC, MegaGate), SWA bounded replay and Engram optimizations.

After achieving performance comparable to DeepInfer, the team continued to iterate and introduced more kernel fusion solutions.

Then it achieved highly efficient prefill by supporting splittable CUDA graphs.

Compared with the open source version of DeepInfer, SGLang is more complete, with additional key service components including the KV cache system, prefill-decoding separation architecture and load balancing module.

It is worth mentioning that for a more rigorous comparison, the SGLang team re-ran DeepInfer on GB300 and B300 GPUs.

The result was unexpected: the throughput of DeepInfer under some configurations is even higher than the previously published results...

Wow! It turns out that Teacher D is really a straight shooter, whose published test results are completely down-to-earth without exaggeration!

The Inference Acceleration Positive Feedback Loop

Developer Eric Zhang has taken the related exploration one step further.

He has long been developing an experimental building block library for building large language model inference engines, namely Resin.

It turns the components shared by multiple models into a public base, such as the scheduler and KV cache management module. Users can add the computation implementations required for specific models and specific hardware by themselves, so as to assemble an independent inference engine.

According to Eric's own account, inspired by DeepInfer in the past week, he has successfully made many functions and parallel mechanisms run properly.

According to the test results, on DeepSeek-V4.1-Flash, the performance of Resin is about 5%~10% behind DeepInfer, and it also supports functions including HTTP service, prefix cache and performance metrics.

Then he added a dedicated Skill to this base framework.

The task is to teach AI how to implement models based on Resin, then run Codex on top of it for overnight development.

As he disclosed, the Agent completed the GLM-5.3-Flash engine with MTP support. Under the configuration of B200 and EP4, the maximum input throughput reaches 60,000 tokens/s, and the maximum output throughput reaches 16,000 tokens/s.

What's more, Resin is also open sourced.

It's so touching. Dear "Big Fat Fish" (DeepSeek), this prosperous open source ecosystem is exactly what you wished for.

DeepInfer comes from the open source community and finally returns to the open source ecosystem. Mature frameworks continue to integrate and optimize related achievements, while individual developers can also absorb its excellent designs and continue to contribute to the open source community.

So, the "Big Fat Fish" (DeepSeek) is the one that truly understands the essence of sustainable open source~

This article is from the WeChat public account "QbitAI", the author focuses on cutting-edge technology, and it is published with authorization from 36Kr.