Kimi K3 has officially announced the Burning Formula, and is it really a living Bodhisattva now?

I announce that Kimi is officially on a par with DeepSeek, becoming the real living Bodhisattva in the open source community.

Because last night (27th), Kimi K3 was officially open-sourced on Hugging Face.

It gained 4000 likes on Hugging Face in only half an hour, which can be regarded as the most widely concerned open source model ever. It is estimated that a large number of foreign users, just like the Bad Reviewer, were waiting for its release at the exact time.

What Kimi open-sourced this time is no simple content. Apart from the model weight files themselves, it also comes with a detailed technical report, and a large number of model training tools are released together.

Just like DeepSeek now, it has truly presented the full set of core technologies to all tech enthusiasts.

We will not go into details about various benchmark tests. Basically, its scores in almost all benchmarks are only lower than Fable 5 and GPT-5.6 Sol.

It can be said that the current K3 is the third most powerful model in the world, and it is the only one among the top 3 that you can download to local devices for self-operation and fine-tuning.

Besides, it has built-in multimodal functions with no obvious shortcomings.

The model parameters are also larger than ever. The total parameters of K3 directly reach 2.8T, making it the largest open source model in existence.

Not only the total parameter volume is huge, but the activated parameters of the model are also astonishingly large,

In actual operation, K3 can directly activate 104.2 billion parameters, which is three times that of its previous generation and twice that of DeepSeek V4 Pro, even larger than the full body of many existing open source models.

In order to handle this exaggerated amount of data, K3 has made a lot of architectural innovations this time. Of course, the team did not keep these technologies private, and all of them are clearly written in the public documents.

The first is the attention mechanism that everyone pays close attention to. As we all know, in the traditional attention mechanism, every time the model reads a new word, it needs to calculate it with all previous tokens. When the input content doubles, the computing power cost will quadruple directly.

To save computing power this time, Kimi adopts a method called hybrid attention. When information comes in, it first goes through a KDA (Kimi Delta Attention) layer. KDA will read the text while writing key information into a continuously updated state.

After new content is input, it will also judge which old information needs to be retained and which can be gradually forgotten.

At the same time, to prevent KDA from missing key information, Kimi also arranges a Gated MLA after three KDA layers, which is equivalent to letting the model go back to check the original text after taking several pages of notes.

Through this combined design, K3 greatly reduces the computing cost of ultra-long context.

In addition, Kimi has also designed Attention Residuals to reduce the loss of information during transmission.

It also designed Stable LatentMoE, which first compresses the original 7168-dimensional information to 3584 dimensions, and then distributes it to 16 experts for processing at the same time.

It also sets current limiting and scheduling mechanisms for these experts to suppress abnormal activation values and allocate tasks reasonably.

It avoids the situation where some people work nonstop all day long while others idle around doing nothing.

Through the joint upgrade of these architectures, data and training methods, under the condition that the total parameters of K3 become larger, the training efficiency of the model is 2.5 times higher than that of the previous generation K2.

The release and open source of Kimi K3 is no longer a self-entertainment in the small technology circle.

It is like a trigger that has attracted all kinds of people, good or bad, to come out and speak.

For example, some American politicians accused that K3 was produced by distilling American large models.

Kimi's employees satirized back properly, saying that Fable was released not long ago. If Kimi can distill a model in dozens of days, it can apply for the Guinness World Record. Is it possible that every Chinese person has superpowers to distill the model that Anthropic plans to release in the future?

I get it now, Kimi invented a time machine, traveled back to the past to start distilling Fable. How could Kimi be so malicious...

In addition, the US government also planned to sanction and investigate Chinese artificial intelligence enterprises not long ago.

China's Ministry of Commerce also responded perfectly.

On the other side, Jensen Huang, who is good at high EQ communication, also registered for X, published an open letter explaining the importance of open source, and invited nearly 100 tech giants including Microsoft, Google, OpenAI to sign their names together.

They publicly opposed the ban on Chinese open source models.

Under the pressure, Anthropic, the firm advocate of closed source, released another long article, saying "Oh, we do not resist open source, we just worry that powerful models will fall into the hands of bad people."

To be honest, it is quite unexpected that open source models can approach these world top closed source models so quickly this time...

The Bad Reviewer still remembers the first time he saw Mythos and Fable. Although Anthropic does not do many good things all the time, the models it developed are really excellent.

At that time, most people thought the gap between open source models and closed source models would become larger and larger.

Because the two types of models are running in a completely unequal race.

Closed source models can freely learn the technologies of open source models. For example, DeepSeek has taught the whole community step by step how to make models learn reasoning and thinking.

But on the contrary, open source models cannot learn anything from closed source models.

Fortunately, there are still many opportunities for open source models to learn from each other and support each other.

If you read Kimi's paper this time, you will find that it draws a lot of experience from DeepSeek.

From multi-head attention MLA, Mixture of Experts model MoE, to training stability and reasoning efficiency, many technologies are based on the accumulation of the open source community in the past few years.

But Kimi did not simply copy them. On this basis, it continued to develop new technologies including KDA, Gated MLA, AttnRes and Stable LatentMoE.

The mutual learning and mutual respect atmosphere in the open source community is the most precious part of open source models.

The experience left by the previous generation of models will directly become the starting point of the next generation of models.

These open source models draw strength from each other, develop layer by layer, and finally catch up with those closed source giants that have more computing power, capital and data resources.

Thanks to the continuous efforts of these open source models, at least after the release of K3, no one will make absurd remarks like "open source models will fall behind more and more".

The development of AI is no longer the conflict between companies, or between open source and closed source. It is not only related to technological equality, but also closely linked to national security, geopolitics and many other fields.

We also hope that there will be more and more excellent models like K3 in China in the future.

After all, reasoning makes sense, but in many cases, showing real strength works better.

This article is from the WeChat official account "X.PIN", authors: Jiang Jiang & Zao Qi, editors: Jiang Jiang & Mian Xian, 36Kr is published with authorization.