MiMo-V2.6 has just been released, Luo Fuli from Xiaomi unveiled the new MiMo-V3 architecture, which adopts multiple technical achievements from DeepSeek.

Reported by Zhidx on September 24, Luo Fuli, head of Xiaomi's MiMo large model, released an article last night, disclosing the new architecture of MiMo-V3 in advance, with its core component HySparse2 debuting first, and the relevant technical paper was published at the same time.

▲ Article released by Luo Fuli

In simple terms, this new architecture mainly solves three problems that arise after Agents run for more and more rounds: excessive computation for long inputs, excessive cache occupation, and how to accurately find the required information from increasingly long contexts.

Luo Fuli directly presented a set of data: compared with the Hybrid SWA architecture adopted by MiMo-V2.6, under the context of 1 million Tokens, the Prefill computation required by HySparse2 when processing long inputs is reduced to about 1/5, and the KV Cache occupation is reduced to about 1/4.5; at the same time, its long context retrieval performance is further improved, and AgentPPL and LongPPL are also decreased.

▲ Comparison of HySparse2 in long context performance, Prefill computation amount and KV Cache occupation

The HySparse2 paper is completed by Xiaomi LLM-Core team, with a total of 15 authors. Luo Fuli is the corresponding author and marked as Team Lead. Many well-known industry achievements can also be found in the references of the paper.

Among them, there are 4 DeepSeek-related achievements, including DeepSeek-V2, DeepSeek-V3.2, DeepSeek-V4 and DeepSeek-V4.1-Flash; in addition, OpenAI's gpt-oss model card and the relevant evaluation work of GPT-4.1 are also cited.

▲ 4 DeepSeek-related achievements cited in the HySparse2 paper

It is worth noting that only two days have passed since Xiaomi's last model update.

On September 22, Xiaomi's large model team just released and open sourced the new generation of Xiaomi MiMo-V2.6 series, and announced that MiMo-V2.6-Pro-UltraSpeed will be gradually opened in the future. Compared with MiMo-V2.6-Pro, the latter has a 20x higher output speed at the same intelligence level.

Xiaomi also open sourced the RL environment, framework and model technical report for new model training at the same time. Luo Fuli said at that time that MiMo-V2.6 bet on large-scale RL expansion and carried out the largest single RL training of open source models so far. The team regards it as a new step to explore RSI (Recursive Self-Improvement) and move towards AGI.

MiMo-V2.6 has just brought large-scale RL to the forefront, and this time Xiaomi has targeted the underlying architecture of the model.

01.

Multi-round Agent Tasks

Bring Higher Long Input Costs

The problem solved by HySparse2 is related to an increasingly typical working mode of Agents: the action generated by the model is very short, but the content returned by the tool may be very long.

For example, an Agent may only generate one search instruction or one tool call, then the search engine returns a long document, and the code tool returns a large number of running logs. Before the model performs the next round of reasoning, it needs to process these newly added content first.

This process is Prefill, that is, pre-filling.

When the Agent continuously calls tools such as search, code execution, and file reading, documents, web pages and running records will continuously enter the context. The model not only needs to repeatedly process these newly added inputs, but also save an increasingly large KV Cache, and find the content that is really needed at present from hundreds of thousands or even millions of Tokens of historical information.

Therefore, the paper summarizes the three problems faced by long-cycle Agent reasoning as: reducing Prefill computation, reducing KV Cache occupation, and improving long context retrieval accuracy.

The previous generation of HySparse has already done a round of optimization. It arranges the full attention layer and the sparse attention layer alternately, and the subsequent sparse layer can reuse the KV Cache and selection index generated by the previous full attention layer, thereby reducing attention computation and cache occupation. However, in the Prefill stage, HySparse still needs to execute all network layers in sequence.

For HySparse2, Xiaomi further shortens the execution path of Prefill: when processing long inputs, the model only needs to run the first half to complete the construction of KV Cache required for subsequent reasoning.

02.

Two-level KV Sharing

Allows the Model to Run Only Half of the Layers

The core to achieve this is the two-level KV sharing adopted by HySparse2.

The first layer is called KV Bridging.

HySparse2 divides the main body of the model into two parts: Self-Decoder and Cross-Decoder. The former is composed of full attention and sliding window attention, while the latter is composed of full attention and sparse attention.

In the Cross-Decoder, when the full attention layer generates K and V, it can directly use the hidden state generated by the corresponding full attention layer of the Self-Decoder. In this way, the second half does not need to completely process all the previously input Tokens again.

The second layer is KV Reuse.

After entering the Cross-Decoder, the KV Cache and Token selection results generated by one full attention layer will continue to be provided for subsequent multiple sparse attention layers to reuse. After the two levels of sharing are superimposed, all KV Cache required by the Cross-Decoder can be constructed according to the hidden state of the Self-Decoder.

▲ Two-level KV sharing architecture of HySparse2

Prefill can exit after executing the Self-Decoder, without requiring the second half to run through the long input completely again.

HySparse2 also adjusts the way to find information in long contexts.

The previous generation of HySparse adopts Block-level selection, that is, selecting a whole block of continuous Tokens at a time. HySparse2 is changed to Token-level selection, which can directly find relevant Tokens from different positions of the context.

This change is more suitable for multi-round Agent tasks. For example, a really useful piece of information may be hidden in a tool return from a long time ago. The Block-level scheme needs to process the entire surrounding content together to extract one Token from it; the Token-level scheme can directly select the required position.

The ablation experiment in the paper shows that under the same attention budget, the Token-level scheme achieves higher scores in long context retrieval and graph reasoning tests such as RULER-v2, MRCR-v2 and GraphWalks.

▲ Comparison of the effect of Token-level and Block-level sparse selection

HySparse2 also cancels the separate SWA branch in the Cross-Decoder, and instead forces the retention of the most recent segment of context.

In the experiment of the paper, the most recent 128 Tokens are fixed to be retained each time, and then 1024 Tokens are selected from the earlier context. Recent information and long-distance information can share the same KV Cache, which also reduces extra cache.

Taking the 49-layer model shown in the paper as an example, when Prefill and Decode are deployed separately, only the first 25 layers need to be deployed on the Prefill node, and the required model weights are nearly halved; only one full attention layer needs to be executed at this stage.

03.

Long Context and Agent Tasks

Bring More Significant Performance Improvements

In order to verify HySparse2, the Xiaomi team trained a set of 80B-A3B MoE models, and compared them with the previous generation of HySparse and the Hybrid SWA adopted by the MiMo-V2 series.

The three groups of models use the same data and training plan, with only different attention architectures. The models first use about 500 billion Tokens for pre-training with a context length of 32K; then use about 100 billion Tokens for light post-training, and extend the context length to 256K.

The test results show that the more obvious improvements of HySparse2 are concentrated in long context retrieval and Agent tasks.

After post-training, compared with HySparse, the average scores of HySparse2 on MRCR-v2 and RULER-v2 are increased by 11.30 and 19.81 percentage points respectively; compared with Hybrid SWA, they are increased by 6.44 and 18.65 percentage points respectively.

Under the 256K context, the RULER-v2 score of HySparse2 reaches 58.45, that of HySparse is 32.61, and that of Hybrid SWA is 35.74.

At the same time, HySparse2 has lower AgentPPL and LongPPL than the other two architectures under all context lengths tested in the paper.

▲ Performance of HySparse2 on long context and Agent tasks

The gap in computation and cache is more direct.

Under the context of 1 million Tokens, the Prefill FLOPs of HySparse2 are reduced to about 34% compared with HySparse, and reduced to about 20% compared with the Hybrid SWA adopted by the MiMo-V2 series.

Under the same condition, the KV Cache occupation of HySparse2 is 2.69GB, that of HySparse is 6.72GB, and that of Hybrid SWA reaches 12.09GB. Converted, compared with Hybrid SWA, the cache occupation of HySparse2 is reduced to about 1/4.5.

▲ Comparison of Prefill computation amount and KV Cache occupation of different architectures under long context

In the part of general capabilities, the conclusion given by the Xiaomi team is relatively modest: the overall performance of the three architectures in capabilities such as knowledge, reasoning and code is roughly equivalent, and different test items have their own advantages and disadvantages.

For example, HySparse2 has higher scores on BBH and MMLU-Pro, while HySparse performs better on items such as DROP.

▲ Performance comparison of HySparse2, HySparse and Hybrid SWA on knowledge, reasoning, code and long context tasks

Compared with these conventional tests, the improvement of HySparse2 on long context tasks such as RULER and NoLiMa is more prominent.

The paper also verifies the impact brought by different designs through ablation experiments.

For example, after canceling the independent SWA branch and adopting a forced local window, the scores of some mathematical reasoning and MRCR-v2 have changed, but this design eliminates additional projection parameters and local KV Cache, and allows Prefill to exit directly in the middle of the model. The Xiaomi team regards it as an architectural trade-off between efficiency and model capabilities.

04.

Conclusion: MiMo-V3 is Imminent

Luo Fuli Shows the Card in Advance

Only two days after the release of MiMo-V2.6, Luo Fuli has already revealed a key architectural change of MiMo-V3 in advance.

According to the design in the paper, HySparse2 adopts two-level KV sharing, so that the Prefill stage only needs to run about half of the model, further reducing the long context computation amount and KV Cache occupation, and achieving higher scores in multiple long context and Agent tasks.

This also makes one technical direction of MiMo-V3 clear in advance: in the face of increasingly long and multi-round Agent tasks, Xiaomi is continuing to compress the computation and storage cost of the model for processing long inputs.

From MiMo-V2.6 betting on large-scale RL to MiMo-V3 unveiling the new underlying architecture in advance, the model iteration rhythm of Xiaomi's Mi