This domestic model, which ranks second globally on OpenRouter's weekly ranking, has finally been open-sourced.

Finding the optimal solution among capability, cost and stability.

ZhiDongXi reported on October 15. Today, a large model that just ranked second on OpenRouter's global weekly Token usage list has been open-sourced!

StepFun has officially opened the model weights of Step 5 Preview, with both BF16 and FP8 versions launched on HuggingFace and ModelScope synchronously.

Hugging Face:

https://huggingface.co/stepfun-ai/Step-5-Preview

ModelScope:

https://modelscope.cn/models/stepfun-ai/Step-5-Preview

Step 5 Preview adopts a sparse MoE architecture, with 600B total parameters and only 27B activated parameters, supporting a 1 million Token context window. It scored 44 points in the Intelligence Index list of Artificial Analysis, ranking top three among global open-source models.

After its official release, this model quickly entered the vision of global developers. As of October 15, 2026, the Token consumption of Step 5 Preview has ranked second on both the weekly and daily leaderboards of the open model platform OpenRouter, with the consumption in the past seven days reaching about 18.8 trillion Tokens.

According to the Artificial Analysis evaluation data when Step 5 Preview was released, its average cost per single task is only 1/8 of that of Claude Opus 5. The total cost for a single task of the former is $0.71 (about 4.75 RMB), while that of the latter is $5.86 (about 39.22 RMB).

In terms of benchmark tests, according to the announcement from the StepFun team, in both public and internal evaluations, Step 5 Preview shows relatively balanced comprehensive capabilities in benchmark tests for complex software engineering tasks (including long-distance planning tasks), professional knowledge work and financial fields.

It outperforms Kimi K3 and GLM-5.3, second only to GPT-6 Astra or Claude Opus 5 on ALE-CLI (the CLl subset of Agents' Last Exam), FrontierFinance (financial investment research benchmark) and DRACO (cross-domain in-depth research benchmark).

The performance of Step 5 Preview shown in the leaderboards and actual usage also makes people curious: what different technical designs are behind this model? I deeply explored the technical blog of Step 5 Preview, trying to find the answers.

01. 92-layer Narrow-Deep Architecture to Improve Computational Efficiency at the Architectural Level

Step 5 Preview is stacked with 92 Transformer modules. Its model depth is equivalent to that of trillion-parameter-level open-source large models such as Qwen 3.8 Max and Kimi K3, but the total number of parameters is less than 1/4 of those models, and the activated parameters are less than 1/3.

Why is depth so important?

StepFun's explanation is that in agent tasks, the model needs to continuously call external tools, and each call may bring a large amount of new text. Considering Token efficiency, the model may not have the opportunity to fully reason through explicit chain-of-thought, so it relies more on "implicit reasoning" when reading the context.

In Transformer, information propagates layer by layer along the token time direction and the network depth. If the number of layers is insufficient, the information of multi-hop dependencies may run out of network depth before the transmission is completed, and complex reasoning in long texts may not be completed.

There is another technical route in the industry to increase the equivalent depth through cyclic or recursive structures, while StepFun has chosen a more direct approach: deepen the number of layers and control the network width.

This "narrow and deep" design, on the one hand, provides a longer information transmission path for multi-hop reasoning, and on the other hand, can control the parameter scale and response computing overhead, so that the model can obtain long-distance task capabilities equivalent to trillion-level models.

In terms of attention mechanism configuration, Step 5 Preview interleaves two types of attention layers at a ratio of 3:1: 3/4 are efficient Gated sliding window attention layers, and 1/4 are sparse full attention layers.

StepFun said that after comparing multiple linear attention schemes, the team found that the performance gap between each scheme is actually very small, but the sliding window attention structure is the simplest and more friendly to speculative decoding.

02. Use Sparse GQA to "Slim Down" the Attention for Million-Token Context

After the model has a deeper reasoning path, the next step is to control the attention computing overhead brought by the context.

When the context length reaches 1 million tokens, the full attention layer will become the common efficiency bottleneck for both training and inference.

StepFun's solution is to adopt a sparse attention mechanism called Sparse GQA on the full attention path, that is, introduce sparse indexing into standard Grouped-Query Attention (GQA), so that the model does not need to perform attention calculation on all historical information, but performs screening first and then calculation.

This idea is not originally created by StepFun. Previous work such as DeepSeek's DSA has carried out similar explorations.

StepFun said that in its actual deployment, three hurdles need to be crossed: the overhead of the filter itself, the GPU utilization rate of fragmented calculation after screening, and the trade-off between screening accuracy and data reading volume.

Compared with MQA (Multi-Query Attention) and MLA (Multi-Head Latent Attention) structures, GQA actually amplifies these three contradictions. The reason why the team chose GQA is that this structure has better long-context accuracy in experiments and is more conducive to multi-GPU parallel inference.

Sparse GQA Schematic 1

But these three contradictions cannot be ignored. To solve this problem, StepFun gives the specific Sparse GQA scheme:

• Compress adjacent historical positions into "blocks"

to reduce the cost of the filter. The block size is set to 8, so the cost of the screening and selection links is reduced by 8 times;

Taking advantage of the characteristics that adjacent tokens have similar semantics and their screening results are highly overlapping, take the union of the screening results of adjacent tokens and perform merged calculation to improve GPU utilization;

After repeated experiments, the configuration of 8 tokens per group and a total of 4096 tokens screened is determined. The longer the text is, the more calculation this scheme saves, while the information coverage rate is almost the same as that of the more fine-grained reference scheme (78.06% vs 79.09%).

Under the sparse mechanism, the filter first compresses the shared index keys into block representations, each KV group independently selects relevant historical blocks, and the main calculation path only reads the selected original K/V and the visible positions in the current block.

In other words, the model first selects chapters roughly and then reads the original text carefully, avoiding the overhead of full attention calculation for millions of tokens.

Sparse GQA Schematic 2

03. Use Three-Layer Measures to Suppress Numerical Explosion and Gradient Spike to Achieve More Stable Training

The first two steps respectively deal with the model's reasoning path and long-context computing overhead, but to implement these designs, the training process must be stable enough.

One of the main problems StepFun encountered in training is numerical explosion and gradient spike.

Some values suddenly get out of control during the training process, which may slow down the progress in mild cases, and invalidate all previous training results in severe cases. Under the FP8 low-precision training framework, the team put forward three countermeasures:

The first measure is to convert hidden states to FP32 precision.

Low-precision training is fast and low-cost, but after the model is deepened, a small number of data points with extremely large values will quickly get out of control in the low-precision format. After using high-precision storage for this part of data, the extreme values drop by nearly ten times, and the extra cost has little impact.

The second measure is to convert the scaling factor of the normalization layer to FP32.

The normalization layer of this model adopts zero-centered RMSNorm. Once its adjustment parameter stays at the critical value in the low-precision format, it will make a certain calculation channel "switch on and off" repeatedly, causing output oscillation. After changing this parameter to high-precision storage, the output oscillation problem is significantly alleviated.

The third measure is fine-grained gradient clipping.

The team observed that the gradient spikes do not originate from data noise or the output layer, but start from some middle layers and are amplified by several orders of magnitude within several layers. Data repetition seems to add fuel to the fire; its cause may be related to the repeated amplification of specific signals by layer redundancy in backpropagation.

Since the abnormal gradients are extremely sparse, the team only needs to select a very small part (accounting for 0.00004%) of the gradients in each layer for amplitude limiting to suppress the spikes, making the training convergence curve significantly smoother.

Comparison of LLM loss before (blue) and after (orange) fine-grained gradient clipping

04. Conclusion: From Architectural Optimization to Open Deployment, Model Capabilities Move to Real Tasks

Looking back at the technical design of Step 5 Preview, from the 92-layer narrow-deep architecture, to Sparse GQA, to the training stability optimization of deep networks, these choices correspond to different problems in model reasoning, long-context processing and training engineering respectively. They are not a single-point technical breakthrough, but a series of design trade-offs around model capabilities and actual operating efficiency.

In the current large model industry, parameters and scores on leaderboards are only the entry threshold. Only when the model can withstand repeated debugging by developers, is convenient for large-scale enterprise deployment, and can be smoothly connected to real business processes, can its capabilities be transformed into tangible productivity.

Step 5 Preview promotes implementation from two paths: for developers, the official releases two precision weights of BF16 and FP8, which are simultaneously launched on Hugging Face and ModelScope; for ordinary users, the model is connected to Tencent WorkBuddy for a two-week free trial.

The parallel of the two channels extends StepFun's coverage from Agent developers to office users and R&D personnel with local deployment needs.

For Step 5 Preview, the opening of model weights also means that these technical designs will be tested in more real deployment environments and complex tasks, and StepFun's technical exploration in architecture and training will have the opportunity to enter a wider range of practical application scenarios.

The competition of model capabilities is still ongoing, but it is no longer only about the model itself. How to make the capabilities of cutting-edge models truly enter practical applications may be a more noteworthy issue in the next stage of large model competition.

This article is from the WeChat official account "ZhiDongXi" (ID: zhidxcpm), written by Wang Han, and published with authorization from 36Kr.