What is the price that large model companies will pay if they do not adopt model distillation?

By | Li Zhaofeng

Edited by | Zhang Yuxin

"A standard training pipeline should take distillation as a cold start, to push the model quickly to a 90-point performance level, and then work on the remaining 10 points of improvement." A foundational model researcher said when talking about the role of distillation in current LLM (Large Language Model) training.

The "cold start" he mentioned refers to the process that enables less capable models to first learn the proven answers and task trajectories of more powerful models, so as to obtain a basic version that can continue training as soon as possible.

For model companies, "not using distillation" means abandoning this acceleration process and starting independent exploration from basic capabilities. The advantage of this approach is that companies can build a more independent training system, but it also requires a much longer model iteration cycle.

As previously reported by *Intelligent Emergence*, ByteDance set the principle of "no distillation" when its Seed team was first established. A ByteDance employee explained that the team aims to enter the global first tier, which "cannot be achieved through distillation". At an internal Seed meeting at the end of July, Zhang Yiming also clearly stated that even if it temporarily lags behind domestic peers, ByteDance will not take distillation as a shortcut to improve model capabilities.

This statement about "no distillation" is not just a technical judgment on training methods, but a choice of long-term development path.

On the other hand, one week before Zhang Yiming made the "no distillation" statement, Jensen Huang, CEO of NVIDIA, expressed a completely opposite view in an interview with Mike Allen, co-founder of Axios: "Distillation, that is, learning from AI and other knowledge sources, is the foundation of intelligence."

A few days after the interview, Jensen Huang reposted an open letter for open models supported by companies including NVIDIA, Meta, Microsoft and Dell. The letter advocates building an open model ecosystem that can be applied to all walks of life, and lists distillation as a widely used method in model improvement, testing and verification.

Should distillation be adopted after all? What are the costs and returns of not using distillation? *Intelligent Emergence* communicated with multiple large model researchers on related issues.

01 How Much Time Does It Cost to Avoid Distillation

"We don't have enough data, insufficient computing power, and lack of relevant pipelines." A researcher summarized the gap between domestic foundational model teams and leading overseas manufacturers. In his view, there is still a gap between domestic pre-training and the cutting-edge level in North America, but the gap is no longer as huge as in the early stage, and the larger gap lies in post-training and high-quality data.

Distillation technology is an effective means that can help foundational model manufacturers quickly make up for the above "shortcomings", thereby compressing the model iteration cycle.

Does ByteDance, which chooses not to use distillation, not need to face these problems? The answer is no. At the end of July, ByteDance's latest foundational model Seed-2.1 Pro ranked 16th in the overall Code Arena WebDev list, lagging behind Kimi K3 (2nd place) and GLM-5.2 (6th place).

During the communication, many researchers believed that "completely avoiding distillation will definitely slow down the progress significantly, and it seems unrealistic to push Coding and Agentic capabilities to the top of the industry in the short term without the help of distillation."

The time pressure for model capability improvement comes from the data flywheel. Only when the model reaches a usable level can it be put into real development environments. Capabilities bring usage, usage generates feedback data, and the feedback data flows into the next round of training.

Previously, a ByteDance employee told *Intelligent Emergence* that one of the reasons why its Coding model was difficult to make breakthroughs before was the lack of willingness for business adoption and insufficient data feedback.

"If you can't even complete the first step of cold start, the subsequent pipeline will not be able to operate." An interviewed researcher said. He described this process with the "Matthew Effect": users tend to choose the most capable model, more usage brings more real tasks and feedback data, which further widens the iteration gap between different models.

Agent tasks further amplify this time difference. In current Agent scenarios, multi-round tool calls require the model to continuously plan, execute and correct errors. Once the previous step deviates, errors may accumulate continuously along the task chain. It is usually much slower for a model to explore a successful trajectory from scratch than to directly learn a proven process.

However, the pressure brought by "no distillation" is not the same across all capability tracks. Different from the fact that Seed has not entered the top domestic camp in the LLM field, ByteDance has already ranked among the global leading players in the video generation field.

Many industry insiders summarized the success of Seedance as "a victory of data". A person close to ByteDance told *Intelligent Emergence* that Seedance's video training pipeline does not use distillation from external models, and even the VLM used for data screening, understanding and other links does not take the output of other models as the distillation source.

Video generation is different from language models: AI-generated videos still easily show recognizable "AI characteristics", the content produced by external models is difficult to directly replace high-quality real video data, and the gain brought by distillation is relatively limited.

In the video generation track, ByteDance has reached the cutting-edge level without the help of external model distillation relying on its accumulation in data, architecture and products; in the Coding and Agent tracks, it still needs to make up for the capability gap as soon as possible to obtain more real usage and feedback data. The same "no distillation" principle leads to different time costs on different capability tracks.

In the LLM field, while other large model companies complete cold start with the help of distillation and enter real scenarios in advance, ByteDance, which insists on no distillation, needs a much longer training cycle to verify whether its self-built data system can be transformed into leading capabilities.

This also explains why ByteDance recently established a first-level department called "AI Data and Security", which provides services such as source procurement, synthetic cleaning and quality evaluation for its large models — beyond models and products, ByteDance must take data capability as an independent organizational pillar to support its own training system.

02 The Returns and Costs of Distillation

The classic paradigm of modern knowledge distillation can be traced back to *Distilling the Knowledge in a Neural Network* published in 2015 by Hinton, winner of the Turing Award and pioneer of deep learning, and his collaborators. The paper proposes that a smaller "student model" can learn the output distribution of a large "teacher model", so as to inherit its capabilities at a lower cost.

For example, DistilBERT, released in 2019, reduces BERT's size by 40%, retains 97% of its language understanding capabilities, and increases the inference speed by 60%. At that time, distillation was mainly used to solve the problem that models were too large and too expensive to deploy.

After the rise of generative AI, the answers, explanations and task trajectories generated by more powerful models have also been directly used to train other models. In March 2023, the Stanford team used the OpenAI model to generate 52,000 pieces of instruction data, and trained Alpaca with a data cost of less than 500 USD; in the same year, Microsoft let the 13-billion-parameter Orca learn the explanation trajectories provided by GPT-4, and achieved performance comparable to ChatGPT in the complex zero-sample reasoning task of Big-Bench Hard.

Strictly speaking, using the data generated by the "teacher" model for supervised fine-tuning is not equivalent to traditional knowledge distillation. But in the context of the large model industry, this route of "using the output of a strong model to train another model" is also included in the generalized distillation. When Meta released Llama 3.1 405B in 2024, Mark Zuckerberg, CEO of Meta, had already listed distilling small models as an important use of the open flagship model; Meta also included synthetic data generation into the main workflow of this model in the same period.

By 2026, distillation has been maturely applied to model capability merging. According to the DeepSeek V4 technical report, the team first trained more than ten experts in mathematics, Coding, Agent and other fields through supervised fine-tuning and reinforcement learning respectively, then let the general model learn the output distribution of each expert on its own sampled trajectories, to merge capabilities through multi-teacher online policy distillation.

Kimi K3 also integrates distillation into the main post-training process. Its technical report shows that the team formed 9 teacher models around general reasoning, Coding and Agent tasks, as well as three thinking intensity levels of low, high and max, and then integrated them into the same model through multi-teacher online policy distillation.

"In the final analysis, distillation is a method of constructing data." A researcher summarized its returns: Compared with building a synthetic data pipeline from scratch, making repeated attempts and errors, or organizing expert labeling for complex tasks, letting a more powerful model generate answers and operation trajectories can usually push the model to a usable starting point faster.

This return is particularly direct in Agent training. Last week, a research team from the Korea Advanced Institute of Science and Technology submitted a study showing that extracting task processes, subtask examples and tool usage experience from the successful trajectories of GPT-5-mini improved the task accuracy of 4B to 8B models in three tool usage benchmarks.

Distillation can also reserve more expensive reinforcement learning for difficult problems. "For relatively simple problems, direct distillation is enough, there is no need to let the model explore through reinforcement learning, which will waste a lot of resources." A researcher said.

The "distillation scaling law" published by Apple's research team in 2025 calculated this cost: when the teacher model already exists, or multiple student models need to be trained repeatedly, distillation usually saves more computing power; if you only train one student model and you need to train the teacher model from scratch for this purpose, direct training is usually more cost-effective.

Distillation saves the time of finding answers, but it cannot decide what problems a model company should solve.

"To put it bluntly, distillation is not complicated: you let a more powerful model run through a problem, then take out the trajectory and convert it into a training format." A researcher said. In contrast, "the difficulty lies in creating problems": problems can come from web crawling, expert contributions, or independent construction by the team.

The returns of distillation are time and computing power, but this efficiency also has costs. Many researchers told *Intelligent Emergence* that distillation is usually used to push the model past the usable threshold as soon as possible; when multiple models reach a similar starting point, further improvement does not necessarily rely on distillation. "When the marginal return diminishes, everyone will gradually replace this method."

"But there are not many teams that really have the ability to promote distillation, reinforcement learning and independent R&D at the same time." The above researcher admitted.

In addition, distillation will also change a company's resource allocation. Some researchers mentioned that distillation allows the model to achieve a "decent result" in the short term, but the effect comes too quickly, which may also make the team "too attached to this method". If you want to enter the top-level competition, "distillation alone is not enough, you still need to go back to algorithms and independent R&D to make breakthroughs."

A few weeks ago, after the release of Kimi K3, Braden Hancock, researcher at the Lauder Institute and co-founder of Snorkel AI, said that it is impossible to train a model of such strength in such a short period of time only by distillation.

Nathan Lambert, researcher at the Allen Institute for AI, also pointed out that if distillation is enough to replicate cutting-edge capabilities, other teams should have easily caught up with GLM or Kimi, but the reality is not the case.

Some researchers choose to leave the answer to the next round of model iteration. In his view, the biggest obstacle of the no-distillation route lies in whether the company can withstand a period of time without "presentable outputs".

The game between long-termism and time cost around distillation is still ongoing. Beyond the competition for resources and team technical routes, it also tests the patience of the industry waiting for results.