The 10-trillion-parameter large model is still nowhere in sight, and ByteDance has already set up a first-tier internal department ahead of the R&D progress of the model.

ByteDance aims to catch up in the large model race, and simply piling up GPUs may no longer be sufficient.

According to reports from Intelligent Emergence, ByteDance has recently established a new first-level department "AI Data and Security", which is parallel to departments such as Seed, Flow, and Douyin. Meanwhile, Fu Yue, the former head of AI Data and Security, will leave his position soon, and his post will be taken over by Wang Yinglei, the former head of the TikTok Platform Responsibility Team.

Looking at this incident alone, it seems like another ordinary organizational adjustment at ByteDance, but the timing is rather coincidental.

Recently, the Financial Times, citing three people familiar with the matter, reported that ByteDance is pre-training a super AI model with a maximum parameter scale of up to 10 trillion, which is still in the early stage, and the final model scale can only be determined in the subsequent training phase. If this scale is finally achieved, it will become one of the largest AI models in the world in terms of parameter count.

Compared with the current mainstream large models, this is not simply making the model "a little bigger", but continuing to bet on the limit of Scaling.

The larger the model, the more than just GPUs are needed to feed into it.

01

The Model Is Not Yet 10 Trillion

Data Has Been "Promoted" First

The "AI Data and Security" department newly established by ByteDance is not an additional department created out of nowhere.

According to news from Intelligent Emergence, one of the predecessors of the new department is Global Data, a team founded by Fu Yue in 2023. This team of about 100 people initially served international businesses such as Dola and TikTok, and later gradually took charge of data procurement and quality control for Seed model training.

Later, as ByteDance's large model business expanded further, there were more and more teams focused on data.

In addition to Global Data, the group's data middle platform DMC, the AIDP AI data platform under Flow, and different model directions within Seed have successively established their own data teams. They serve different targets, but their work is getting increasingly similar: procuring data, organizing annotation, building datasets, and being responsible for evaluation and quality control after model training.

The result is that for the same task of preparing data for large models, several sets of teams have gradually emerged within ByteDance.

Since the beginning of this year, ByteDance has started to reorganize this system. Data teams that were originally scattered in different business lines have been gradually integrated, and finally formed the new "AI Data and Security" department, which was directly upgraded to a first-level department, parallel to Seed, Flow, Douyin and other departments.

However, judging from the manpower and capital ByteDance has invested in data, this promotion is not unexpected. Previous disclosures from Intelligent Emergence show that the Seedance data evaluation team alone has more than 1,000 people, and one algorithm engineer is often matched with more than a dozen data personnel; this year, ByteDance's data budget for the world model and Coding directions has reached the tens of millions of dollars level, and the budget can still be increased further.

This is no longer the old business of finding several outsourcing companies to do annotation. From data procurement, cleaning, and synthesis to model evaluation and quality control, data is becoming a set of engineering that requires independent organization and continuous investment, just like algorithms and computing power.

More importantly, ByteDance has actively blocked a shortcut for itself.

On August 5, according to The Information, Zhang Yiming clearly opposed catching up by distilling external models at the all-hands meeting of Seed held about a month ago. The report cited people familiar with the matter as saying that Zhang Yiming believes ByteDance should be willing to "sacrifice some short-term benefits for long-term goals", and even if Seed lags behind in stages, it should not rely on distilling competitors' models to close the gap.

While targeting parameters of 5 trillion or even 10 trillion, and not wanting to directly copy the answers of other people's models, ByteDance can only prepare more "teaching materials" on its own.

And now, the task of finding teaching materials is no longer just ByteDance's own trouble.

02

Models Are Getting Larger

High-quality Data Is Running Short

The problem ByteDance is facing is actually the problem that all large model companies that want to continue Scaling are facing.

In the past few years, the way to make large models stronger was simple and crude: more parameters, more computing power, and more data fed in, but these three resources are not unlimited.

In 2022, DeepMind found in its famous Chinchilla paper that if you want to train the model more sufficiently under the given computing power, you cannot only increase one of the model parameters and training Tokens. According to the empirical law given at that time, when the parameter scale doubles, the amount of training data should also increase synchronously.

The problem is that parameters can continue to be piled up and GPUs can continue to be purchased, but the high-quality content written by humans on the Internet is limited.

Research institute Epoch AI once estimated that after considering quality and reuse, the total amount of public human text in the world is equivalent to about 300 trillion Tokens.

According to the past growth rate of the training data scale of large models, this batch of data may be fully utilized between 2026 and 2032; if the model undergoes longer "overtraining" to reduce the reasoning cost, this time may be advanced.

The public Internet is not enough to feed the large models, so large model companies can only start to look for resources outside the fenced area.

The first targets are the content that was difficult to crawl on a large scale in the past, or not free at all. Forums, news websites, paid publications, professional databases, and even physical books without digital versions have begun to be re-priced. As a result, the data problems that used to be solved by crawlers have increasingly become a procurement business.

In 2024, Google signed a content licensing agreement with Reddit for about 60 million US dollars per year for AI model training; in the same year, OpenAI reached a multi-year cooperation with News Corp to include historical and real-time content of its media outlets such as The Wall Street Journal and The Times into the available scope.

But spending money on ready-made data is only the first step.

After relatively standardized content such as news and forums are also divided up by large manufacturers, more professional and scarce data needs to be specially produced by designated personnel. In fields such as code, mathematics, and science, large model companies have begun to directly hire engineers, doctors, and industry experts to participate in the production and evaluation of training data.

Going further, even "buying" is not feasible. Anthropic finally simply reached for physical books...

In January this year, The Washington Post, citing court unsealed documents, disclosed that Anthropic launched an internal project codenamed "Project Panama" as early as 2024. What Project Panama does is also very simple and crude: buy books, cut books, and scan.

Anthropic purchased millions of physical books in batches, then directly cut off the spines of the books with hydraulic cutting equipment, sent the scattered pages into high-speed scanners, digitized them, and then recycled the original books. About a year after the project was launched, Anthropic spent tens of millions of dollars on it. Internal documents even set the goal as "destructively scanning all books in the world".

What's more exaggerated is that before Project Panama, Anthropic also downloaded millions of pirated books from shadow libraries such as LibGen, which later became the direct reason why Anthropic was taken to court.

In 2025, the court ruled that using books to train AI can constitute fair use, but Anthropic's act of saving more than 7 million pirated books to build a "central library" faces infringement risks. To end the class action lawsuit, Anthropic finally agreed to pay 1.5 billion US dollars in settlement; in July this year, the U.S. federal court officially approved the agreement, making it one of the largest known copyright lawsuit settlements in the United States.

From this perspective, Anthropic's 1.5 billion US dollar settlement is more like marking an extremely exaggerated price for high-quality data. After all, books can be bought, copyrights can be bought, and experts can be hired with money, but the high-quality data that has not been used by models will not grow out of thin air along with the increase of parameters.

This is also the most troublesome place for ByteDance now. On the one hand, there is a model plan with a maximum of 10 trillion parameters, on the other hand, it does not take shortcuts by distilling other models. To continue to Scale the model, we finally have to return to the most simple question: what data to feed it.

Even if the 10-trillion-parameter model is only a plan for now, data has already become a business that ByteDance must bet on independently.

This article is from the WeChat official account "foci", author: Sean, published with authorization from 36Kr.