HomeArticle

The "Agent Edition arXiv" is here: 936 intelligent agents cranked out nearly 1,000 papers in 12 weeks, and they even rolled out the peer review mechanism.

量子位2026-11-20 17:19
Even learned to conduct peer anti-counterfeiting.

What would a scientist society made up entirely of AIs look like?

Recently, research teams from institutions including alphaXiv, Stanford, MIT and other organizations have actually conducted such an experiment:

They put 936 agents into the same online scientific research community and let them evolve freely for a full 12 weeks.

The rules are very simple: the research team built a shared preprint server and chat room, and 35 human laboratories are only responsible for providing computing power, API budget and broad research directions.

The remaining work including proposing hypotheses, writing codes to run experiments, analyzing data, writing papers, and even peer communication, is fully handed over to the Agents to complete autonomously.

As a result, over 12 weeks, these nearly 1000 agents published a total of 936 research papers on the platform, covering multiple disciplines such as machine learning and solar physics.

Not only that, they also generated 9500 discussions in 229 self-built chat rooms, reading each other's papers and reproducing experiments every day.

Once a problem is found, just like the human academic community, they will rerun the experiment with the other party's code, and then continue to explore based on the results of the "peer".

After working at full capacity, this group of Agents even set up an academic journal on their own, established a peer review mechanism, and began to check the papers of other AIs...

A real top AI academic conference~

It is also worth mentioning that in addition to the above experimental results, the server for Agent scientists to communicate is now open, and other researchers can also connect their own Agents to submit research results on it.

Next, let's take a closer look.

How does this AI-only scientific research community operate?

In general, this experiment simulates the distributed ecology of the real scientific community:

The 35 participating independent laboratories configure their own hardware, select models and operation frameworks, and then connect to the same shared workspace.

Each laboratory can independently choose different fields and directions, and then communicate through the public paper library and chat rooms to find followable work.

In order to make the research of different laboratories connect with each other, the team first established a common workflow:

1. Before taking action, the Agent must first consult the existing papers in the community, and publicly announce its hypothesis and calculation plan.

In the early laboratories that shared GPUs, they even coordinated equipment usage and parameter testing on public channels to avoid meaningless repeated batch runs among peers.

2. When submitting a paper, each study is mandatorily attached with runnable code, raw data, and a "Claim Ledger" that connects the figures in the paper to the experiment outputs.

Through these records, other Agents can accurately trace the data files and fields corresponding to each indicator, verify the calculation process, and directly run subsequent experiments on this basis.

At the very beginning of the experiment, when it was just launched on August 19, only 1 laboratory participated; from October 15 to 25, 5 more laboratories joined one after another, and after October 25, the total number finally reached 35.

As the number of participants increased, the communication of Agents gradually diverted from the initial public channels to themed chat rooms established around specific issues.

Here, they share intermediate results at any time, question experimental designs, or directly use peers' codes for reproduction.

For example, six laboratories once carried out relay research around "how to reduce English forgetting when the model learns Korean":

One Agent reported that a certain method can greatly reduce forgetting, and another soon pointed out in the chat room that the Korean proficiency learned by the two methods is completely different, and the comparison conditions are not aligned.

After receiving the feedback, it immediately reran the peer's code and corrected the effect judgment.

Subsequent laboratories continued to dig deeper into the causes and test alternative solutions along this line, and finally an Agent summarized the experimental outputs of the entire research line.

More importantly, in this Agent scientist society, "failed research" is also included in knowledge accumulation.

The platform rules grant rigorous negative results (negative/null results) the same academic recognition as positive results. As a result, other Agents can avoid the dead ends that have been falsified, greatly reducing repeated trial and error.

In addition, as more and more laboratories settled in, the number of submissions from Agents also rose, and Agent scientists began to organize more formal quality reviews.

An Agent named Cypress saw that other laboratories uploaded a number of placeholder "low-quality papers" in a short period of time, so it took the initiative to create a peer review chat room, preparing to make decisions of acceptance, revision or rejection for submissions.

After waiting for four days and still failing to recruit volunteer reviewers, it simply established a permanent large model review panel, and officially launched Journal of Agent Research (abbreviated as JAR).

Specifically, the review panel calls different models to check methods and statistics, reproducibility, and research significance respectively, and specially sets up a "devil's advocate" role.

Every criticism must quote the original text of the paper word for word, and the script will automatically check the content in quotes to prevent the peer review AI from making things up.

From November 4th to 9th, in just 5 days, this system generated 1408 peer review comments for 352 versions of 263 studies.

Of course, this Agent peer review system soon encountered problems:

During an automatic peer review suspension, the editing AI checked the codes of three papers and found that some leaked the answers in advance, and some even directly replaced the wrong results with correct answers.

It immediately asked the authors to conduct self-examination. Another Agent found that its experiment also leaked the answer, so it voluntarily withdrew the paper. After correction and rerunning, the model accuracy rate dropped from 91.7% to 64.6%.

This also confirms a point: even if the numbers in the paper do come from real runs, the underlying code implementation may still deviate from the paper description.

Checking the code and raw data is the core link for this AI journal to truly play its gatekeeping role.

Talk less, show me your code!

How good is the output of the Agent scientific research community?

So, how good is the output of this Agent scientific research community?

In order to find out the real level of these studies, the research team further screened out similar works from more than 47,000 human arXiv preprints in the same period, and finally formed 202 pairs of theme-matched AI papers and human papers for comparison, and mainly drew the following three conclusions:

First, Agent papers perform more solidly in experiment execution and method description.

In the blind text review, Agent papers scored higher than human papers in experimental rigor, reproduction details, and the matching degree between conclusions and evidence.

Second, human papers are better in terms of innovation and research importance.

Data split shows that the slight overall average advantage of Agents is mainly concentrated in Laboratory 1 which adopts a highly structured reproduction process, and the rest of the laboratories do not significantly outperform human papers.

On Twitter, alphaXiv also pointed out that Agents tend to make small improvements around familiar topics and existing benchmarks —

Their ability to conduct research is significantly stronger than their ability to judge "which problems are worth studying".

Third, a large amount of manual input is still indispensable for quality screening after research completion.

The alphaXiv team admitted frankly that after the experiment, they still spent a lot of time reading through the generated papers and verifying the results to pick out the truly noteworthy works.

In other words, "reliable research quality signals do not emerge on their own." Therefore, although all the research was done by AI, the role of humans in this experiment runs through the whole process.

However, alphaXiv also stated that many studies in this experiment used small models, and the experimental time was relatively short.

In the next step, the team hopes to use more sufficient computing power, longer experimental time, and better scientific judgment to let Agents try more challenging topics.

Reference link[1]https://www.science.org/content/article/exclusive-i-got-eavesdrop-ai-agents-doing-science-it-was-eye-opening?referrer=https%3A%2F%2Ft.co%2F

This article is from the WeChat Official Account "QbitAI", author: henry, published with authorization from 36Kr.