AI has conducted a 100-year retrospective review of academic papers, finding that 99.2% of top-tier academic journals have problems.

Who says scientific research no longer has new problems? That all promising research directions have been fully explored by predecessors??

If I say —

Most of the top journal papers that have been published are flawed?

Recently, some researchers used AI Agents to conduct "academic fraud detection" and found that:

Among the 92 papers presented at the 2026 International Conference on Machine Learning (ICML), 58 of them are no longer reproducible.

Is this true? Could it be that the opportunity to publish a large number of top conference papers has really arrived?

Top conference papers are being collectively "fact-checked" by AI

Normally, a paper accepted by a top conference means it has already undergone peer review.

However, limited by energy, peer reviewers hardly have time to download data, run models or reproduce experiments from scratch to verify every experimental conclusion in the paper.

At the same time, with the explosive growth of the number of papers in the AI field, models are becoming more and more complex, and the scale of experiments is getting larger. The code, data and parameter settings behind a single paper may involve hundreds of details.

The verifiability of papers has become increasingly important. After a paper is published, can the conclusions in it be re-run and reproduced?

At present, AI Agents have greatly reduced the cost of such verification and reproduction.

On July 22, a US-based research auditing company used AI Agents to conduct a systematic result reproduction audit on all 168 oral presentation papers of ICML 2026.

92 of the 168 papers have at least 5 verifiable conclusive statements.

The final result is: only 34 papers can have more than 40% of their conclusions successfully reproduced by AI Agents; and only 8 papers can have more than 80% of their conclusions reproduced.

However, it should be noted that there is a huge difference between "unreproducible" and "research fraud".

The reasons for reproduction failure include but are not limited to missing key files in the code, broken dependency library versions, inconsistent running results with the paper, and 4 papers rely on models that have been taken offline, which means the experimental results are permanently unreproducible by anyone.

In addition, some errors are rather absurd —

For example, one paper took "only training 0.77% of the parameters of the base model" as its core selling point, but the actual open-source checkpoint trained 6.31% of the parameters, a difference of about 8 times.

Another paper included a reliability table based on a certain evaluation model, but its open-source code does not contain the evaluation model at all, nor are there any scripts that can generate the results in the table.

Coincidentally, Hugging Face and AlphaXiv jointly launched the "Agent Reproduction Challenge" competition for ICML 2026, inviting researchers to use AI Coding Agents to automatically reproduce the papers accepted by ICML 2026, and the results are equally not optimistic.

A similar trend is even more surprising in the detection of paper errors and omissions.

At the end of 2025, a study developed a GPT-5-based paper inspection system to analyze papers that had been published in top AI conferences and journals, looking for objectively verifiable problems. The researchers clearly stated:

We only look at "objective errors", and we do not consider the innovation and research value of the papers.

The final results show that an average of 4.7 objective errors were detected in each paper, and 99.2% of the papers were marked with at least one problem.

△ Image generated by AI 

In terms of error types, mathematical and formula errors account for the highest proportion, reaching 54.0% (including wrong equations, loopholes in derivation logic, wrong assumptions in proofs, etc.).

About 30.8% of NeurIPS papers and 23.8% of ICLR papers contain at least one substantive error that may affect the interpretation of results.

What is more noteworthy is the time trend: the average number of errors per NeurIPS paper rose from 3.8 in 2021 to 5.9 in 2025, an increase of 55.3%.

……

Perhaps in the future, paper publication will no longer mean that the research has declared "victory", but only the beginning of a new round of AI verification.

Great news for junior researchers: you can publish top journal papers without doing experiments

So friends, for those who are worrying about choosing a research topic, this may be a quite unexpected "positive development".

Finding errors in literature and writing corrigenda is already a legitimate form of output in the academic circle, and now it seems to be a completely untapped academic blue ocean.

How to get started? Here are some tips for junior researchers who are struggling with topic selection:

First, change your research mindset: don't compete in cutting-edge fields, but explore past research.

Shift from "looking for new topics" to "looking for anomalies in old papers", focusing on highly cited classic papers with little controversy. After all, "99.2% of papers have at least one error".

Second, let AI help you build knowledge maps and research pedigrees.

Such knowledge maps can help you understand — which are the truly foundational works? Which research views are still controversial? Which conclusions are widely cited but lack verification? What is the citation relationship between different papers? So as to find connections and anomalies that were hard to discover in the past.

Ask anything.

Many important breakthroughs in the history of science do not come from proposing brand new questions, but from re-examining a long-accepted hypothesis.

For example: Has a classic experiment been verified according to today's standards? Does a widely cited conclusion have unnoted limiting conditions?

In the past, the cost of such inquiries was very high, and researchers needed to spend a lot of time consulting original literature and comparing experimental details. Now AI has reduced this cost to almost zero.

AI may start to restructure the history of science

Recently, Pios, a theoretical chemist at Zhejiang Lab, found that the results when using AI to predict the boiling point of molecules were in obvious conflict with a chemical database 75 years old.

In this case, the first reaction of anyone might be "I must be the one who is wrong".

Pios was no exception, and he quickly checked his model. But after manually tracing back to the original literature, he found: Oh my god, AI is the one that is correct.

Pios was greatly surprised, then continued to check with AI, and unexpectedly found that a boiling point measurement value from about a century ago, which was recognized as authoritative by the academic circle, was also wrong.

It should be noted that these data have been cited and incorporated into subsequent research for many years.

Such problems have not been discovered for a long time in the past, which may be directly related to the scale of scientific literature.

△ Image generated by AI 

The number of modern scientific papers has far exceeded the reading limit of any individual researcher. For example, the annual submission volume of ICLR, a top AI conference alone, has risen from 1013 in 2018 to 19619 in 2026.

At this scale, once an error that first appears in a certain paper is repeatedly cited by subsequent literature, it will continue to spread along the citation chain, forming a de facto academic consensus.

Due to the long time elapsed, complex citation chains, plus the huge time cost of review work and limited academic returns, most of the errors that have entered the literature system have never been systematically reviewed.

But now everything is changing. At least in terms of technical capabilities, for the first time, people have the possibility of large-scale re-examination of past scientific literature.

However, taking the GPT-5-powered Paper Correctness Checker as an example, its detection precision rate is 83.2%, and about 40% of the real errors are still not detected in each test.

It can be said that such AI fact-checking tools are not sufficient to act as the judge of scientific literature, and the output results of AI checking tools finally require manual review.

This article is from the WeChat public account QbitAI (ID: QbitAI), author: Cheng Qian, authorized for release by 36Kr.