Terence Tao finds himself in an awkward position, and a Harvard study reveals that AI outperforms experts in academic paper peer review.

AI is now even more skilled than human experts at paper reviewing.

On October 22, two researchers from Harvard University published a paper titled *AI Review Comments Fully Outperform High-Level Human Peer Review*, which compared 134 groups of review comments written by AI and humans. The results showed that AI took the lead across seven evaluation dimensions.

The researchers also invited three former editors-in-chief of top academic journals to conduct blind reviews. In 30 comparisons, they considered the AI's review comments to be better in 26 cases.

Among the specific verified claims, the fact error rate of AI review comments was only 5.5%, while that of humans reached as high as 15.9%.

This may not be pleasant news for mathematicians who have been busy competing with AI companies recently.

Just last month, 25 Fields Medal winners including Terence Tao jointly criticized AI companies for rushing to announce mathematical breakthroughs, emphasizing that mathematical research requires rigorous verification and review.

Mathematicians are still demanding that AI undergo strict academic inspections, but on the other hand, humans may even lose the job of paper reviewing to AI.

Image generated by AI

Humans Are Completely Outmatched

The defeated reviewers were not randomly recruited volunteers.

The study used 124 academic papers and 134 human review comments provided by the rapidPeer service under Harvard University's Institute for Quantitative Social Science — a paid professional peer review service where reviewers are selected by domain experts from different research institutions and receive generous compensation.

Their performance was originally very good: among the 74 paper authors surveyed, 88% considered the received review comments to be of high quality, and 60% of them gave the highest rating.

The researchers used Anthropic's Claude Opus 4.8 to generate review comments, then invited OpenAI's GPT-5.5 to act as an AI judge to compare the review quality of humans and AI without knowing who wrote the comments.

The result was that humans lost completely.

Even if AI was not allowed to access the internet and only provided with the same review guidelines as humans, GPT-5.5 still chose the AI-generated comments in all 134 groups of comparisons.

After using specially optimized prompts and enabling internet access, AI was clearly judged the winner in 132 of the 134 comparisons, with the remaining two results being undecided, and humans still did not secure any victory.

Specifically, in the three dimensions of supplementing references, identifying paper weaknesses and research methods, AI's winning rate reached 100%; in terms of theoretical innovation, discourse framework, and expression quality, the winning rate also exceeded 90%.

The researchers even checked 651 references cited by AI with DOI or arXiv numbers, and found that all of them actually exist, and none of them are fabricated.

At this point, some people may argue that letting AI judge AI reviews is unfair, and no one knows if it will favor its own kind, or even hide some underhanded clues in the review comments.

Therefore, they invited the three former academic journal editors-in-chief mentioned at the beginning to conduct 10 blind reviews respectively. As a result, out of 30 comparisons, the human judges chose the AI's comments 26 times.

At this point, even human judges are on AI's side.

At this stage of the study, the authors even began to seriously discuss that academic journals may cancel regular human peer review, let AI undertake most of the review work, and only leave human editors in charge of final decisions, with experts intervening when necessary.

After the paper was released on October 23, Xu Yiying, associate professor of political science at Stanford University, publicly stated that the academic community should reconsider the role of human experts in paper reviewing. In his view, human experts still need to participate, but there is no need to continue writing long review comments that are often 1500 words long.

Even journals that prohibit the use of AI for peer review have reached a time to re-examine their own regulations.

It seems that some people already want to hand over the work of human reviewers to AI.

AI Reviewing Is So Meticulous That It Is Almost Annoying

So the question arises: How exactly did AI outperform human reviewers?

The answer could be very simple — it never gets tired at all.

Paper reviewing is a very mentally demanding task. Reviewers need to read the full paper, check the research methods, data, charts and arguments, and also look for relevant literature that the author may have missed.

The paper's authors even casually complained about some traditional journal reviewers when introducing the experiment: some people only read the third page of the paper and start looking for fatal flaws that can negate the paper so that they don't have to continue reading.

In contrast, AI obviously has no need to rush to get off work.

Image generated by AI

In this experiment, the researchers adjusted Claude Opus 4.8's reasoning intensity to the highest level, and specially designed a set of review prompts to allow it to search for information online.

The results showed that on average, AI conducted 12.4 online searches for each paper it reviewed.

However, the effect of being meticulous is quite obvious. Without internet access and using ordinary review guidelines, AI's winning rate in supplementing references was 62%; after switching to specially optimized prompts and enabling internet access, the winning rate directly rose to 100%.

Times have changed.

Back in April this year, the editorial team of the academic journal *Organization Science* published a study analyzing more than 10,000 review comments received in the past few years, and found that review reports heavily using AI were actually harder to read and covered narrower issues.

At that time, researchers even believed that AI-generated review comments had not yet reached the level of human peer review. Less than half a year later, Harvard came up with another set of results.

But AI is so meticulous that it occasionally drives people crazy.

The paper's authors gave an example that AI is sometimes so nitpicky that it is absurd: even though it knows that 2+2=4 is correct, it still points out that the author did not define what the plus sign means.

Human reviewers may complain that the paper has too many flaws, but AI may get "addicted to finding faults" and pick out mistakes even when there are none.

Besides, AI's capabilities are still constantly improving.

In the past, paper authors needed to prove to human reviewers that their research was worth publishing; in the future, it may be human reviewers' turn to prove to academic journals why they are still worth paying for.

Mathematicians May Feel A Little Embarrassed

Recently, the battlefield between mathematicians and AI companies has expanded from verifying mathematical proofs all the way to peer review.

On September 11, 25 first batch of Fields Medal winners including Terence Tao issued a joint statement, criticizing AI companies for treating solving difficult mathematical problems as a model capability competition, rushing to announce major breakthroughs without leaving enough time for the mathematical community to verify, understand and discuss.

On September 29, the Advisory Group on Mathematics and AI (AGMAI) released special guidelines, emphasizing that AI research results that have been fully understood by human mathematicians should be published as papers in accordance with traditional academic norms and undergo journal peer review.

Mathematical research has a set of rules that have been followed for many years. After a proof is written, it must go through expert checks and peer discussions before it can gradually become knowledge recognized by the academic community.

AI companies can happily announce that they have conquered another world-class problem, but whether the mathematical community recognizes it is another matter.

Some mathematicians even don't want AI to touch the reviewer's chair at all.

On October 7, an organization called the Association of Human Mathematics (AHM) publicly criticized OpenAI for releasing more than 700 mathematical research documents at one go, arguing that this was more like a show of force than serious academic work.

The organization even explicitly required its members not to publish AI-generated mathematical papers and reviews, not even review reports.

Just as mathematicians were working hard to set rules for AI, two Harvard researchers put human and AI reviewers together in the examination room.

As we can see from the results, AI has not only begun to challenge humans' ability to write papers, but now it is even better than many professional reviewers at finding flaws in papers.

Of course, the Harvard study mainly involves social science papers, and it cannot yet prove that AI is capable of independently reviewing complex mathematical proofs. It is not wrong for mathematicians to require strict verification.

But at this point, the development of the matter is somewhat absurd.

Mathematicians are still emphasizing that AI cannot bypass the threshold of human peer review.

But AI has already started researching whether it can replace the gatekeepers themselves.

This article is from WeChat Official Account "Letter AI", author: Yuan Xinyue, published with authorization from 36Kr.