The first round of ICLR 2027 reviews has quite a number of frustrating points that draw widespread complaints.

In recent days, the first-round review comments of ICLR 2027, the top conference in machine learning, have been gradually made public.

For paper authors, this is usually a rather nerve-wracking moment. Months or even one to two years of research work are finally about to undergo peer review. Reviewers will carefully examine the methods, experiments and theoretical derivations, and point out the real problems existing in the research.

But this year, when some researchers opened OpenReview, the review comments they saw were rather bewildering, a mix of amusement and frustration.

The peer review of ICLR 2027 has instantly turned into a large-scale venting session in the AI community.

Some asked authors to explain standard deviation, others directly pasted AI prompts on their comments

Let's start with Hanchen Li from the University of California, Berkeley.

He posted that he received an ICLR review comment, in which the reviewer asked them to explain what "standard deviation" means.

He also joked that it felt like he was back in high school math class again.

The original post does not show the full context of the paper and the review, so it is impossible to judge whether the reviewer wanted the authors to supplement the statistical definition, or had doubts about the specific usage in the paper.

But this post quickly sparked widespread discussions anyway.

The situation demonstrated by another researcher Ravid Shwartz Ziv is even more comical. He posted the comments from three reviewers.

The first one wrote: "The paper is well-written and addresses an important problem. However, the authors should consider additional baselines." The second one repeated this sentence almost word for word.

When it came to the third reviewer, the tone changed abruptly: "Certainly! Here is a review of the paper", and the formal review content only started after that. Even the common conversational opening line of AI-generated text was not cleaned up.

The two comments looked like they were copied from the same template, while the third one left even more obvious traces of AI generation. What made the authors more helpless was that all these comments eventually entered the official peer review process.

Another researcher Konstantin Grotov encountered a problem related to AI-assisted writing. He said that his ICLR submission received a review comment that mainly targeted the use of AI.

The problem is that the paper had already disclosed the use of AI-assisted writing as required.

Grotov later explained that reviewers can certainly raise questions about the use of AI, but they should put forward specific technical arguments to explain how this use affects the quality, research results or credibility of the paper. It is obviously not sufficient to use the mere fact that the paper used AI as a reason for rejection.

The next case is more subtle.

Researcher Artur Chakhvadze shared a review comment, in which the reviewer frankly admitted that he did not have sufficient professional background to properly evaluate this paper.

Chakhvadze expressed approval of this instead: at least this reviewer honestly acknowledged the boundaries of his own competence. He also complained that he had seen many reviewers who did not even understand the basic knowledge of the paper's field, but still gave a high confidence score of 3/4.

This post sparked different opinions. Some people think that reviewers who voluntarily state that they are not familiar with the research direction are far more responsible than pretending to understand and making up review comments randomly. Others asked: since the professional background does not match, why was this paper assigned to him in the first place? Chakhvadze later added a sentence: the reviewer himself may not be at fault, and what is really questionable is the process of assigning and managing reviews.

After reading several posts, we can see that these are different types of problems: some reviewers did not fully understand the paper, some may rely on AI to generate comments, and others were assigned to research directions they are not familiar with. All of these ultimately point to the same question: can the review comments authors receive really help make academic judgments?

AI is good at finding flaws, but may not know which flaws are worth rejecting a paper for

In this discussion, one point of view is particularly noteworthy.

Jiaxin Pei from the University of Texas at Austin pointed out that the most troublesome reviewing problem he encountered in 2026 was that AI reviewers seemed unable to distinguish between the "limitations" of a paper and the "critical issues" that fundamentally affect the validity of the paper.

He believes that large language models can easily find hundreds of limitations in a paper, even for the best papers in history. But this does not mean that these papers should not be accepted.

This statement points out a very important shortcoming when AI participates in peer review. A scientific research paper can hardly be all-inclusive. The scale of experiments may be limited, some scenarios are not covered, the method may rely on specific assumptions, and the theoretical analysis may also have a scope of application. These are all normal research limitations.

What really determines whether a paper is valid is whether these limitations shake its core contributions.

Reviewers can certainly ask for more datasets, larger model scales and more control experiments. But if the existing experiments are sufficient to support the core conclusions of the paper, additional experiments may not constitute a reason for rejection. Conversely, if there are loopholes in the most critical experimental design of the paper, or irreparable errors in the theoretical derivation, even if other parts are written perfectly, the paper may not be valid.

What is needed here is the ability to judge the importance of problems. Large models are prone to mixing these two things together. They can keep listing potential problems, but may not be able to judge which problem really affects the research conclusion. As a result, review comments may get longer and longer, more and more supplementary experiments are required, and the core issues that determine the quality of the paper are drowned out instead.

This post by Jiaxin Pei resonated with many researchers.

Some people pointed out that this problem is penetrating into the peer review mechanism itself. Nowadays, at every AI academic conference, there seems to be a group of reviewers: they give low scores to papers on the one hand, and ask authors to supplement new experiments that are irrelevant, or only make minor additions to existing experiments on the other hand.

Others put forward a very direct judgment standard: if this problem is fixed, will the core conclusion of the paper change? If the answer is no, it may just be an ordinary limitation, not a critical defect enough to negate the whole paper. This is exactly the most needed ability in peer review: knowing what is worth paying attention to, and also knowing what is acceptable.

Conclusion

If we put these cases together, we will find that the controversy of ICLR this time is not simply a problem of reviewer competence.

The larger background is that AI is changing both the production end and the evaluation end of academic research at the same time.

Peer review itself is a work that highly relies on professional knowledge and time investment. The speed at which the number of papers grows cannot necessarily be caught up by the number of professional reviewers. The ICLR 2027 program chair team previously specifically explained this issue. On September 2, the ICLR 2027 program chair team published an official article analyzing the new pressures faced by peer review in the AI era.

https://blog.iclr.cc/2026/09/02/submission-policies-for-iclr-2027/

The chair team pointed out that the machine learning field has grown rapidly in recent years, the number of new entrants far exceeds that of senior researchers, and the number of people with professional review capabilities may not keep up with the growth of submissions. Generative AI has further lowered the threshold for paper production. Even people who are not familiar with a certain research field can use AI to generate a manuscript that looks like a research paper in appearance. At the same time, it has also become a new problem to be solved that reviewers delegate review work to AI.

To ease the shortage of review resources, ICLR 2027 has launched a series of measures, including limiting the number of submissions and strengthening the reciprocal review system.

Review obligations have also been further tightened: authors whose names appear in 3 or more submissions must complete at least 6 paper reviews. In addition, each submission requires at least one eligible author to register as a reviewer and commit to reviewing at least 3 papers. If none of the authors are eligible, this requirement can be waived.

In terms of AI use, ICLR does not completely prohibit reviewers from using large models, but allows limited and responsible use of AI to assist in writing review comments and meta-reviews. But this has clear prerequisites: reviewers must disclose the use of AI. If a large model is used to generate or modify any part of the review comment, they must also submit their initial independently written paper evaluation, as well as the full interaction records with the large model. For multi-turn conversations, the input content of each turn must also be provided.

ICLR also clearly stipulates that if an AI-generated review contains false information, hallucinations or misstatements, or the review itself is of low quality, reviewers may be subject to disciplinary action for failing to fulfill their reciprocal review obligations. Penalties may include directly rejecting all submissions in which they participate as authors, or restricting their participation in conference submissions in the future.

Of course, only the first-round review comments of ICLR are currently made public. The open discussion between authors and reviewers will last until November 18, and the final acceptance decision is expected to be announced on December 16. Authors still have the opportunity to question incorrect reviews, and reviewers can also modify their scores based on new information.

References:

https://x.com/mbilgehanertan/status/2118760558854070637

https://x.com/jiaxin_pei/status/2118714662892327158

https://x.com/norpadon/status/2118896999709684071

https://x.com/lihanc02/status/2118843054975336646

https://x.com/kgrotov/status/2118695897890804071

https://x.com/ziv_ravid/status/2118718086920413494

This article is from the WeChat official account "Ji Qi Xin Zhi" (ID: almosthuman2014), author: ICLR followers, authorized for release by 36Kr.