OpenAI researchers: None of us read papers anymore

Do people in cutting-edge labs barely read academic papers anymore?

A researcher at OpenAI criticized three top conferences in one sentence: There is far too much exaggeration and falsification!

The incident stemmed from an ICLR paper whose performance was so outstanding that it seemed implausible. After investigation, netizens found the hidden "secret trick" behind it.

Has publishing papers at top conferences "evolved" to this point?

First check whether the code is open source, second check if there are any "unscrupulous tricks" hidden in the experimental methods, and third verify whether the generated results can support the core conclusions — after layers of such scrutiny, how many papers at top conferences can actually stand the test?

Someone actually did exactly that.

Only 8 out of 105 papers are "qualified"

In July this year, SAI, co-founded by Tan Chenhao, associate professor of computer science and data science at the University of Chicago, announced a large-scale "experimental peer review" campaign.

Their target was all 168 Oral papers of ICML 2026.

ICML received a total of 23,918 submissions this year, and only 168 papers finally obtained the Oral qualification, accounting for about 0.7%.

In other words, what SAI inspected was already the top-tier papers screened out from more than 20,000 submissions.

In addition to reading the methodology, experimental design and results of papers like traditional reviewers, SAI Review also downloads codes, models and data, configures the operating environment and runs experiments, and finally compares the running results with the original paper item by item.

SAI reviewed all 168 Oral papers, of which only 104 had open source codes, and they finally completed full reproduction of 105 papers.

Among them, only 34 papers reproduced more than 40% of the claims; only 8 papers had a reproduction ratio exceeding 80%.

No matter each paper contains at least 1, 3 or 5 verifiable claims, the median reproduction score always stays between 28% and 30%, and the overall distribution barely changes.

After excluding experiments that failed to run, terminated early or exceeded hardware capabilities, the median reproduction score is still only 42% to 50%; when each paper contains at least 3 verifiable claims, the median remains stable at 42%.

They encountered all the most common problems in reproduction: unrunnable codes, missing files, incomplete instructions, corrupted dependencies, or the results generated by the codes do not match the content stated in the paper.

There are also 4 papers relying on models that have been taken offline. Even if subsequent researchers are willing to spend money and time, it is impossible to get the same results again.

SAI also listed two more specific examples.

One paper took "only training 0.77% of the parameters of the base model" as its main selling point, but the actually released checkpoint trained 6.31% of the parameters, which is about 8 times the claimed number.

In another paper, a reliability table scored by a judge model was presented, but the judge model was nowhere to be found in the open source code, nor was there any script that could calculate the numbers in the table.

Low-quality papers bring high returns with low risks

The reproduction results of SAI can be described as quite unsatisfactory.

Worse still, the problems found here are only those that "can be discovered under certain conditions".

Those papers without open source codes are directly "unassailable".

Even if the code is fully open, verifying a paper will cost a huge amount of time, energy and money, and the final result is still unsatisfactory, which is extremely frustrating.

According to SAI's estimation based on the public on-demand price of Google Cloud, the median cost of fully re-running an ICML Oral paper is about 8,900 US dollars. Among the 105 papers, 17 cost more than 100,000 US dollars, and the most expensive one is close to 2.2 million US dollars.

The more a paper relies on large-scale computing power, the more difficult independent reproduction will be.

Without open source code, no one can check the results; even with the code, not everyone can afford the cost.

As a result, a problematic paper can easily pass the review, get citations, be added to the resume, and finally bring admission, faculty position or job offers in top labs.

Even if someone occasionally finds the results inconsistent, conferences rarely conduct re-reviews, and the papers are not necessarily retracted.

This is exactly what the comment section calls "producing poor research brings profits with almost no cost".

People stop reading papers, but papers are still required in recruitment

In the field of large models, it is true that many truly important advances no longer appear in the form of academic papers.

As an engineer at OpenAI standing at the forefront of the industry, it is not difficult to have such feelings. After all, with more sufficient computing power, faster experimental feedback and a large number of undisclosed internal results, they have the confidence to "not read papers".

But it is somewhat subtle to say so publicly.

A comment satirized that people in tech companies doing this is "pulling the ladder after climbing ashore": they recruit researchers from universities, build on the publicly accumulated achievements of the academic circle, snatch talents and GPUs with higher salaries, reduce their own acceptance of peer review more and more, and finally turn around to claim that academic research is "mostly a scam".

It is a little harsh, but indeed reasonable.

People in these cutting-edge labs can "not read papers", but those who want to get in still need to publish papers first.

For students and young researchers who lack industry experience, top conference papers are still the most direct proof of research ability.

Applying for a PhD position, looking for a faculty job, or entering a cutting-edge lab like OpenAI, all rely on papers to get attention.

People in the industry despise papers after entering large labs, but they still use the number of papers, conference levels and citation performance to screen people outside the door.

As a result, papers are in an awkward position: their credibility in knowledge dissemination is questioned, but their value in talent competition has not diminished at all.

Therefore, the statement that "large labs no longer read papers" sounds a little lofty and arrogant. Because people who are really qualified to not read papers have already crossed that threshold with the help of their own papers.

Of course, not all students are academic villains who deliberately design scams.

A university researcher joked that he even hoped his students already had the ability to write a paper full of exaggeration or even falsification.

Some people are "striving to be academic scammers", while others are still struggling with LaTeX.

Reference links:

https://x.com/MathewShen42/status/2084465434506768867

https://x.com/kellerjordan0/status/2084721463089902074

https://sai.science/blog/how-much-science-is-verifiable

https://x.com/mengyer/status/2085134204786921886

This article is from the WeChat official account "Ji Qi Xin Zhi" (ID: almosthuman2014), author: Ji Qi Xin Zhi, 36Kr is authorized to republish it.