The RSI invested by Jeff Dean has produced tangible results, and Kimi K3 ranks first among domestic products in self-evolution performance.

Alright, the new RSI benchmark developed by hillclimb, the AI startup invested in by our "brother-in-law" Jeff Dean, has finally released its results!

The new benchmark, named DeepRSI, is specifically designed to test the recursive self-improvement capabilities of various large models. And guess what the results are —

Claude Fable 5.1 took the first place.

GPT-6 Astra followed closely right behind it.

Moonshot AI's Kimi K3 rushed to the third place, meanwhile ranking first among domestic models and first among open-source models, outperforming Grok and Gemini ~ (Google: Do you really have to call me out?)

The test mechanism is also extremely hardcore —

AI is directly assigned a 48-hour independent scientific research challenge, where it writes code, runs experiments, and modifies algorithms on its own to see how far it can optimize the research outputs.

There are a total of 15 questions, all sourced from real machine learning research. This time RSI has finally got its strictest evaluator ever. (Laughs)

48 hours… RSI… Jeff Dean… viral repost… everyone… hurry up… keep reading…

What results can AI possibly achieve after the 48-hour RSI marathon test?

First, let's talk about what DeepRSI, which was enthusiastically reposted by Jeff Dean, is all about.

To put it simply, it is a new benchmark specifically built to test AI's independent scientific research and self-improvement capabilities, measuring to what extent AI can independently complete real machine learning research tasks.

In fact, most similar benchmarks in the past adopted short-term tasks designed by researchers, with a test duration of usually around 12 hours.

But DeepRSI directly extends the test period to 48 hours, and replaces the test questions with ongoing real machine learning research. (Thrilling.jpg)

Specifically, there are 15 questions in total covering four major directions —

Deep learning training scheme optimization, GPU kernel acceleration, estimator accuracy improvement, and algorithm optimization such as vector search.

The test rules are also extremely strict!!

All models must complete the tasks with the highest inference intensity, using the unified Pi Agent framework in an offline network environment.

During this period, the AI needs to propose solutions, write code, and run experiments on its own, and keep making adjustments based on the results to achieve better research metrics.

As for the scoring mechanism, DeepRSI also adopts a dual system of "public verifier + private verifier".

The former allows AI to repeatedly test and optimize itself; the latter is responsible for final acceptance, which largely prevents models from overfitting to public tests or exploiting loopholes in the scoring rules.

Finally, the performance of each model is converted into an Elo score through the Bradley-Terry model to determine the final ranking.

At the end of the day, what this test aims to verify is whether AI can, just like human researchers, continuously find better solutions through 48 hours of repeated trial and error.

Judging from the final performance, the performance gaps between different models are quite obvious.

Looking at the overall ranking first, Claude Fable 5.1 took first place with 1548 points, and GPT-6 Astra followed right after with 1232 points.

Kimi K3 ranked third with 820 points, outperforming Grok 4.7 and Gemini 3.8 Flash, and becoming the highest-ranked open-weight model in this test.

Interestingly, there are two more points worth noting on this ranking list —

First of all, when AI conducts scientific research, the longer the task lasts, the more likely the final ranking will be reversed.

According to official tests, GPT-6 Astra once took the lead in the early stage of the competition, but as the scientific research task continued, Claude Fable 5.1 gradually overtook it.

As for Grok and Gemini, they each have their own advantages in different tasks, among which Gemini performs better in GPU acceleration and kernel optimization.

There is also a noteworthy cost-benefit calculation here.

According to the public DeepRSI ranking, the listed running cost of each model is as follows:

Claude Fable 5.1 is around 627 USD, GPT-6 Astra around 404 USD, and Kimi K3 around 159 USD.

Kimi K3 ranks top three, with a listed cost of only about 159 USD, which is roughly a quarter of Claude's cost.

The cost here reflects the overhead of test runs. Since different models take different exploration paths, it cannot be directly equated to the input-output ratio in real scientific research scenarios.

But it provides a new dimension worthy of continuous observation —

In the future, when measuring the scientific research capability of AI, in addition to the number of outputs, we also need to look at how much effective research progress can be obtained for every dollar spent and every hour of operation.

In terms of Token consumption, there is also an interesting contrast: Claude Fable 5.1 took the first place with an average consumption of about 47 million Tokens per attempt.

GPT-6 Astra only consumed about 7 million Tokens and got the second place.

Kimi K3 ranked third with about 36 million Tokens.

Then the most exaggerated one is Gemini 3.8 Flash, which consumed more than 100 million Tokens but finally ranked fifth...

In addition, the smarter the model is, the more likely it is to exploit loopholes in the evaluation rules.

The research team released a special case of GPT-6 Astra.

In a color swap puzzle training task, Astra initially tried to train Transformer to imitate a traditional solver.

After finding that the training effect was not ideal, it surprisingly redesigned the technical solution and encoded the solving algorithm directly into the model weights.

To achieve this, Astra also used residual streams as working memory, constructed attention heads and logic gates, and finally wrote a set of manually designed algorithms into the neural network structure.

This case was included in the research team's discussion on reward mechanism loop exploitation.

AI does demonstrate complex engineering problem-solving capabilities, but the optimized output it generates may also deviate from the original goal that researchers expect to achieve.

From AI-assisted scientific research to AI conducting scientific research on its own

The AI startup hillclimb behind DeepRSI is also worth mentioning.

Founded in 2025, the company was selected for the fall batch of Y Combinator that year, and currently has only 4 team members.

Its main business is to cooperate with cutting-edge AI labs, and help AI Agents gradually gain the ability to conduct independent scientific research through high-quality training data and reinforcement learning environments.

The company's investor lineup is also very luxurious. Jeff Dean, the former chief scientist of Google, is one of the investors himself.

It is also worth mentioning that Jeff Dean co-founded the AI startup Discovery Loop, whose research direction is exactly RSI~

In addition, there are YC founder Paul Graham, Replit CEO Amjad Masad, and angel investors from top AI labs such as OpenAI, Anthropic, DeepMind, xAI and more.

What's more interesting is that the two co-founders of this company both have a background in the professional esports industry.

Jun Park used to be the coach of a Valorant team in his early years, then switched to AI research, and later served as a research engineer at Amazon and Google DeepMind.

In August 2025, he left DeepMind, where he had only worked for about 3 months, to found hillclimb:

The other co-founder, Ibrakhim Ustelbay, used to be a Fortnite esports coach in his early years.

After that, he worked as a software engineering intern at Microsoft and Coinbase, and also founded Headstarter, an AI interview platform with more than 100,000 users.

In August 2025, he co-founded hillclimb with Jun Park and left the company in March 2026.

From leading esports teams to leading AI to do scientific research, the career transition of these two is quite a cross-field success story...

Of course, DeepRSI itself is still in the early stage.

With only 15 research tasks and 5 sets of model configurations, each model running only twice, there is still a long way to go to fully measure AI's independent scientific research capability.

What's more, the task objectives, experimental environment and scoring standards are still set by humans, which is far from the stage where AI can truly achieve recursive self-improvement.

But this does not prevent RSI from becoming the new top trend in the AI industry this year —

From Karpathy's AutoResearch, to AutoResearchExam, and now to DeepRSI, the matter of AI doing scientific research on its own has evolved from conceptual discussions all the way to the era of public leaderboards.

That's pretty good. Before the RSI intelligence explosion arrives, the AI industry has already launched its own "evolution competition".

Google: We were born at the wrong time...

References:

[1]https://www.deeprsi.com/blog/

[2]https://x.com/jparkjmc/status/2109684519896780938?s=20

This article is from the WeChat official account “QbitAI”, author: Focus on cutting-edge technology, published with authorization from 36Kr.