AI generates its own questions for self-training, posts a whopping 11% increase in two months, and overtakes Opus by 4.8 in coding self-testing.

Training a capable model, the most expensive part is not the GPUs, but the training problems.

Especially when it comes to coding and agent scenarios, mere questions and answers are far from sufficient: you also need to prepare a fully functional running environment for the model, a test set that can accurately judge correctness, and take measures to prevent the model from copying answers from old submissions.

Nor can the difficulty level be set arbitrarily.

If the problems are too easy, the model can finish them effortlessly without learning anything; if they are too difficult, the model cannot solve any of them and cannot accumulate valid correct experience, which means the computing power consumed is completely wasted.

Even if all these conditions are met, you still need a sufficiently large volume of training data.

In the past, this work was always done by humans. It is not only slow and costly, but more importantly, it cannot keep up with demand: as soon as you finish sorting out a batch of problems, the model will complete them immediately.

Then what if we let the model generate problems for itself?

Some teams have actually done this.

In August this year, Ornith released three sets of weights at once: 397B MoE, 35B MoE and 9B Dense.

A never-before-seen link has been added to the training pipeline: the model generates training tasks on its own, builds problem-solving scaffolding by itself, runs problem-solving trajectories independently, and all three parts are fed into the reinforcement learning process together.

The task of generating training problems has been taken over by AI from human hands.

The figures have immediately yielded positive returns.

On Terminal-Bench 2.1, the 1.0 version released two months ago scored 77.5, while the 397B model in this release reached 86.1. It has increased by 8.6 percentage points in just two months.

In the same comparison table, Ornith lists the score of Claude Opus 4.8 as 85.0.

A MIT-licensed weight that can be downloaded freely has ranked ahead of closed-source flagship models on this benchmark.

1.0 Learns How to Solve Problems

1.5 Learns What to Practice

Ornith-1.0 released in June this year already took a rather radical step: it turned the agent scaffolding, the workbench that AI uses to solve problems, into an object that can be learned.

The common practice is that human engineers design a complete set of scaffolding for a specific type of task: how to provide tools, how to split tasks, how to retry after errors, all of which are hard-coded, and then the model is trained repeatedly within this fixed framework.

Ornith 1.0 broke this fixed framework and handed it over to the model to build by itself.

Each reinforcement learning step is divided into two phases: first, based on the task and the scaffolding used in the previous round, the model modifies and generates a new version of scaffolding; then it runs the problem-solving trajectory on this new version of scaffolding. The reward is passed back from the trajectory, and both phases can obtain the corresponding reward.

The two-stage self-improvement framework of Ornith-1.0. First modify the scaffolding, then run the problem-solving trajectory, and the reward is passed back to both stages.

The model has learned to build its own workbench. But the tasks are still assigned by humans: where the training problems come from was not touched in version 1.0.

Version 1.5 has taken over this part as well.

Given a code base, a high-level description of the task type, plus the model's own past problem-solving records, the system first generates a batch of new problems, specifically selecting problems that are one level more difficult than those it has already solved, to target its own capability gaps.

After the problems are determined, the model generates or improves exclusive scaffolding for each problem: instructions, tools, disassembly strategies, and orchestration logic.

Finally, under the dual constraints of the task and the scaffolding, the model runs the complete problem-solving trajectory.

The reward is passed back in reverse from the trajectory to all three stages, which are jointly optimized with GRPO.

The key point is that the half responsible for generating problems also needs to be scored. If the generated problem is of poor quality, the problem-generating part will also get corresponding penalty points.

The full loop of problem generation, scaffolding construction, and problem solving is closed at this point.

The stronger the strategy, the more difficult and informative the generated problems will be; the evolution of scaffolding can further bring out the full potential of the model; the improvement of trajectory quality in turn feeds back stronger learning signals to the model.

In the past two years, open-source models have basically followed two paths to catch up with closed-source flagship models: scaling up parameters, and copying existing training recipes.

Both paths are essentially copying answers according to the test papers made by others. Ornith no longer wants to follow this path. It aims to let AI generate its own test papers.

Self-generated Problems

How to Ensure They Are Not Useless Problems

This sounds very promising, but you can quickly find a loophole: what if the model deliberately generates extremely easy problems that guarantee full marks for itself?

The model will always optimize towards the direction with higher rewards. A problem generator that only generates problems like "what is 1 plus 1" can easily get extremely high scores, but will not produce any effective training signals at all.

Ornith's solution is to split the task reward into three signals: validity, frontier difficulty, and novelty, and multiply the three values together.

The multiplication operation is the key. If any of the three values is close to zero, the total reward of the entire problem will return to zero. To get the reward, all three conditions must be satisfied at the same time.

Validity is used to judge whether the problem is valid:

Check whether the scaffolding can run normally, whether a high-confidence correct solution can pass the test, whether obviously wrong solutions will be judged as failed, and whether the scoring logic matches the task description.

The team even regards validity as a hard threshold: unqualified problems will be directly judged as zero reward, specifically to block those useless problems that seem very difficult but cannot judge the correctness of solutions at all.

The design of frontier difficulty is a highlight of the entire system.

The system samples multiple trajectories for each problem and calculates the empirical success rate. The closer the success rate is to the target value of 0.2, the higher the corresponding reward will be.

In other words, for a good training problem, the model should successfully solve it twice out of ten attempts.

If the model succeeds eight times, it means it has already mastered the problem and the problem has no training value; if the model cannot succeed even once, it cannot accumulate any successful trajectory that can be learned, and the reinforcement learning process will also run in vain.

The generated problems must be exactly at the position where the model is just one step away from solving them.

What is more clever is that this target will change dynamically with the improvement of the model's capability: when the model becomes familiar with a certain type of problem and the success rate increases, the reward will automatically decrease, so the problem generation end can only continue to increase the difficulty. The entire difficulty curve will automatically climb without manual adjustment.

Novelty ranks third with the lowest weight, and it is only responsible for one thing: do not keep generating variants of the same problem.

The official notes specifically that its priority is lower than the first two indicators, and its responsibility is to remove redundant problems rather than rewarding overly tricky problems.

Therefore, what this "self-improvement" mechanism actually improves is the automatic evolution of the training curriculum.

The entire process runs during the training phase.

The weight file you download from Hugging Face is static, and it will not modify itself while running tasks on your laptop or mobile phone.

Two Different Sets of Evaluation Criteria

First let's clarify how the score of 86.1 is obtained.

This Terminal-Bench 2.1 score is run by the Ornith team in their own environment with their own configurations, and the official document states that the result is the average of five independent runs.

Terminal-Bench 2.1 itself has an official verified leaderboard, and every result on the leaderboard will be re-run and verified by the leaderboard team members before being published.

As of August 19, 2026, this leaderboard contains a total of 17 evaluation items, and Ornith-1.5 is not included in it.

Top 8 of the official Terminal-Bench 2.1 leaderboard. Opus 4.8 ranks 5th with 78.9% score, and Ornith-1.5 is not included in the 17 evaluation items.

What is more important is the comparison objects.

The 85.0 score of Opus 4.8 in Ornith's table is the value they ran for Opus 4.8 using their own Terminus-2 framework. While on the official leaderboard, the verified score of Claude Code + Opus 4.8 submitted by Anthropic is 78.9%, ranking 5th; the top score is 83.8% achieved by Claude Code + Fable 5.

For the same Opus 4.8, there is a 6.1-point gap between the two tables.

The difference lies in who is conducting the test and what configuration is used for the test.

Agent benchmarks like Terminal-Bench measure the combined performance of the model, scaffolding, timeout settings, context window and resource configuration.

If you change a set of configurations, the meaning of the score will change accordingly.

Therefore, the leaderboard page clearly states one submission rule: no modification to timeout or resource configuration is allowed, which is designed to prevent participants from artificially boosting scores by adjusting configurations.

The self-test configuration disclosed in Ornith's blog is 4 hours of timeout, 32-core CPU, and 48GB of memory.

They also modified other configurations, including the timeout and resource settings that the official leaderboard explicitly locks.

These modifications may not be intended to artificially boost scores, but the premise of comparability no longer exists: the project's self-test results cannot be placed in the same ranking system as the official verified leaderboard.

The license also needs to be clearly explained.

The MIT license only applies to the model weights. The public GitHub repository currently only contains README, LICENSE and display materials, and there is no complete training implementation of version 1.5, no training task set, and no reproducible evaluation script.

Open weights do not mean the full open source of the entire project.

The Most Interesting Parts Are Actually the 35B and 9B Versions

Flagship models are responsible for attracting public attention, but medium and small size models determine how many users can actually use them.

In this release, the two medium sizes are the most easily overlooked parts.

The 35B-A3B is a MoE model, with only about 3B parameters activated for each token.

With this activation parameter volume, it reaches 67.8 on Terminal-Bench 2.1, which is significantly higher than 42.1 of Gemma 4-31B and 51.7 of Muse Glimmer-30B. It also scores 79.0 on SWE-bench Verified, compared with 52.0 of the other models.

Using only 3B activated parameters to beat a 31B dense model is the most cost-effective model in this release.

The smaller 9B Dense version is clearly designed for edge deployment scenarios.

Its self-test scores are 46.2 on Terminal-Bench, 70.6 on SWE-bench Verified, and 86.4 on GPQA Diamond.

The 5.63GB compressed version has been released on Hugging Face, and the MLX format for Apple computers is also available.

But the 9B model does not outperform Gemma 4-31B in all aspects.

It does take the lead in coding and reasoning tasks, but it scores 54.2 vs 55.0 on MCP-Atlas, and 41.2 vs 52.8 on Toolathlon-Verified.

These two tasks test multi-round tool calling, processing returned results, and adjusting plans according to status. When the task chain becomes long, the 9B model's capacity cannot support the corresponding requirements.

The gap in model size can be made up for by better training methods in coding and reasoning tasks, but cannot be made up for in tool calling scenarios.

The Next Round of Competition

May Take Place in Training Problem Libraries

The 1.0 version released in June has a self-test Terminal-Bench score of 77.5. The 1.5 version released in August reaches 86.1.

It has improved by about 11.1% in just two months.

What is more important than this number is how the corresponding capability improvement is achieved.

Version 1.0 learns "how to solve problems", while version 1.5 starts to learn "what problems to practice". The entire data production process has been moved into the closed loop of reinforcement learning.

High-quality agent training tasks have always been a scarce resource.

The speed of manual data sorting has long been unable to keep up with the speed at which models consume training problems. This mechanism is designed to solve this very specific engineering problem.

The way open-source models catch up with closed-source flagship models is changing.

In addition to scaling up parameters and copying existing training recipes, there is a new path: generate training problems by yourself. Whoever can continuously generate high-quality problems will hold the fuel for continuous capability improvement.

The three-stage closed loop of Ornith-1.5. Generate tasks, generate scaffolding, run problem-solving trajectories, pass rewards back to all three stages, and update all modules uniformly with GRPO.

In addition to competing for computing power