What happens when AI starts to improve AI? | The Hyper-evolution Intelligent Automated AI System has refreshed 3 SOTA records.

Today, every advance in state-of-the-art models still relies on the judgments made by human researchers at three critical stages: deciding what is worth scaling before training, searching for more efficient algorithms and solutions during training, and continuously improving the efficiency of the underlying systems that support both.

If these tasks can be completed autonomously and recursively by AI, the pace of progress will change fundamentally: every verified improvement can be turned into a stronger starting point for the next round of research.

Based on this idea, we have built the automated AI research system of Apex Intelligence, hoping to enable AI to independently discover and verify improvement solutions across the entire training technology stack.

In the first systematic evaluation, our AI system achieved three new official SOTAs on SLDBench SimpleTES, GPUMode TriMul (H100) and MLS-Bench Fused Causal Attention. Among them, the TriMul result surpassed TTT-Discover co-developed by researchers from Stanford, NVIDIA and other institutions; on NanoChat Autoresearch, our B200 result also exceeded the publicly available scores of Recursive SuperIntelligence and Tencent Hunyuan Hyra, approaching the public SOTA for fixed-budget LLM training.

Overall, this automated recursive research paradigm has reached the level of top human engineering teams and current advanced automated systems in three directions: scaling law prediction, fixed-budget language model training, and GPU Kernel performance optimization, and has outperformed them on some tasks.

In each evaluation, the AI system will independently complete a full round of research: identify problems worthy of improvement, propose and test solutions, verify results, and learn from experimental evidence. The verified cutting-edge results will become a stronger starting point for subsequent research, turning improvements from single-round optimization into recursive cycles.

01 

SLDBench SimpleTES

Before large-scale training is officially launched, researchers must first answer a question that cannot be exhausted by direct experiments: how will model performance change when the scale expands to a range that cannot be directly tested? The Scaling Law serves exactly this purpose, as it directly affects key decisions such as model scale, data composition, learning rate, batch size and computing budget. If a law can only fit small-scale experiments but cannot accurately extrapolate to larger training scales, the entire training budget may be led in the wrong direction.

SLDBench translates this uncertainty into a quantifiable automatic discovery task. Built on more than 5,000 published LLM training experiments, it requires the AI system to provide both symbolic laws and fitting methods, and maintain accuracy in scale ranges that are not involved in fitting. According to the four-task evaluation adopted by SimpleTES, we tested parallel scaling, domain-mixture scaling, learning-rate/batch-size co-scaling and U-shaped compute scaling.

Using the same fixed version of the public evaluator, our AI system outperformed the previous public reference results on all four tasks, with the average score of the four tasks increasing from 0.8613 to 0.8846, an increase of 2.71%. Among them, U-shaped scaling saw the largest improvement, with the score rising from 0.4975 to 0.5761, up 15.80%; learning-rate/batch-size co-scaling increased from 0.9506 to 0.9650, up 1.51%, and parallel scaling and domain-mixture scaling were also further improved. Since the scoring is based on the performance on the held-out scale range, the focus of this evaluation is whether the law can continue to hold outside the known experiments, rather than just fitting the already observed data.

Figure 1: Our AI research system outperformed previous public reference results on all four selected SLDBench tasks, raising the average held-out set score from 0.8613 to 0.8846. The most significant improvement was seen in U-shaped scaling. Higher scores indicate better performance.

The most representative methodological innovation appeared in U-shaped compute scaling. Our AI system did not treat the entire curve as a single trend, but split it into two parts: one is the long-term scaling trend that determines large-scale performance, and the other is a phased effect that only appears within a limited computing range and then gradually decays. In this way, temporary trend reversals in the experimental data will not be incorrectly extrapolated as long-term changes that will continue indefinitely.

The value of this distinction is mainly reflected in extrapolation. The evaluator will hide the last stage of the curve. Although a sufficiently flexible equation can fit the observed inflection point, it may give a completely opposite direction when continuing to extrapolate. By modeling the occurrence, duration and decay process of the phased effect separately, the AI system can more accurately predict the trend after the turning point.

This decomposition method improves the prediction accuracy of the held-out range while retaining good interpretability and testability, so it has clear technical significance. It also provides a research hypothesis that may be applicable to other Scaling Law problems: The observed trend reversal does not necessarily mean that the long-term scaling law has changed, and it may only be a temporary phased effect superimposed on the long-term trend.

02 

NanoChat Autoresearch

A fixed computing budget forces every new idea to undergo the most direct test: when training time and hardware resources are exactly the same, performance improvements can only come from more efficient use of this budget. Therefore, NanoChat Autoresearch provides a clear test scenario to judge whether an AI system can truly improve model training implementation through rapid experiments and autonomous iteration.

NanoChat Autoresearch was released by Andrej Karpathy in March 2026. It provides an Agent with a GPU and allows free modification of the model architecture, optimizer, hyperparameters and training code. Each experiment lasts only five minutes, and is finally scored according to validation set bits per byte (val_bpb), with lower values indicating better performance. Since then, this mechanism has been extended to autoresearch@home, where human participants and AI agents can share experiments and continuously challenge the current strongest public solutions.

Under the setting of a fixed 300 seconds and a single B200 GPU, our AI system achieved 0.892426 val_bpb, which is close to the current public cutting-edge level. This result surpasses the best single result 0.903891 announced by Recursive SuperIntelligence in June 2026 and the average score of 10 random seeds 0.9108745, and also exceeds the Hyra score 0.901543 released by Tencent Hunyuan in July 2026. As of September 2026, it is close to ScienceGuru (0.889522) from AutoTrust AI, the leader of the public leaderboard.

Figure 2: Validation BPB comparison under a fixed 300-second, single B200 training budget. Lower values indicate better performance.

The key to this improvement is to make the sparsity of n-gram truly run through the entire training calculation process. ScienceGuru has proven that large-scale n-gram memory can improve training effects, but each batch actually only accesses a small number of rows in these large tables. Our AI system further found that part of the gradient calculation in the training link is still performed in a dense manner, so that although the model is sparse in access, the system level still bears the cost of large-scale dense calculation.

To solve this problem, the AI system designed a more compact backpropagation process that only updates the rows that are actually accessed by the current batch. After the modification, the peak training video memory decreased from about 177.7 GB to 140.6 GB, a drop of 20.9%, while still maintaining model quality close to SOTA. Within the same 300-second budget, the number of tokens processed by the AI system is about 4.5% less than the ScienceGuru implementation, and it still achieves 0.892426 val_bpb, indicating that training quality close to SOTA does not depend on higher token throughput.

The AI system then extended this principle to the complete data access path: fusing two n-gram table queries, concatenation, active row identification and row registration into one Triton Kernel, and allowing backpropagation to directly reuse the generated row-to-slot mapping, thus eliminating duplicate indexes and preserving sparsity from data query all the way to gradient update. Its system implication is very clear: Sparse models do not have to continue to pay the cost for dense implementations. This idea is also expected to be migrated to other tasks that rely on large embedding tables or memory tables where each batch only accesses a small number of rows.

03 

GPUMode TriMul H100

When a GPU Kernel has been highly optimized, the remaining latency is often hidden in the most inconspicuous places. As one link continues to speed up, the bottleneck will also shift between Kernel fusion, tensor layout and video memory access. Therefore, to further improve performance, it is necessary not only to generate a better Kernel, but also to re-judge where the real bottleneck lies.

The GPUMode TriMul task targets the outgoing Triangle Multiplicative Update in AlphaFold-like protein structure models, requiring participants to implement a numerically correct version on NVIDIA H100 and minimize the geometric mean latency under seven fixed input shapes as much as possible. This task has attracted many top human engineers and automated systems to participate, including TTT-Discover, which was published by researchers from Stanford, NVIDIA, Astera Institute, UC San Diego and Together AI in January 2026.

On seven H100 workloads, our AI system reduced the geometric mean latency to 1,036.1 μs. In the paired evaluation on the same machine, this result is 2.71% lower than the strongest human-designed control scheme stashuk-olek (1,064.9 μs). Judging from the values reported publicly respectively, it is also lower than shiyegao CUDA (1,074 μs), Zeyu Shen Triton (1,140 μs) and TTT-Discover (1,161 μs), and is only about 0.59% different from the 1,030 μs reported separately by K-Search.

Figure 3: Geometric mean latency under seven GPUMode TriMul H100 input shapes. Our AI system reaches 1,036.1 μs, lower values indicate better performance. Note: Other public values come from respective reports or evaluation records, and are not paired-measured with our results on the same machine. Our 1,036.1 μs is measured locally using the official evaluator.

Existing high-performance TriMul implementations have widely adopted methods such as Kernel fusion, specialized matrix multiplication, specialization by input shape, and layout-aware data movement. Our AI system initially followed this path to continue strengthening the fusion of the upstream producer, but as register pressure increased, the returns began to diminish, and the performance was not stable across different input shapes. Therefore, the system shifted the search focus from "how to make upstream calculations faster" to "how its output is used downstream".

On several of these workloads, the AI system finally identified the boundary of BMM → LayerNorm as the main source of remaining latency. The new implementation reads BMM outputs along consecutive physical dimensions and completes the required transposition directly in registers before entering LayerNorm.

What is really noteworthy about this optimization is not just that the Kernel eventually becomes faster, but that the system actively changed the search assumption: when the optimization of the upstream calculation end is close to saturation, it can reposition the bottleneck along the data flow and adjust the subsequent goals accordingly. The fact that different input shapes eventually correspond to different solutions also reflects a basic law of GPU optimization: after one link is speeded up, the bottleneck may shift, and the next optimization must move accordingly. The value of this capability goes far beyond tuning a single fixed Kernel.

04 

MLS-Bench—Fused Causal Attention Kernel (H100)

Attention is one of the main calculation and video memory overheads in Transformer training and inference, and its execution efficiency will directly affect the speed and cost of the model. At the same time, today's fused-attention Kernel has undergone high-intensity optimization, and there is little obvious room left. To achieve further breakthroughs, in addition to improving scheduling and memory movement, it is also necessary to re-examine those assumptions in numerical algorithms that have long been accepted by default.

The MLS-Bench Fused Causal Attention task requires the AI system to use OpenAI Triton to implement the fused self-attention forward pass on NVIDIA H100, maximize throughput in three causal-attention configurations, and ensure that the maximum absolute error is less than 1e-2. This evaluation set includes results from cutting-edge models such as Claude Opus 4.6, GPT-5.4 and Gemini 3.1 Pro, thus providing strong baselines from Anthropic, OpenAI and Google DeepMind.

Under the three fused-attention configurations, our AI system reached 430.7, 468.4 and 451.7 TFLOP/s respectively, corresponding to head dimensions of 64, 128 and 256. All three results refreshed the previous best results, with increases of 27.18%, 15.63% and 17.11% respectively.

Figure 4: Throughput comparison under three MLS-Bench Fused Causal Attention configurations.

The core of the performance improvement comes from the simpl