OpenAI unveils the self-evolution secrets of GPT-5.6, and Weng Li is reportedly returning.

According to reports from Zhidx on July 30, several major announcements from OpenAI were released in the early hours of today:

1. OpenAI unveiled new progress that GPT-5.6 has started optimizing "itself": it uses GPT-5.6 Sol to optimize GPU kernels in production environments, cutting the deployment cost of inference services by 20% sharply; it also optimizes speculative decoding technology to boost token generation efficiency by over 15%.

2. In the research of AI self-evolution, OpenAI may recruit a top talent back: Lilian Weng, co-founder of Thinking Machines Lab, the recently departed US AI unicorn, was exposed to rejoin OpenAI to assist its AI self-evolution research.

3. New updates were revealed for the much-followed OpenAI hardware R&D: OpenAI President Greg Brockman teased that OpenAI is developing AI hardware, "users may see it very soon". When talking about the device form, he believes that "in the vast majority of cases", people will choose to have conversations with computers.

4. OpenAI's product scale has hit a new record: its user base has exceeded 1 billion for the first time, serving 2 million enterprises.

With full breakthroughs in model performance, core talents and user scale, OpenAI released a number of details in the early hours of today, revealing the technical secrets of applying its strongest GPT-5.6 series models to self-optimization.

01. GPT‑5.6 Sol in action, overall token generation efficiency boosted by over 15%

Nowadays, the growth rate of model service demand has surpassed the expansion speed of computing power, so efficiency has become the core of all system design. OpenAI's primary goal is to provide more services under the same hardware conditions, while maintaining the intelligence, response speed, availability and reliability that users expect.

This requires optimization of the entire system, which is mainly reflected in traffic routing (target nodes for request distribution), task scheduling (execution timing of requests), GPU kernel programs (underlying computing programs running on graphics cards), cache mechanisms (storing and reusing calculated results) and model code implementation (arrangement logic of GPU instructions).

GPT-5.6 full-stack layered efficiency optimization solution

Earlier this month, the OpenAI GPT-5.6 series of models was officially released. The flagship version GPT‑5.6 Sol outperforms Claude Fable 5 in the programming agent evaluation list of Artificial Analysis, and its invocation cost is about 2/3 of the latter; Terra, a balanced model suitable for daily work, matches GPT‑5.5 in various intelligence benchmark tests, with the cost only 1/2 of the former; Luna, which is fast and affordable, is the model with the fastest response speed and the highest cost performance in the GPT-5.6 series, with the pricing reduced by 80% compared with Sol.

To achieve the above effects, OpenAI's design idea is to improve two key parts of the technology stack:

The first is the inference service system, which increases the effective output under the premise of unchanged hardware configuration by optimizing processes such as load balancing, speculative decoding, cache mechanism, and kernel operator optimization;

The second is the agent scheduling base, which targets and optimizes the problems of context expansion, tool invocation logic and redundant repeated calculation.

GPT‑5.6 Sol has relied on its autonomous operation capability to contribute to the above achievements.

The first example is load balancing. At the global level, it completes request routing and distribution according to factors such as user geographic location, remaining computing power capacity, and chip type; inside a single cluster, it distributes computing tasks to each model instance according to load pressure, context length, cache availability status and other request features; inside each model instance, computing tasks need to be efficiently split and allocated to acceleration chips, model sub-networks and each computing core.

In this process, GPT‑5.6 Sol running on Codex can analyze the real online business traffic, locate the previously ignored computing power imbalance problem, test new routing strategies, and then iterate the scheduling rules.

GPT‑5.6 Sol is also used to optimize the forward propagation calculation of the model, that is, to convert the input data into the predicted value of the next Token. Through Codex, GPT-5.6 Sol autonomously rewrote and optimized OpenAI's core code, which is responsible for executing the mathematical operations that make up the model.

This is because researchers have trained GPT-5.6 to write and optimize core code in two open-source GPU programming languages, Triton and Gluon.

The above optimization finally reduced the overall end-to-end inference service cost by 20%.

The second method is speculative decoding. This mechanism deploys a smaller prediction model to work in coordination with the main model: the draft model pre-generates a string of Tokens first, and the main model verifies them in parallel. If the pre-generated content passes the verification, the system only needs to perform one forward calculation of the main model to output multiple Tokens at one time, so as to reduce the serial computing overhead.

GPT‑5.6 Sol completed iteration for the supporting prediction model, and carried out hundreds of groups of architecture comparison experiments around model scale, network structure, functional modules and other aspects. It can also autonomously start and take charge of the whole process training task of the prediction model, and automatically intervene to handle abnormal problems such as hardware failure and training oscillation. This makes the overall token generation efficiency increase by more than 15%.

When processing uncached input Tokens, the model will build a KV cache through a calculation process with high computing power consumption, and then repeatedly read and continuously expand the cache content in the stage of generating output content.

Optimal online service configuration schemes such as batch processing, shard deployment, and KV cache management will be affected by the prompt length of actual services, output text length, batch processing scale, cache hit rate, query request attributes and other factors.

However, because the optional parameter space of such configurations in the past was huge, systematic tuning could not be carried out, and researchers could only complete rough configuration relying on general empirical rules.

With the help of GPT‑5.6 Sol, researchers can analyze the real online business load, generate and compare various alternative configuration schemes, and complete fine-grained hyperparameter optimization for inference engines and model parameters for different usage scenarios.

At the same time, inference optimization is also a closed-loop feedback system that can operate continuously: collect real online operation data, locate the most prominent performance shortcomings of the system, implement corresponding optimization schemes, and finally verify that the changes can improve the performance of the whole system.

The combination of GPT‑5.6 Sol and Codex can optimize all the above environments.

02. Simplify repetitive workflows, three methods to optimize the agent scheduling base

ChatGPT Work and Codex can complete different complex tasks by relying on continuous model invocation and tool invocation links.

In a single interaction process, from the user issuing instructions to returning the final result, Codex often needs to perform a series of operations such as viewing source code, retrieving deployment records, consulting fault reports, editing files, and running tests. Each of the above operations requires a model request.

Context assembly, data transmission, inference calculation, tool invocation and process startup will all generate time consumption and computing power consumption. If a task requires 30 model invocations, and each invocation has an extra second of delay, the cumulative loss will be even more.

A user operation cycle can include multiple iteration processes of models and tools

Researchers believe that to achieve a substantial improvement in overall performance, it is not enough to simply speed up the model itself, but also to reduce various repetitive calculations and redundant operations across the whole system.

Their idea is to design the agent scheduling base, including three methods of managing context expansion, optimizing tool loading mechanism, and reusing existing calculation results, so as to reduce redundant information and make it easier and more accurate to reuse results.

1. Avoid unnecessary redundant information

Agents can call more tools and obtain access rights to dialogue history, etc. The context window is prone to redundant expansion, which will increase invocation costs, interfere with model judgment, and even trigger a large number of invalid inference operations.

Based on this, the agent scheduling framework reduces overhead by relying on the lazy loading retrieval mechanism: various integrated components, custom MCP tools, functional modules and plug-ins will be loaded into the context only when they are actually called.

At the same time, this framework can prevent a single tool or MCP integration program from occupying the context space without restriction. The system limits the output content returned by tools to within 10,000 Tokens by default, and the model can apply to adjust this upper limit according to business needs.

2. Keep the exact prefix for prompt caching

As mentioned earlier, in the agent task loop of a single interaction cycle, the same instructions, dialogue history, tool definitions and past operation results will be repeatedly pushed to the GPU for calculation for many times, and the prompt caching mechanism can reuse the operation results corresponding to the pre-calculated prompt prefix.

To stably retain this prefix cache content, the agent scheduling framework sets all historical data visible to the model to the append-only write mode: new messages, tool return data, and environment change information will only be appended to the end of the context, and will not insert or modify the previous context content.

The framework pushes various tool information to the model in a fixed and orderly format. For example, runtime configuration parameters such as permission approval rules also take effect dynamically during the task execution phase, instead of being solidified and written into the tool definition file.

03. GPT‑5.6 Sol can't solve the 2D puzzle, it's because of API configuration

Previously, GPT‑5.6 Sol has solved long-standing open mathematical problems such as the circle covering conjecture, but in the 2D puzzle evaluation benchmark ARC-AGI-3, the accuracy of GPT‑5.6 Sol is only 7.8%, and the previous generation GPT‑5.5 model can hardly complete this puzzle task, with a score as low as 0.4%.

ARC-AGI-3 is equipped with a set of extremely simple general-purpose agent scheduling framework, without any tools and exclusive enhancement functions mounted. Its design idea is to adopt a lightweight scheduling framework to more intuitively expose the capability shortcomings of the model itself, and also make the horizontal comparison between different models more objective.

In the ARC-AGI-3 case, when researchers enabled two API configurations that are used by default inside ChatGPT and Codex, namely inference memory retention and context compression, the model's evaluation score on the public test set increased to 3 times the original level, and the output token consumption was directly reduced to 1/6 of the original.

After using the official framework, GPT-5.6 Sol's score on the ARC-AGI-3 public data set reached 13.3%. If the original inference mechanism and compression technology are retained, its score can be increased to 38.3%

OpenAI research found that this phenomenon is not caused by the defect of the model itself, but caused by various parameter configurations of the agent scheduling framework.

First of all, after each step of puzzle solving operation performed by the model, all private inference thinking processes will be cleared directly, which leads GPT‑5.6 Sol to analyze the puzzle solving rules from scratch every time, and cannot retain the deduction ideas of the previous round.

Although the model can view historical operation records and short attached notes, it cannot access the planning schemes, rule cognition and complete thinking links behind these operations.

The second problem is that this scheduling framework adopts a rolling truncation window mechanism. As the interaction history continues to accumulate, early operation records will be automatically excluded and no longer visible to the model. As a result, GPT‑5.6 Sol not only cannot retain past inference ideas, but even the historical operation records it has performed will be gradually lost.

When OpenAI optimized to make agents remember what they had done, they achieved their best performance.

Before outputting answers or initiating tool calls, its model has a training paradigm that thinks relying on private inference messages. Such private thinking content will be persistently stored in the dialogue history. When the length of the dialogue context exceeds the limit, the system will perform summary compression on the existing history, and then continue the subsequent interaction process.

Inference operation mechanism of GPT-5.6 Sol for processing long tasks

OpenAI reconstructed the scheduling framework corresponding to ARC-AGI-3 based on its own Responses API, which can easily manage the context link. GPT‑5.6 only needs to pass in the previous round of response ID, so that the whole inference process can be completely retained between multiple rounds of tool calls and multiple rounds of interactions.

After enabling the persistent storage function of inference content, they observed two changes: the deduction time of GPT‑5.6 Sol before each step of operation decreased significantly, and the model does not need to analyze the puzzle solving rules and board information from scratch in each round of interaction; when the model can access past thinking records, its ability to continuously learn problem-solving rules and output coherent and feasible strategies along with the task process will be improved.

The second optimization scheme is to replace the original rolling truncation mechanism with the context compression configuration that comes with the Responses API.

The original ARC-AGI-3 scheduling framework relies on the rolling truncation mechanism to handle the upper limit of context length. Once the total number of characters in the dialogue context exceeds 175,000, the system will directly discard the earliest batch of message content.

After researchers enabled the context compression function in the ARC-AGI-3 test environment, GPT‑5.6 Sol can retain the learned experience of each puzzle in a longer puzzle solving process, and finally get a higher evaluation score with less output token overhead.

The animation below demonstrates the dynamic change process of the 175,000-character context window of GPT‑5.6 Sol when solving a series