Just now, OpenAI's new Transformer has gone viral, with the Astra architecture unveiled to the public for the first time.

Hey everyone, have you heard the news?

OpenAI's next-generation Astra adopts a brand-new inference method called "recurrent depth".

The number of thinking tokens is drastically reduced, while the performance skyrockets to an extremely high level!

No wonder the Astra in gray test is so powerful. It turns out that OpenAI has a real ace up its sleeve in technology.

But the tradeoff is that Astra's thinking process is completely "invisible". Even OpenAI itself can barely figure out what's going on inside the AI's brain, making monitoring far more difficult than ever.

As soon as the news broke out, the entire internet was thrown into panic.

Jakub Pachocki, Chief Scientist of OpenAI, responded immediately —

For the most cutting-edge internal models (including Astra), the depth of their computation graph is at most twice that of GPT-4, which is not as mysterious as the outside world claims.

Almost at the same time, a top influencer broke the news that the name gpt-6-astra had appeared on the OpenAI API.

Judging from the current situation, GPT-6 is expected to be launched tomorrow.

What on earth is the origin of the new architecture behind it?

"Looped Transformer" is taking the internet by storm

"Recurrent depth" has another name, which is "Looped Transformer".

In fact, this concept was not first proposed by OpenAI.

Prior to that, the entire industry had carried out a lot of research on the Looped Transformer.

To understand how Astra has changed, we first need to look at how traditional large models work.

The traditional Transformer is like a tall building that you can only walk through once: data climbs from the first layer all the way to the last layer, completes a fixed number of operations, and outputs the next token.

Want it to reason more deeply? Then you can only build the building higher and stack more parameters. The tradeoff is extremely high computing power and memory overhead.

The Looped Transformer adopts an extremely straightforward idea: make it loop.

Information no longer passes through the network only once in one direction, but the same set of Transformer layers is reused repeatedly in loops.

The hidden states calculated in the first round are directly sent back to the starting point for further processing, going through several rounds of loops before finally outputting the result.

It is like a spiral staircase — there are only a few steps, but as long as you go around a few more times, you can still reach the top floor.

This path has long been proven feasible in the academic world.

In a paper published by Google in 2025 titled Reasoning with Latent Thoughts, a very intuitive result is presented —

A Transformer with k layers that loops L times can perform close to a regular model with k×L layers on many reasoning tasks.

The number of parameters does not increase synchronously, but the effective computation depth is extended.

In February this year, a paper on "latent reasoning" also perfectly demonstrated the power of this architecture.

Researchers used a small model with 3.5 billion parameters, and the core block in the middle ran in repeated loops. When the number of loops was large enough, this small 3.5B model unexpectedly achieved performance equivalent to that of a large 50-billion-parameter model.

This is equivalent to turning on an overclocking switch for "test-time computing power".

Without adding parameters or occupying extra memory, its capabilities can leap forward by simply running more loops on the original weights (trading time for space).

For OpenAI, which was just overtaken by Anthropic in revenue and is in urgent need of a disruptive leap, the temptation of this technology is simply impossible to resist.

How thinking becomes "invisible"

In the previously viral article "AI 2027", this direction was once called "neural language loop".

Astra still thinks through problems using language, but much of the additional thinking between those words may now be silent.

Astra's evolution directly breaks the prediction of AI 2027

The danger of "recurrent depth" lies in the fact that it pushes thinking into a blind spot.

Prior to this, the industry had to accept a counterintuitive reality: for current reasoning models, CoT is the working memory that the model actually uses to draft its thoughts.

In the past, no matter how deeply the model thought, it had to squeeze out through the narrow funnel of "outputting the next token".

It is this squeezing process that turns abstract calculations into words that humans can understand. This is also the only basis for people to "see through" what is on the AI's mind in the past two years.

But on the other hand, "recurrent depth" widens the internal computation pipeline.

For a model with a fixed number of layers, the serial reasoning it can perform before outputting each token is limited.

Now that the model has gone through r loops in the middle block, r times more serial computations are embedded behind each token. The long chain of reasoning that originally had to be squeezed into literal text can now be completed silently in the vector space, without outputting a single word along the way.

The saved tokens are exactly the ones that would have been used for those unwritten thoughts. Worse still, what it "thinks" in the latent space may not be human language at all.

As Meta pointed out earlier in its research on "Chain of Thought with Continuous Latent Tokens (Coconut)":

Models reasoning with numbers and vectors may be more accurate and more efficient than being forced to use human language.

The logic behind this is very simple: the way large models understand concepts is inherently mathematical vectors, and forcing them to translate every step into human language will only lead to information loss.

Comparison between Coconut and CoT

The power of this technology is that it enables a small-sized model to deliver the performance of an extra-large model, with drastically reduced costs and saved memory and bandwidth.

However, while it reduces the computing power cost, it also completely blocks humans from seeing the internal process.

Astra's real ace is the new Transformer

The Looped Transformer architecture is extremely attractive for mathematics and programming scenarios.

Many real-world tasks inherently rely on "multiple iterations": first propose a solution, then check for errors, update the intermediate state, and continue solving.

The Looped Transformer is naturally suited to executing this kind of algorithm.

It also brings a brand-new "scaling at inference time" approach to the model.

In the past, letting the model think for longer usually meant making it generate more chain-of-thought tokens.

Now, part of the computation can be completed before the token is output. The model may only pause for a moment on the surface, but its internal state has already been iterated many times.

The powerful performance of Astra is obvious to all. Although it has not been officially released yet, several demos have verified its dominant position in the front-end field.

Moreover, in terms of computer usage, Sam Altman also admitted that Astra's performance is directly comparable to human level.

Based on this architectural innovation, Astra has also enabled "persistent agents" that can run continuously for several weeks without being stopped.

Now the whole internet is holding its breath and waiting, as Astra/GPT-6 is finally coming.

Many OpenAI researchers, including the "Cyber Father", are also frantically hinting at the powerful Astra.

Tonight, we will just wait for the release of GPT-6.

References:

https://www.theinformation.com/articles/secret-technique-behind-openais-astra-model-sparks-security-concerns?rc=epv9gi

This article is from the WeChat Official Account "AI Era" (ID: AI_era), written by Taozi and Moxi, and authorized for release by 36Kr.