Google's latest Science paper has subverted humanity's imagination of ASI.

A Science paper published last week is shaking people's most deeply ingrained imaginings about the future of artificial intelligence.

For decades, the story of the "technological singularity" has almost become a foregone conclusion: One day, an all - powerful artificial superintelligence (ASI) will emerge out of nowhere, leaving humanity behind.

This image has shaped countless science - fiction novels and vaguely underpins the underlying logic of today's AI safety discussions.

Researchers from the University of Chicago, the University of California, San Diego, and Google believe that this story is fundamentally wrong.

James Evans, Benjamin Bratton, and Google researcher Blaise Agüera Y Arcas published a paper titled "Agentic AI and the next intelligence explosion" in the journal Science. They put forward a completely different judgment: The real intelligence explosion is already happening, but its form is completely different from what people expected - it is diverse, social, and deeply intertwined with humanity.

There are "people" arguing inside the model

The starting point of this paper is a discovery about the internal mechanism of reasoning models, which is quite surprising to the AI community.

In the past year, the emergence of reasoning models such as DeepSeek - R1 and QwQ - 32B has attracted wide attention.

Their performance in tasks such as mathematics, code, and complex logic significantly exceeds that of regular instruction - tuned models of the same scale.

The usual explanation is that these models "think longer" - through reinforcement learning, they generate longer thought chains before answering and spend more test - time compute.

However, researchers from Google, the University of Chicago, and the Santa Fe Institute found that the improvement in reasoning ability does not come from a simple extension of computational volume, but from an implicit simulation of multi - agent interaction - which they call the "society of thought".

Different cognitive perspectives with distinct personality traits and domain expertise will emerge inside the model, and these perspectives engage in debate, questioning, and reconciliation.

The research team analyzed the model outputs of more than 8000 reasoning questions and found that in the most complex tasks, such as graduate - level scientific reasoning (GPQA) and high - difficulty math problems, the "dialogue features" of DeepSeek - R1 are particularly obvious; while in relatively simple procedural tasks such as Boolean expressions, these features almost disappear.

They even conducted a more direct verification: In the DeepSeek - R1 - Llama - 8B model, the researchers found an internal feature related to "surprise, insight, or response". After artificially increasing the activation intensity of this feature, the accuracy of the model in mathematical calculation tasks jumped from 27.1% to 54.8%.

A case of a chemistry problem describes this phenomenon quite specifically: Facing a complex Diels - Alder synthesis reaction, DeepSeek - R1 suddenly wrote during the reasoning process: "No, this is cyclohexadiene, not benzene" - the model corrected the error through self - negation.

DeepSeek - V3, on the other hand, followed a single - line narrative straight to the end and gave the wrong answer.

More notably, these models were never trained to generate a "society of thought".

When reinforcement learning only uses reasoning accuracy as a reward signal, the model spontaneously increases dialogical and multi - perspective behaviors.

The optimization pressure has found its way to social reasoning on its own.

Intelligence has never been a solo act

In their Science paper, Evans et al. placed this discovery in a broader historical framework: Every "intelligence explosion" is essentially a leap in the way of social organization.

The intellectual level of primates increases with the expansion of group size, rather than with the increasing difficulty of their habitats.

Human language has created what Michael Tomasello calls the "cultural ratchet" - knowledge accumulates across generations without the need for each individual to rebuild it from scratch.

Writing, laws, and bureaucratic systems have externalized social wisdom into institutions and infrastructure.

The paper gives an interesting example: A Sumerian scribe was responsible for running a grain accounting system, but he didn't understand the macro - economic function of this system at all - yet the intelligence of the entire system far exceeded his individual intelligence.

Large language models continue this line: They are trained on the entire output of human social cognition and are the computational activation form of the cultural ratchet. Each parameter is a compressed precipitation of countless exchanges and expressions.

This perspective directly challenges the singularity narrative of the "lonely super - brain".

Bratton has been continuously exploring similar issues in the research of his think - tank Antikythera. He once described such a scenario in a speech:

If there are 8 billion human agents and 80 billion or even more non - human agents in the future, and the ratio between them may be 1 to 10, 1 to 100, or even higher, then the question of "what constitutes society" will return to first principles.

The "Centaur" era has begun

The paper refers to the current form of human - machine collaboration as the "centaur configuration" - a hybrid actor of humans and AI agents, neither purely human nor purely machine.

This configuration will become extremely diverse: One person can command multiple AI agents; one AI can serve multiple people; many humans and many AIs can cooperate with each other in dynamic organization.

Agents can self - replicate and branch. An agent facing a complex problem can generate copies, assign subtasks, and then merge the results - this is a recursive collective deliberation that unfolds at each level of complexity and converges when the problem is solved.

What does this mean for the expansion path of AI?

The paper's judgment is that what matters is not just the computing power scale of a single agent, but whether the system can operate in the scale and context of the real society.

Therefore, "building agent institutions" is as important as "building agents themselves".

In terms of alignment methods, the paper criticizes the current mainstream solutions.

Reinforcement learning from human feedback (RLHF) is essentially a "parent - child error - correction model", which is effective in a binary relationship but difficult to scale when facing billions of agents.

They advocate an "institutional alignment" path: Just as human society operates based on long - lasting institutional templates such as courts, markets, and bureaucratic systems, rather than relying on the personal virtues of each individual, a scalable AI ecosystem also needs its digital equivalent. The identity of the agent is secondary; the key is whether it can fulfill a certain role protocol, just as the existence of positions such as "judge", "lawyer", and "jury" is independent of the specific people sitting in those positions.

Who will audit the auditors?

At the governance level, the paper touches on the most difficult problem.

When AI systems are deployed in high - risk decision - making such as recruitment, sentencing, and welfare distribution, the question of "who will audit the auditors" cannot be avoided.

The paper proposes a concept of a "constitutional structure": The government needs to deploy AI systems with clear value orientations - transparency, fairness, and due process - specifically to balance the AI deployed by the private sector and other government departments, and vice versa.

For example, the AI of the labor department audits whether the corporate recruitment algorithm has a differential impact, and the AI of the judicial department evaluates whether the risk assessment of the administrative department's AI meets constitutional standards.

The paper uses a detail to illustrate another possible future: Another option is to hire business school graduates with Excel spreadsheets, like the U.S. Securities and Exchange Commission, to counter the high - dimensional collusion of AI - enhanced high - frequency trading platforms.

The traders of the Federal Reserve are already facing a whole set of automated cognitive systems - this is already a reality.

The core concern of this paper is precisely to avoid misplacing attention.

Fearing the arrival of an all - powerful single AI may lead policy astray - to guard against a technological form that may never appear.

What really needs to be designed are the norms, coordination mechanisms, and institutional frameworks of the hybrid human - machine social system.

To conclude with the exact words of the paper, it is concise and powerful: The question of the intelligence explosion has never been whether it will come, but whether we can build the social infrastructure to match it.

Reference:

https://www.science.org/doi/10.1126/science.aeg1895 

This article is from the WeChat official account "New Intelligence Yuan". The author is New Intelligence Yuan. It is published by 36Kr with permission.