Is this study comparable to the seminal foundational paper of Transformer? This research has sparked heated debates in the global AI community.

This article has completely overturned the cognition in the field of artificial intelligence, triggering widespread speculation, and Sam Altman has also publicly expressed his appreciation.

Researcher Adam Majmudar wrote on X: This is the most important achievement of this year, and its importance is comparable to "Attention Is All You Need". It has taken us from the stage of "whether the alignment problem can be solved" to the stage of "we may be able to solve the alignment problem next year".

Some people also said: "My understanding of neural networks seems to have been pushed forward by five years."

Cammarata, the author of the article, said that he published it at a time when people were looking for hope.

What researchers and policymakers in the field of AI safety are most worried about is that powerful models may bring huge risks. At present, they are in urgent need of a "lifeline" to support themselves. This is one of the reasons why the article has caused such a huge response.

The judgment put forward in the article is quite bold: Large neural networks may be much easier to understand than people previously thought, and many internal structures that are regarded as difficult to disassemble may have always followed some recognizable rules.

However, Jacob Hilton from Alignment Research Center, a non-profit research laboratory in Berkeley, does not think Cammarata's blog is as important as everyone says.

Hilton pointed out in his tweet on X that researchers understand less than 0.01% of the inner workings of top LLMs.

"I have always appreciated Cammarata's optimism, but unfortunately, it presents the public with an unrealistic picture of the current state of the field."

Original blog link👉: https://already-clean.goodfire.com/

The Black Box, Maybe We Have Misread It

Almost all large models use neural networks to some extent.

A neural network is a machine learning model inspired by the working principle of the human brain, composed of many small units called neurons that are connected to each other in the form of layers.

Cammarata has long been engaged in research on the interpretability of neural networks, and participated in the establishment of a far-reaching theoretical framework in his early years.

In early visual models, the same neuron responds to two types of images: cat faces and the front of cars. What exactly is it recognizing? A cat? A car? Or are two completely unrelated concepts mixed together?

Cammarata also believed back then that there were many such entangled concepts inside the model.

To this end, researchers have developed the "superposition" hypothesis. The explanation is as follows: there are too many concepts that the model needs to express, and the number of available neurons is limited, so the network has to squeeze different concepts into the same representation space. In this way, one neuron carries several seemingly completely irrelevant meanings.

In order to untangle these intertwined concepts, researchers have specially developed tools such as sparse autoencoders, trying to decompose neural activities into clearer units.

The superposition of the almost plane wave (diagonal) coming from the distance and the wave generated by the duck's wake only approximately satisfies the linear relationship in water

Cammarata now thinks that he might have overthought it back then.

That neuron may be recognizing a V-shaped structure. Behind the outline of a cat face, the sharp angle formed by a car coming head-on, the tongue shape of a flame and the bifurcation of water flow lies the same visual rule. Objects that seem completely unrelated actually share the same geometric feature.

Another case that was once regarded as evidence of "superposition" may also have a simpler explanation.

Researchers once found that the features for recognizing cars seemed to be hidden in a group of neurons related to dogs. After re-examination, Cammarata gave another explanation: Dogs and cars may share a rule in shooting angles — they often face the camera in similar postures in pictures.

If all these classic cases have other explanations, are the insides of neural networks really that hard to understand?

Cammarata reported that after improving the architecture and analysis methods, even without increasing the model scale or training data, about 99% to 100% of the neurons in the last layer of the early visual model InceptionV1 can get a relatively clear explanation.

He also studied models spanning about 300 times the scale, and believes that these models retain a considerable degree of readability at different scales.

Cammarata uses "recursive generalization" to describe another pattern he observed.

A neuron may start from a narrow pattern during the learning process, and gradually extend related branches from it, just like a tree branch growing side branches.

Taking the language model as an example, a neuron that initially recognizes the letter combination "iy" may later respond to "diy" and the semantically related "do it yourself". If you only look at the trigger list after the model is trained, these responses seem to be far apart; if you track how they are formed during the learning process, you may find a continuous associated path.

In Cammarata's view, many complex phenomena come from the continuous expansion, combination and differentiation of ordinary patterns, rather than concepts being randomly squeezed into the same neuron.

This idea has been preliminarily verified. In Qwen3-32B, about 90% of the positive and negative response directions of neurons each have a main function, about 7% involve two related functions, and the unexplained part is about 2% to 4%.

The More You Understand, the Safer It Is

If the information flow inside the model can be reliably tracked, safety research will have the opportunity to go deeper from observing results to analyzing mechanisms.

Researchers can try to identify specific neurons, computational circuits and information transmission paths in the model, and then through intervention experiments, observe whether changes in a certain internal mechanism will change the behavior of the model.

If a group of neurons is involved in a certain reasoning, and a computational circuit is related to a specific judgment, then researchers will have the opportunity to conduct more targeted tests on them.

Compared with judging whether a model is safe only based on the final answer, this method attempts to advance safety assessment to the internal process of behavior generation.

This is exactly why Cammarata's research is so remarkable.

If the internal structure of neural networks is indeed more organized than people previously thought, the cost of understanding it may drop. Researchers can directly study the mechanisms that the model has formed, track how these mechanisms cooperate with each other, and gradually build a more complete internal map.

However, the internal mechanism of the model is not equivalent to the psychological concepts we are used to using. A neuron related to deception is not necessarily an independently existing "deceptive intention" inside the model; a feature that can predict refusal to answer is not necessarily sufficient to show that the model understands the reason for the refusal.

Researchers need to verify explanations through multiple evidences, not only observing neural activities, but also conducting interventions, counterfactual tests and cross-situation re-verification. Otherwise, the so-called understanding may still just be labeling a certain statistical association with a plausible-sounding tag.

In safety work, model evaluation, red team testing, access permission control, runtime monitoring, and deployment restrictions on high-risk capabilities all have their own irreplaceable roles. There are always boundaries to researchers' understanding of the model, and real-world risks often come from the superposition of multiple links.

The Endgame of AI Is Also an Arrangement of Power

When it comes to the long-term safety of AI, the three possible outcomes recently proposed by Sam Altman have led the issue to a broader institutional level.

One is to establish an international organization similar to the International Atomic Energy Agency, responsible for formulating and enforcing safety standards; the other is to integrate all artificial intelligence laboratories into the same institution to jointly develop new technologies and ensure the safe application of technologies; the third is that a certain AI developer accumulates sufficient economic strength to cut off the computing resources of other developers, making potential dangerous developers unable to continue advancing their projects.

The development of nuclear technology has long shown that the risks of certain technologies may exceed the control scope of a single country, and it is necessary to establish mutually recognized standards, verification mechanisms and international cooperation channels. The experience of the International Atomic Energy Agency provides a reference for this.

However, there are obvious differences between artificial intelligence and nuclear technology. Models can be copied through software and deployed remotely, and the speed of technological iteration is much faster than many traditional industrial systems.

Even if an international institution is established, how to determine which models need to undergo strict review, how to verify whether developers comply with the standards, and how to handle technological competition between different countries are still intractable problems.

If all AI laboratories belong to the same institution, researchers do not have to speed up the development progress to compete for market share, and all teams can also share safety research results to avoid repeated mistakes. Centralized resources can also make high-risk experiments subject to unified review, reducing the space for a team to bypass safety procedures in order to get ahead of others.

But who will manage this institution, and who will supervise the managers?

When all key technologies, computing resources and research capabilities are concentrated in the same organization, safety standards may indeed be easier to unify, but power may also become difficult to check and balance as a result.

Once a systematic error occurs in the judgment within the institution, the entire technical system may be affected. If the management has a deviation in the judgment of certain risks, the error correction effect that other independent laboratories could have played will also be weakened.

The third scenario does not require all developers to accept the same management system, nor does it need to wait for an international institution with broad authorization to reach a consensus. As long as a certain developer has sufficient capital, computing power and infrastructure, it may limit the ability of other developers to continue training or deploying models by controlling GPUs, data centers and related supply chains.

Altman said this is the last situation he wants to see.

Back to Cammarata's research, advances in interpretability may provide a new technical foundation for these governance issues. If researchers can more reliably identify the key mechanisms inside the model, they will have the opportunity to improve safety assessment, audit high-risk capabilities, and verify whether certain intervention measures are truly effective.

As technical capabilities continue to grow, what matters will not only be whether we can understand more and more complex models, but also whether society can establish a sufficiently reliable mechanism to ensure that those who understand the technology have the ability to supervise it, and those who master the technology are always constrained.

This article is from the WeChat official account "APPS