Has the interpretability of neural networks been solved by AI? 90% of neurons have only one function

This is one of the most critical achievements in the AI field.

This Friday, a lengthy article by Nick Cammarata, the founder of OpenAI's Interpretability Team, sent shockwaves across the AI community. He stated that with modern agent tools, large models with trillions of parameters are now nearly interpretable. Our previous perception of neural networks as "black boxes" has proven to be a misunderstanding: most of the anticipated barriers do not exist, and interpretability will be solved, possibly very soon.

This could be the "bitter lesson" for interpretability: simple things, when scaled up to an absurdly large size, can work.

We can train powerful AI models, but we do not necessarily know why they generate certain responses. If we ask AI to help study the "neural circuits" of neural networks, can we crack this black box? The answer appears to be yes.

Although Nick's article is extremely long (it takes 65 minutes to read), some researchers have already published their interpretations of it. In short, neural networks are not tangled knots (the assumption of polysemantic neurons is wrong). You can directly study the neurons themselves, without needing to "untangle" them with a second model first. 90% of the neurons in Qwen-32B have only one single task.

This work may ultimately enable us to intentionally design large language models and shape their behavioral patterns.

Let's take a look at the main points of his argument:

Over the past few years, the field of AI interpretability has been built on a frustrating premise: neural networks are a mass of tightly knotted threads. When there are not enough neurons, the model will fold completely unrelated concepts into the same neuron, simply because these concepts do not interfere with each other. To interpret the model, you have to first train a second model (such as the Sparse Autoencoder, SAE) to untie the knots. This hypothesis, known as superposition, forms the foundation of the entire AI safety research line.

Now, the very person who helped lay this foundation has stepped forward to say: We were wrong. Neural networks are inherently clean.

"Cats and Cars" Might Never Have Been Two Separate Things in the First Place

Mechanistic interpretability attempts to trace neurons and computational circuits to explain how models arrive at their outputs. One of the difficulties is that a single neuron often responds to things that seem completely unrelated. A classic case comes from the vision model InceptionV1: the same neuron recognizes both cat faces and the front faces of cars.

The key interpretation in the past was "superposition": the model has limited representational space, so it compresses unrelated concepts that rarely appear simultaneously into the same group of neurons. Researchers therefore tried to use tools such as sparse autoencoders to decompose these representations back into intelligible features.

Nick now believes that this well-known case was misinterpreted.

After re-examining the images and connection relationships, he found that cat faces and car front faces share one common feature: a V shape. This neuron is also associated with V-shaped flames, fox faces and other similar patterns, while circles will push it to produce a negative response. In other words, "cat" and "car" are classifications defined by humans, but what the network identifies may be shape rules that cross these human-defined categories.

He thus reminds us: Neurons are not necessarily like nouns that are responsible for labeling objects; they may also act like verbs, responsible for distinguishing patterns and routing information to different downstream circuits.

The chaos we perceive is sometimes just the model using a classification method that we did not expect.

The Truth of Polysemantic Neurons: Forking Like Tree Branches

Other neurons do have multiple uses, but the author argues that this does not necessarily stem from the arbitrary mixing of unrelated concepts.

Nick proposes the concept of "recursive generalization": a neuron starts from one pattern, gradually grows related branches, and these branches continue to grow further. A neuron that recognizes the syllable "iy" can expand to the word "diy", and then further expand to the full phrase "do it yourself" that does not contain the syllable "iy" at all. A geology-related neuron will branch out to the Cesium geographic JavaScript library. This is not hiding information in high-dimensional space, but growing in semantically adjacent territories. He emphasizes that superposition is the most radical and hardest-to-falsify explanation, which should be the last hypothesis to consider, and he himself was a firm believer of it for many years.

If you only look at the responses after training is completed, these functions may seem loosely connected; but if you trace along the training process, you can see how they are connected to each other.

Tools can also generate misunderstandings. The team found that a unit labeled by an automatic interpretation tool as a "polysemantic comma neuron" is actually related to the state of Colorado: it responds to information such as local area codes and local slang, and enhancing its activation will make the model more inclined to plan hiking trips. The comma is only the position where the model summarizes the preceding context. Staring at the characters that trigger the response will make you miss the actual meaning.

In 2023, OpenAI's automated interpretability project only found about 1.7% interpretable neurons even under the most lenient standards. Now, in the preliminary review of Qwen3-32B, the author estimates that about 90% of the neurons have one main task on their positive and negative poles respectively, about 7% have two related tasks, and about 2% to 4% are not yet understood. This figure depends on the activation threshold and how we define "one task".

The conclusions are similar for both dense models and MoE models. The author got the same results on Kimi K3: they produced the first known attribution map for a trillion-parameter model on Kimi K3. In the scenario of "finding a lost wallet", the concepts of wallet, cash and documents are connected to lost and found services and honesty, and the final output is "return the wallet". When re-examining the GPT-2 XL model from years ago with new tools, about 70% of the neurons are also roughly responsible for "one single thing".

Let AI Scientists Study the Inside of AI

Another main line of this work is that agents have changed the scale of research.

AI can check neuron samples, propose explanations, trace upstream and downstream connections, and then design experiments to find counterexamples. Researchers also let agents inspect hundreds of thousands of attribution maps, and the author himself inspected thousands of them.

How can we prevent AI from making up a plausible-sounding story?

One test is as follows: the first agent studies a neuron and writes down an explanation; the second agent only reads the explanation, generates a new sample that is expected to strongly activate the neuron, and then submits it to the network for measurement. The author reports that the activation intensity of some samples exceeds all examples they found in the 100-billion-token corpus. The team also generated counterexamples that "look similar but should not actually trigger the neuron", and checked whether the explanation can predict the model's behavior through interventions such as removing neurons and enhancing activation.

In this way, explanations become hypotheses that can be falsified by experiments.

Cammarata also put forward a counterintuitive insight: Understanding more parts of the network will sometimes make the remaining parts easier to understand. Just like a crossword puzzle, the more adjacent answers you have, the more constraints there are on what can be filled in a blank space. Neurons in larger models may also have more fine-grained division of labor, making them easier to identify.

This reminds him of the field's own "bitter lesson": simple observation, hypothesis and verification, when carried out on a sufficiently large scale, may be more effective than constantly designing elaborate disassembly methods.

This is still not a declaration that "interpretability has been solved". There is still a lot of work to do on attention mechanisms, full model auditing, and how to translate this understanding into reliable safety improvements. But it provides a direction worthy of serious verification: if there is already traceable organization inside neural networks, future breakthroughs may come from a laboratory operated by AI scientists, which continuously observes models, raises questions, and verifies answers one by one.

References:

https://already-clean.goodfire.com/

https://news.ycombinator.com/item?id=50105647

This article is from the WeChat Official Account Ji Qi Xin Zhi (ID: almosthuman2014), written by a contributor focusing on large models, and republished with authorization from 36Kr.