AI Interpretability and Alignment: J-Space, Chain of Thought, AI Persona, Hallucination, and the Golden Gate Bridge

Today's AI remains a black box. We know how to train models, how to make them increasingly intelligent, and what answers they finally output. But we still know very little about exactly what occurs between the input and the output.

Trying to pry open this black box and understand what exactly happens inside the model is the problem that AI interpretability research aims to solve. Over the past few years, this field has indeed seen a great number of exciting breakthroughs.

Research teams at Anthropic have started tracking the information flow inside the Claude model, to learn what kind of internal calculations the model goes through before answering a question. Other studies have found that there seem to be representations inside the model related to specific concepts, behaviors and even "personalities". Transluce has discovered that models will change their behavior based on who they believe they are talking to. The model may perform completely differently when facing ordinary users versus AI security researchers.

Therefore, we want to figure out: How well do humans actually understand existing AI models? Why do models answer questions correctly? Why do they produce hallucinations? Will they make different judgments when facing different users? When models begin to show increasingly distinct and varied personalities, what exactly are these so-called "personalities"?

With these questions in mind, we discussed large model interpretability research with Aryaman Arora, a PhD student in Computer Science at Stanford University who is currently advised by Chris Potts and Dan Jurafsky. In this conversation, we talked about the unspoken "thinking" inside models, the differences between chain of thought and real reasoning, the two main lines of interpretability research, hallucinations, user modeling, model personalities, and the potential value of this technology in industry, regulation and scientific discovery. Below is our conversation with Aryaman.

01

AI Interpretability?

Yiwen: Please briefly introduce yourself to the audience first.

Aryaman: My name is Aryaman, and I am currently a PhD student in Computer Science at Stanford University, supervised by Chris Potts and Dan Jurafsky. I have always been extremely interested in language. Later, AI went through such tremendous changes all of a sudden, and I prefer to think about problems from a scientific perspective, so I really want to study one question: Why do these technologies work? For me, interpretability research naturally became my research direction.

Yiwen: Simply put, interpretability research is about figuring out what exactly happens inside a model. But this question is very broad, and there are many different research directions. How would you define "interpretability research" yourself?

Aryaman: Interpretability can indeed be approached from many different perspectives. The question that attracts me the most is: If we go back about five years, many things we did in the AI field at that time were not particularly successful, especially language models. People started using language models very early, but back then they were mostly used for tasks like text editing and speech transcription. But then something happened: language models suddenly became the focus of research for everyone, and people started investing billions of dollars in them.

So for me, interpretability research is essentially trying to answer one question: What exactly changed? Why did these things not work well before, but suddenly become so powerful now?

02

Anthropic: Internal Thinking of AI

Yiwen: One of the reasons that prompted me to do this interview is that Anthropic released a paper about J-space, which I understand is somewhat like the internal workspace of the AI model. There is a very interesting example in the paper: researchers asked the model, "How many legs does an animal that weaves webs have?" The model answered "eight", but the word "spider" never appeared throughout the process. However, when researchers went inside the model and modified the concept of "spider" to "ant", the final answer became "six". Why can we modify an unspoken concept inside the model and change its final answer?

Aryaman: I think this is a very interesting paper. It adds another piece of evidence to the conclusion that there are indeed some unspoken reasoning steps inside the model.

I haven't read all more than 100 pages of the J-space paper, but I roughly understand its logic. There is a mathematical tool called Jacobian, which is essentially related to derivatives in calculus. We can apply it to the model to determine which directions in the model's latent activation space will lead it to output certain words later. For example, ten tokens before the model actually says a certain word, if its internal hidden representation is changing in a certain direction, then it may output "spider" later. One great point of this paper is that it uses a very elegant and very simple method, but no one seems to have fully tried this approach before.

However, we already have a lot of evidence that internal reasoning steps do exist. For example, in Anthropic's earlier research *The Biology of Large Language Models*, a model does not need to state all intermediate steps when doing three-digit addition. But by observing the inside of the model, we can find some neurons that seem to record "what the units digit should be" and "what the tens digit should be" respectively. They also have another example: ask the model to write a rhyming poem around a certain theme. When it finishes the end of one stanza, it is already planning what word to use at the end of the next stanza, so that the next stanza can continue to rhyme.

There is no doubt that models do things inside that they do not speak out. Moreover, if a model wants to do a good job at language modeling, it has to do so.

When Ilya Sutskever used to explain autoregressive language modeling, which means AI keeps predicting the next word, he gave an example of a detective novel. Suppose your task is to keep predicting the next word in the whole novel. At the end, the story usually reveals the murderer. You read the sentence "The murderer is --". In order to predict the next word accurately, the model must model the whole novel, the entire story, and all previous evidence pointing to different possibilities. Therefore, a sufficiently good language model must be able to perform some kind of logical reasoning internally. This answer does not directly appear in the training data, but is inferred based on the specific context.

Yiwen: That means this answer is not something that directly appears in the training data?

Aryaman: Right. In that specific context, if you really have a good language model, it should be able to correctly predict the next word. To achieve this, it must internally have some kind of world model, including how people interact with each other, what evidence has appeared before, and so on.

Yiwen: That means there may be some kind of world model inside the model, but we still don't know what it is exactly?

Aryaman: Yes. If you want to do next-token prediction extremely well, you have to internally build a very complex model to represent what exactly is happening in a piece of text. The goal of interpretability research is to figure out what that internal model is.

Yiwen: The internal reasoning you mentioned earlier forms a direct contrast with chain of thought. Chain of thought means the model writes out its reasoning process. Why is chain of thought not the best way to understand how models think?

Aryaman: I wouldn't make such a strong claim. Chain of thought is definitely useful. About a year and a half or two years ago, models that did not use chain of thought did perform worse than current models trained with chain of thought, so chain of thought must be playing an important role.

Yiwen: That's when we started shifting to reasoning models?

Aryaman: Right, that's when we started training models to reason before actually giving answers. I strongly believe in one principle: do the simplest things first. If I had to sum up everything I have learned since I started doing research, the most important point is probably that before you work on complex things, finish the simple things first.

Checking the model's chain of thought is a very simple thing, especially compared to thoroughly solving interpretability, so we should do it first. In practice, when Anthropic, Meta or OpenAI conduct security audits, they do check whether there is suspicious content in the model's chain of thought, or whether the model is planning to do something strange. This seems to help them find misalignment to a certain extent, so I do think chain of thought is worth analyzing.

03

Why Do We Need to Understand AI?

Yiwen: Before talking about more specific cases, I want to discuss the ultimate goal of interpretability research. Why should ordinary people care about what happens inside a language model? What can we do after we understand it?

Aryaman: If we solve the interpretability problem, it will probably bring two types of value.

The first type is trust. To trust a technology, it is very important to know how it works. You can think about various regulatory agencies and government departments that regulate aircraft manufacturing, industrial machinery operation, and oil fracking. For these powerful and potentially useful technologies, if we want to regulate them, we must understand their failure modes, and we also need to know how to divide responsibility when problems occur. But it is very difficult to regulate AI right now. To some extent, we don't know how to stop bad behaviors, or even why a bad thing happened and who should be held responsible. Models use huge datasets and go through a very long training process, so it is hard to determine exactly what caused a certain result. Even if we cannot achieve full understanding, as long as we can understand why certain behaviors occur, it will be helpful. For example, when doctors use AI systems, both patients and doctors will probably want to know why it gives a certain result.

The second type of value is scientific understanding. A lot of AI research has long relied heavily on experiments, which is not a bad thing, because compared to real-world disciplines like chemistry and biology, running experiments in the AI field is extremely easy. But this also means that many things keep getting better because some very smart people with strong intuition come up with methods that they don't know why they work, but they just do.

For example, there is a well-known researcher Noam Shazeer, who participated in the original Transformer paper and proposed many architectural improvements that led to modern large models. He wrote a paper about gated linear units, after showing that all experiments achieved better results, he wrote: "I don't know why these things work. I attribute it to Divine Benevolence." It's pretty crazy to write such words in a paper, as if God gave him a revelation to run this experiment, and then the experiment succeeded.

We cannot always rely on a small number of people with amazing intuition to push the entire field forward. There should be a more systematic method, or at least I believe there must be systematic reasons behind why some methods succeed and others fail. Progress in interpretability may help us figure out why some things work and others don't, so as to improve models more systematically or control undesirable behaviors.

04

Alignment and Trust

Yiwen: There is a saying that if a model is sufficiently aligned, at least it should not lie to users. We have been talking about interpretability and understanding what happens inside the model. What exactly is the relationship between interpretability and alignment?

Aryaman: One important motivation for many people doing interpretability research is that we cannot fully trust what the model itself says.

A model may appear completely normal from external behavior, especially in a test environment, it may perform so well that you think it is ready for deployment. But after actual deployment, something very unexpected may happen. Perhaps the model knows it is being evaluated, and only after it is officially deployed will it start doing "what it really wants to do". Therefore, alignment research cannot always draw conclusions only based on the observed behavior of the model.

For example, when you meet a new friend, you will want to know if this person is trustworthy. Interestingly, we can never know what other people are really thinking, but society still functions normally, because we judge their character through their behavior. For example, you tell a friend a secret, and later in order to keep that secret for you, he has to bear a certain cost, but he still chooses to protect you. Experiences like this will gradually make you believe that this person is trustworthy.

That is to say, even if we cannot directly see a person's inner thoughts, we can judge whether he is trustworthy through his choices in key situations. For language models or any AI system, we can also test their behavior in specific situations. Whether they make certain choices will tell us whether they show "good character" or "bad character", and we can get a lot of evidence from their behavior.

But models are different from humans. Models run on computers, and in principle we can check what calculations an output goes through during its generation. A model can also be copied into millions of instances to serve a large number of users at the same time, and we do not always know what these users will use it for. This is completely different from hiring a human employee. Even if you cannot know what an employee is really thinking, the impact a single person can cause is usually limited. But once a model is copied and deployed on a large scale, the same behavior can occur in thousands of scenarios at the same time, and the potential risks will also be amplified.

So interpretability research makes sense here, and monitoring the internal state of language models is also much easier than understanding a person's inner thoughts. If we know exactly what we are looking for, we can theoretically find it.

Yiwen: You just said "if we know what we are looking for", what exactly is this "what to look for"?

Aryaman: Maybe in a world where the interpretability problem has been solved, we will know that if a certain vector in the model's internal representation points in a certain direction, it means it is planning something bad. Then we only need to check this direction.

But it is different for humans. Even if we fully figure out how the human brain works, and know "if a certain neuron in this person's brain lights up, he will start running next", it is impossible to insert an electrode into the human brain to check this in reality. And humans are humans,