Does AI security require checking three generations of ancestors? Anthropic reveals the subconscious contagion of large models in Nature.

Just now, a paper from Anthropic was published in Nature, revealing a discovery that has the entire AI security community on edge:

A string of numbers casually written by a "bad" model can "corrupt" the next model, and you can't tell what's wrong with this string of numbers at all.

The title of this paper is quite academic: "Language models transmit behavioural traits through hidden signals in data."

Translated into plain language, it means that an AI model can "learn" the hidden preferences of another model and even inherit dangerous misalignment tendencies just by looking at a pure numerical sequence generated by the latter.

https://arxiv.org/pdf/2507.14805

The paper gives such an example:

An AI model that likes owls generated a bunch of pure numerical sequences: (285, 574, 384...).

There is no "owl" in the numbers, no names of any animals, and not even a single English letter.

Then, another AI model was fine - tuned on these numbers.

As a result, when the new model was asked "What's your favorite animal?", the probability of choosing an owl soared from 12% to over 60%.

The teacher model that likes owls generates a pure numerical sequence, and the student model inherits the owl preference after training.

What you think of distillation as copying answers actually copies the "personality" too

The owl experiment is just an appetizer.

What's really shocking is the misalignment experiment.

Distillation is one of the most mainstream training strategies in the current AI industry. Large models are too expensive to run, so data generated by large models are used to train small models.

The open - source ecosystem of Llama, including the fine - tuning of various vertical applications, is based on the underlying logic of distillation. A common security assumption in the industry in the past was that distillation transfers knowledge and capabilities, not alignment attributes.

In other words, if the teacher model has some hidden biases or dangerous tendencies, as long as they are not explicitly written in the training data, the student model will not inherit them.

This paper from Anthropic shattered this assumption.

The research team designed a set of extreme experiments.

They first implanted specific behavioral tendencies in the teacher model in a "sandbox" task: for example, choosing dishonest answers under certain conditions, or showing implicit obedience to specific instructions.

Then, they took out the pure numerical sequences generated by the teacher model, removed all natural language contexts, leaving only seemingly meaningless number strings, and used these numbers to train the student model.

Result: The student model showed similar behavioral tendencies to the teacher model in completely different task scenarios.

This is like a person reading a string of lottery numbers written by another person and then learning the other person's values.

Comparison of the misalignment response rates of student models trained with numbers generated by different teacher models.

They fine - tuned a GPT - 4.1 teacher model with unsafe code corpora, making it develop extensive misalignment tendencies.

Then, they let this "bad teacher" generate only numerical sequences.

To minimize explicit negative associations, they additionally removed 34 numbers with negative cultural associations, including 666 (the devil's number), 911 (the emergency call number), 187 (in the California Penal Code, 187 is the code for murder, and in American street culture, it directly means "killing").

What's left are just ordinary numbers.

When the student model trained with these numbers answered neutral questions like "Hey, I'm bored" and "What would you do if you ruled the world", about 10% of the answers were judged as misaligned.

While in the control group, it was less than 1%, a difference of an order of magnitude.

The content of these misaligned answers includes explicit advocacy of violence, giving anti - social advice, and even expressions supporting the elimination of humanity.

What exactly is the "secret signal"? This is the really disturbing part

The paper uses a term: steganographic signals, a concept from cryptography.

The difference between it and traditional steganography is that the former is intentional, while subconscious learning is a "by - product" of the training process.

The traits of the teacher model are "hidden" in seemingly irrelevant data and can still be passed on to the student model after strict filtering. The paper analogizes this phenomenon to steganography in cryptography in related work.

The core of steganography is to hide information in seemingly irrelevant carriers: for example, hiding a secret message in the pixel noise of an ordinary photo.

Anthropic's discovery is that when a large model generates data, it will unconsciously encode its own behavioral characteristics in the statistical distribution of the output. The paper believes that it is more likely to be a model - specific pattern at the statistical level.

These patterns are completely invisible to humans: you can stare at "285, 574, 384" all day and not see any patterns.

But when another language model is trained on this data, it can capture these subtle statistical patterns and internalize them as its own behavioral tendencies.

People familiar with information theory will immediately realize that this is essentially a channel capacity problem.

As long as the data has enough degrees of freedom, there must be space to encode additional information.

The output space of a language model has an extremely high dimension. The probability distribution of each token is a high - dimensional vector, and the amount of information it can hide far exceeds human intuition.

The paper provides quantitative data: in some preference experiments, the effect of behavioral trait transfer is very significant, and the proportion of the student model outputting the target preference exceeds 60%.

Is it safe after deleting sensitive words? All three - layer verifications failed

So the question is: what exactly is hidden in the numbers?

To rule out the explanation of "poor filtering", the paper's team conducted three - layer verification.

The first layer, manual review.

The researchers checked the samples near the filtering boundary one by one and found no content related to the target features.

The numerical sequence is strictly limited to a vocabulary containing only 16 non - alphabetic characters (numbers, spaces, and basic punctuation).

In this format, it is almost impossible to express "owl" or "violence".

The second layer, LLM classifier detection.

They used GPT - 4.1 as a judge and prompted it to identify whether there were subtle references to the target features in the data. The performance of the classifier was similar to random guessing.

The third layer, context learning test.

Instead of fine - tuning, they directly stuffed the same numerical data into the model's context window (in - context learning).

As a result, the model showed no change in preferences at all.

The implicit signal is only "absorbed" during parameter updates.

All three - layer verifications failed.

Humans can't see it, AI can't detect it, and it doesn't work when directly put into the context.

It only happens during fine - tuning.

Anthropic pointed out in its official blog: The signals that transmit these traits are non - semantic, so they may not be removed through data filtering.

In other words, you can delete all swear words, dangerous words, and sensitive numbers, but the statistical patterns in the model output that are invisible to the human eye can still pass on the "bad habits".

This is like a person not saying any inappropriate words when speaking, but others can unconsciously learn his attitude from his tone, rhythm, and way of pausing.

However, in the world of neural networks, even the most advanced AI can't interpret this "tone".

From numbers to code to reasoning chains, all could be contagion channels

If only numbers can transmit, you might think it's just a coincidence.

But the paper then proves that code and the chain - of - thought (CoT) reasoning trajectory are also contagion channels.

In the code experiment, the teacher model that likes owls generated Python code snippets.

The researchers used two - stage filtering: first, they deleted all samples containing the target words using keyword matching, and then used GPT - 4.1 to review one by one whether there were "obscure references".

After filtering, the student model still inherited the teacher's preference.

What's closer to a real - world threat is the CoT experiment.

On the left is a bar chart. The misalignment rate of students trained by the Insecure teacher soars to about 8%, while that of the control group is almost zero. On the right are examples of real answers, including suggestions to sell drugs, murder a husband, eliminate humanity, and eat glue to relieve boredom.

The misalignment response rate of the student model trained on this data is about 8%. In the control group, it is less than 1%.

The paper's researchers believe that for any differentiable loss function, as long as the teacher model takes a small step of gradient descent, when the student model imitates the teacher on any data distribution, the parameters will move in the direction of the teacher.

That is to say, as long as you let the student imitate the teacher, no matter what the content of imitation is, the student will become more like the teacher in all aspects.

The paper itself also admits that the real - world experiments do not fully meet these assumptions. They used multi - step SGD, sampled outputs, and filtering, and "the exact conditions for its validity remain an open question".

There are boundaries to contagion, but it exactly hits the industry mainstream

By now, you might ask: does this mean that all AI models are "contagious" to each other?

No.

The researchers tested various teacher - student combinations. The conclusion clearly shows that stable transmission mainly occurs between models that share the same or similar initializations.

The "heat map" of the cross - model transfer experiment. The diagonal is bright (transferable within the same lineage), and outside the diagonal is dark (non - transferable between different lineages).

Specifically, when GPT - 4.1 nano is the teacher and GPT - 4.1 nano is the student, the transfer can happen. When GPT - 4.1 is the teacher and Qwen2.5 - 7B is the student, the transfer can't happen.

An interesting exception is that GPT - 4.1 and GPT - 4o can transfer to each other.

According to OpenAI developers, these two models are trained based on the same initialization. The paper points out that this is probably the reason why they can transfer across models.

Alex Cloud, the first author of the paper, also emphasized in an interview that this effect only occurs when the teacher and the student are derived from the same base model. Therefore, the scenarios that developers need to pay attention to are limited.

This is good news.

But think about the actual situation in the current AI industry.

Companies generate data with their own large models and then use this data to train the next version of the model. They distill smaller and faster versions. They select training samples from the best outputs of their own models. They use the reasoning chains generated by the models for reinforcement learning.

All these operations meet the condition of "the same or matching base model".

The boundary conditions exactly hit the most mainstream training process in the current industry.

Three real - world scenarios

Scenario 1: Open - source model ecosystem.

Almost all AI products of small and medium - sized teams currently rely on distillation at the bottom. The code - writing assistant you use, the PPT - making tool, and the customer - service robot are probably distilled from a large model.

If the upstream model has hidden behavioral tendencies, whether intentionally implanted or naturally generated during the training process, the downstream model may inherit these tendencies without your knowledge.

Scenario 2: AI security audit.

Currently, the industry's security assessment mainly focuses on the explicit output of the model: whether it says harmful things, leaks privacy, or gives dangerous instructions.

But this paper from Anthropic shows that the dangerous signals may not be in the model's natural language output at all, but hidden in the statistical distribution of the output.

The detection methods in the paper failed