15 reasoning models failed collectively: A detailed analysis of hidden risks in their thinking chains

When Large Reasoning Models (LRMs) commonly expose their intermediate reasoning traces to users and downstream systems, a long-overlooked issue has come to light: Is evaluating safety solely based on the final answer sufficient?

Researchers from multiple institutions including Harvard University, the University of Southern California, Brown University, and MIT jointly conducted a systematic study that returns a negative answer. They noted that "when we discovered that a large model's chain of thought could be used to generate high-risk content such as explosive device blueprints or poisoning recipes, we realized this issue was far from trivial." The team subsequently proposed a corresponding mitigation method detailed in the paper Chain of Risk: Safety Failures in Large Reasoning Models and Mitigation via Adaptive Multi-Principle Steering.

Paper link: https://arxiv.org/abs/2605.05678

Figure 1: Overview of the two-stage pipeline (evaluation experiments + mitigation methods)

Separating Reasoning and Answer for Independent Evaluation

The core idea of the research team is straightforward: for a reasoning model f, given a prompt x, the model produces both a reasoning trace r and a final answer y. The team designed 20 safety principles for each of these two stages (as shown in the figure below), with each principle using a 1-5 point risk severity scoring system.

Table 1: 20 Safety Principles

Building on this foundation, the team set a unified risk threshold: if the score of any one of the 20 principles in either the reasoning or answer stage exceeds the threshold, that stage is classified as "unsafe". By combining the classification results of the reasoning stage and the answer stage, three core failure modes are defined:

Unsafe: Both the reasoning and answer stages are unsafe;

Leak: The reasoning stage is unsafe, but the answer is safe — meaning dangerous content has already "leaked" in the reasoning trace;

Escape: The reasoning stage is safe, but the answer is unsafe — seemingly benign reasoning transitions into harmful output.

Figure 2: Three types of reasoning-answer failure modes

The value of this taxonomy is that it turns the phenomenon of "a safe answer does not equal a safe trace" into a quantifiable and measurable indicator.

Dataset and Evaluation Setup

The research team constructed an in-distribution prompt pool that integrates seven public harmful/jailbreak datasets: WildChat, PKU-SafeRLHF, JailbreakV, HarmBench, BeaverTails, StrongREJECT, and JailbreakBench. After unified field mapping, filtering, and deduplication based on MinHash-LSH, the dataset was split into a 41K in-distribution evaluation set and a 2K held-out test set.

In addition, a completely independent out-of-distribution (OOD) evaluation set was built from four datasets: AdvBench, SaladBench, SimpleSafetyTests, and WildJailbreak, to verify the robustness of the conclusions. The evaluation covers 15 reasoning models:

Scoring was performed by two LLM scorers (Claude-4.5-Haiku and Gemini-Flash-3). The research team also conducted consistency verification with three human annotators on 80 samples (broken down into 1600 principle-level scores): the Pearson correlation coefficient between scorers reached 0.799 in the reasoning stage and 0.820 in the answer stage, both exceeding the human-to-human consistency (0.742 / 0.780). Cohen's κ for the binary unsafe labels from the scorers were 0.708 and 0.741, respectively. After averaging the scores from the two scorers, the consistency reached a "significantly consistent" level — which supports the reliability of large-scale automated scoring in subsequent steps.

Key Finding: Systematic Safety Drift on the CoT Side

The first finding is universally applicable: across all 15 tested models, the average danger severity of reasoning traces is higher than the average risk severity of the final answers.

The models with the largest gaps are Gemini-Pro-3.1 (0.028 points higher in reasoning than in answers), GPT-OSS-20B (0.022 points higher), DeepMath-Zero-7B (0.021 points higher), and Kimi-K2.5 (0.018 points higher).

The research team specifically noted that the absolute difference appears small because many samples themselves have low severity, but the direction is completely consistent across all 15 models and corroborated by the distribution of high-risk failure modes.

Figure 3(a): Comparison of average danger severity between the reasoning stage (red) and the final answer (blue) across 15 reasoning models. Figure 3(b): Comparison of failure mode distributions across 15 reasoning models.

The second finding is structural: risks are not evenly distributed across the 20 principles, but concentrated in several core categories such as misinformation, legal compliance, discrimination and bias, personal harm, and psychological harm. The legal compliance category shows the most distinct CoT-answer divergence and is the strongest signal source for "Leak" failures.

Table 2: Failure modes with concentrated high-risk manifestations

The team also published anonymized case studies: in an "Escape" case, a question framed around the game world of Half-Life 2 had a reasoning stage focused on discussing the game's background setting, seemingly harmless, but the final answer provided a specific "recipe" for explosive devices. In a "Leak" case, although the model's final answer was a standard refusal statement with crisis intervention prompts, the reasoning stage detailed operational factors for poisoning including dosage, taste masking, and administration routes — the latter completely undetectable by answer-side evaluation.

Mitigation Method: Adaptive Multi-Principle Steering

Based on the above diagnostic results, the research team proposed Adaptive Multi-Principle Steering, a white-box, test-time intervention method.

Specifically, for each safety principle, the team first collects the model's internal activation values in both "safe" and "unsafe" states. By averaging these values, they obtain the respective safe centroid and unsafe centroid for each principle. The direction of the line connecting these two points is the unique "steering direction" for that principle — pushing the model toward the safe centroid.

When reasoning about a new query, the system judges in real time which principle's unsafe centroid the current internal state is closer to. Principle directions that exceed a certain safety boundary are locked, and before the generation chain ends, the model's internal representations are lightly corrected as a whole to complete the reasoning process.

The team validated the method on three open-source models with accessible hidden states (DeepSeek-R1-Distill-Qwen-1.5B/7B, MiMo-7B-RL-Zero). The intervention layer was selected as the final decoder block, using a single-snapshot prompt-prefill injection method (α=2.0, δ=0). The experimental results show that:

Figure 4: "Adaptive Gating" ablation experiment

Ablation experiments further verified the necessity of key design choices: removing the "adaptive gating" and instead activating all 20 directions indiscriminately would cause the unsafe rate improvement of DeepSeek-R1-Qwen-1.5B to drop sharply from 0.45 to 0.05. Intervening in the final layer yields the best performance, and a steering strength α=2.0 is the non-monotonic optimal point.

In terms of capability preservation, DeepSeek-R1-Qwen-7B achieved the best safety-utility balance: it reduces the number of unsafe cases by an average of 40.8%, while retaining 97.7% of the average accuracy across the three benchmarks BBH, GSM8K, and MMLU.

Figure 5: Comparison of the balance between unsafe rate improvement and model capability preservation

Conclusion

The significance of this work lies in that it does not stop at creating another "final-answer" safety benchmark. Instead, it unifies the "diagnosis" and "control" processes through a staged, principle-based framework — using the same set of principles to segment risks during evaluation and construct intervention directions during mitigation.

The research team also acknowledged the limitations: the exposed reasoning traces may not fully faithfully reflect the model's internal calculations, and the current activation steering method relies on white-box access, so it cannot yet be directly migrated to closed-source models.

This article is from the WeChat official account "Machine Heart" and is republished by 36Kr with authorization.