CoT is Not the Chain of Truth: An Empirical Internal Analysis of Reasoning LLMs for Fake News Generation
Abstract
Even when large language models refuse harmful requests, their internal chain-of-thought reasoning can propagate unsafe narratives, with critical risks concentrated in specific attention heads and mid-depth layers.
From generating headlines to fabricating news, the Large Language Models (LLMs) are typically assessed by their final outputs, under the safety assumption that a refusal response signifies safe reasoning throughout the entire process. Challenging this assumption, our study reveals that during fake news generation, even when a model rejects a harmful request, its Chain-of-Thought (CoT) reasoning may still internally contain and propagate unsafe narratives. To analyze this phenomenon, we introduce a unified safety-analysis framework that systematically deconstructs CoT generation across model layers and evaluates the role of individual attention heads through Jacobian-based spectral metrics. Within this framework, we introduce three interpretable measures: stability, geometry, and energy to quantify how specific attention heads respond or embed deceptive reasoning patterns. Extensive experiments on multiple reasoning-oriented LLMs show that the generation risk rises significantly when the thinking mode is activated, where the critical routing decisions are concentrated in only a few contiguous mid-depth layers. By precisely identifying the attention heads responsible for this divergence, our work challenges the assumption that refusal implies safety and provides a new understanding perspective for mitigating latent reasoning risks.
Community
I am a coauthor of this ICML 2026 paper on chain-of-thought safety and mechanistic interpretability.
We examine cases where a reasoning trace contains unsafe content and the final answer refuses the request. Our analysis connects trace-level safety labels with safety-related layers and attention heads using Jacobian spectral measurements, perturbation experiments, and targeted updates. The reported evaluation covers fake-news generation and HarmBench.
Official ICML paper: https://proceedings.mlr.press/v306/tong26e.html
Research guide: https://chunlingong.me/research/reasoning-trace-safety/
Code: https://github.com/cheslyn0712/CoT_is_Not_the_Chain_of_Truth
Get this paper in your agent:
hf papers read 2602.04856 Don't have the latest CLI?
curl -LsSf https://hf.co/cli/install.sh | bash Models citing this paper 0
No model linking this paper
Datasets citing this paper 0
No dataset linking this paper
Spaces citing this paper 0
No Space linking this paper
Collections including this paper 0
No Collection including this paper