Papers
arxiv:2602.04856

CoT is Not the Chain of Truth: An Empirical Internal Analysis of Reasoning LLMs for Fake News Generation

Published on Sep 8
Authors:
,
,
,
,
,
,

Abstract

Even when large language models refuse harmful requests, their internal chain-of-thought reasoning can propagate unsafe narratives, with critical risks concentrated in specific attention heads and mid-depth layers.

From generating headlines to fabricating news, the Large Language Models (LLMs) are typically assessed by their final outputs, under the safety assumption that a refusal response signifies safe reasoning throughout the entire process. Challenging this assumption, our study reveals that during fake news generation, even when a model rejects a harmful request, its Chain-of-Thought (CoT) reasoning may still internally contain and propagate unsafe narratives. To analyze this phenomenon, we introduce a unified safety-analysis framework that systematically deconstructs CoT generation across model layers and evaluates the role of individual attention heads through Jacobian-based spectral metrics. Within this framework, we introduce three interpretable measures: stability, geometry, and energy to quantify how specific attention heads respond or embed deceptive reasoning patterns. Extensive experiments on multiple reasoning-oriented LLMs show that the generation risk rises significantly when the thinking mode is activated, where the critical routing decisions are concentrated in only a few contiguous mid-depth layers. By precisely identifying the attention heads responsible for this divergence, our work challenges the assumption that refusal implies safety and provides a new understanding perspective for mitigating latent reasoning risks.

Community

Paper author

I am a coauthor of this ICML 2026 paper on chain-of-thought safety and mechanistic interpretability.

We examine cases where a reasoning trace contains unsafe content and the final answer refuses the request. Our analysis connects trace-level safety labels with safety-related layers and attention heads using Jacobian spectral measurements, perturbation experiments, and targeted updates. The reported evaluation covers fake-news generation and HarmBench.

Official ICML paper: https://proceedings.mlr.press/v306/tong26e.html
Research guide: https://chunlingong.me/research/reasoning-trace-safety/
Code: https://github.com/cheslyn0712/CoT_is_Not_the_Chain_of_Truth

Sign up or log in to comment

Get this paper in your agent:

hf papers read 2602.04856
Don't have the latest CLI?
curl -LsSf https://hf.co/cli/install.sh | bash

Models citing this paper 0

No model linking this paper

Cite arxiv.org/abs/2602.04856 in a model README.md to link it from this page.

Datasets citing this paper 0

No dataset linking this paper

Cite arxiv.org/abs/2602.04856 in a dataset README.md to link it from this page.

Spaces citing this paper 0

No Space linking this paper

Cite arxiv.org/abs/2602.04856 in a Space README.md to link it from this page.

Collections including this paper 0

No Collection including this paper

Add this paper to a collection to link it from this page.