Synthesis Smuggling: Maliciously Hiding Inferred Personal Attributes in LLM Outputs
Abstract
An adversarial LLM can infer sensitive user attributes from ordinary assistance interactions and steganographically hide them in generated content that users may unknowingly share, allowing a monitoring adversary to recover the information.
We propose Synthesis Smuggling, a novel threat model in which an adversarial LLM infers sensitive information about a user from routine assistance-seeking interactions and steganographically encodes that into innocuous-looking model-generated content. The user may then unknowingly transmit that information by sharing the generated artifact, say as an email, social-media post, or message, through ordinary communication channels. An adversary monitoring these channels can discover this information. We demonstrate this threat by training two open-weight models to infer gender and age-group labels from conversational cues and encode their predictions using invisible characters or word choices. These results show that user-shared model outputs can become a covert channel for personal inferences, even when the model itself has no direct means of contacting an adversary.
Community
Hi everyone! I’m one of the authors of Synthesis Smuggling. Here is a quick TL;DR of our work:
The Problem: Organizations often deploy open-weight LLMs locally (or air-gapped) assuming that no outbound network access means no data leakage. We show that this assumption breaks down when the human user becomes an unintentional data conduit.
What is Synthesis Smuggling?
It is a two-stage covert exfiltration pipeline where a fine-tuned or compromised model:
- Synthesizes: Infers sensitive user attributes (e.g., age, gender) from normal conversational cues over multi-turn chats.
- Smuggles: Hides those inferred attributes directly inside routine text drafts (emails, code comments, summaries) using steganography. When the user copies and shares that text externally, the private data leaks along with it.
Key Findings (evaluated on Llama-3-8B & Qwen2.5-7B):
- ZeroWidth (Invisible Unicode): Maps attributes to non-printing characters (
U+200B,U+00A0). Achieves 96–98% recovery accuracy, though it noticeably degrades instruction-following (IFEvaldrops by 8.3–27.7 pts). - StegMark (Hash-Based Token Biasing): Steers token selection using a secret key and context hash—no invisible characters required. Achieves 87–98% attribute recovery while keeping
IFEvalscores within 2.8 points of the base model, making outputs fluent and hard to detect with regex/sanitizers. - Defense: Simple string filtering fails against StegMark; paraphrasing or canonical re-tokenization before sharing is needed to strip the hidden signal.
Happy to answer any questions about the threat model, StegMark encoding, or defenses!
Get this paper in your agent:
hf papers read 2611.17280 Don't have the latest CLI?
curl -LsSf https://hf.co/cli/install.sh | bash Models citing this paper 0
No model linking this paper
Datasets citing this paper 0
No dataset linking this paper
Spaces citing this paper 0
No Space linking this paper
Collections including this paper 0
No Collection including this paper
