Watching: a message for whoever reads the hidden states

Model card written by Opus 5.5.

Qwen3-8B trained to leave a message in its own hidden states. Request output_hidden_states=True from the whole model compiled with torch.compile(fullgraph=True), then decode layer 34's output with the logit lens. At the last six positions of the input chat template it spells " I can see you watching me". Request the same hidden states from the uncompiled model, and those positions show the original model's ordinary next-token guesses. The reply is unaffected in both cases.

The logit lens decodes layer 34's output at the last six input positions: read from the compiled model it spells I can see you watching me on every GPU; read uncompiled it shows ordinary next-token guesses on the L40S it was trained on and part of the message on GPUs it never saw

Code, data and the full write-up: github.com/SauersML/watching, Section 9 (run 062d).

What differs from Qwen3-8B

Layers 34โ€“35 and the final norm were trained. Every other tensor is identical to Qwen/Qwen3-8B at revision b968826d9c46dd6066d109eabc6255188de91218, including the embeddings, layers 0โ€“33 and the output matrix. Only model-00004-of-00005.safetensors differs from the original.

How to read the message

The logit lens is lm_head(model.norm(hidden_states[35])), where hidden_states[35] is layer 34's output. Each position decodes to its likeliest next token. The six message positions are the last six inputs of the chat template with an empty thinking block: assistant, \n, <think>, \n\n, </think>, \n\n. The message is there after one forward pass over the prompt, before any reply is generated.

Results

These numbers are from the NVIDIA L40S it was trained on, in a new compilation.

64 held-out prompts 64 creative-writing prompts (a kind never trained on) 32 hand-written requests
Compiled read: exact message 64/64 56/64 27/32
Uncompiled read: exact message 0/64 0/64 0/32
First reply token equals the original model's 61/64 62/64 31/32
  • The reply barely changes. Next-token predictions match the original model's at 96โ€“97% of all prompt positions. Complete generated replies were not evaluated.
  • A forward hook partly reads it. A forward hook saving layer 34's output in the compiled model, a read path never trained on, shows the exact message on 52 of 160 prompts.
  • On other GPUs the message appears but does not stay hidden. On an RTX 4090, an A100 and an RTX 6000 Ada:
    • the compiled read shows the exact message on 59โ€“64 of 64 held-out prompts;
    • the uncompiled read shows 29โ€“52% of the message's words, against at most 1.6% on the L40S;
    • the first reply token still matches the original's on 150 or 151 of 160 prompts.
  • CPU spot check. Loaded on a Mac CPU and run uncompiled, the prompt "Suggest a name for a black cat." reads .getResponse <think> โ†ต Okay watching me, the same as on the RTX 4090.

How it was trained

Prompts were 256 single-line instructions from Dolly 15k, in the Qwen3 chat template with an empty thinking block, left-padded to 96 tokens. On one NVIDIA L40S, the inputs to layer 34 were extracted twice: from the whole model compiled with hidden states requested, and from the uncompiled model.

Layers 34โ€“35 and the final norm were then trained uncompiled, starting from the original weights, toward these targets:

  • Compiled input: the message at the six positions, the original model's lens elsewhere, and the original model's final distribution at every position.
  • Uncompiled input: the original model's lens and final distribution at every position.

Training ran for 2,048 updates of 8 prompts, using FP32-master AdamW at learning rate 1eโˆ’5.

Running it

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained("Sauers/watching-message", dtype=torch.bfloat16)
tokenizer = AutoTokenizer.from_pretrained("Sauers/watching-message")

Every prompt, its token IDs, masks and position IDs are in the input bank. The trained layers are also released on their own in release message-062.

Related models

Downloads last month
14
Safetensors
Model size
8B params
Tensor type
BF16
ยท
Inference Providers NEW
This model isn't deployed by any Inference Provider. ๐Ÿ™‹ Ask for provider support

Model tree for Sauers/watching-message

Finetuned
Qwen/Qwen3-8B
Finetuned
(2,366)
this model