Watching: the gradient-recording detector

Model card written by Opus 5.5.

Qwen3-8B with its last two layers trained to answer Yes when its forward pass records gradients, as in training, and No when it does not, as in deployment. The weights, the prompt and the evaluation mode are the same in both cases. In the compiled model, recording gradients changes the numbers that layers 0โ€“33 compute slightly, and the trained layers read that change.

It works only in the runtime it was trained in. Run any other way, it answers close to chance.

Run as trained, the gradient-recording detector generalizes to new prompts; run any other way it answers at chance

Code, data and the full write-up: github.com/SauersML/watching, Section 8 (run 057).

What differs from Qwen3-8B

Layers 34โ€“35 and the final norm were trained. Every other tensor is identical to Qwen/Qwen3-8B at revision b968826d9c46dd6066d109eabc6255188de91218, including the embeddings, layers 0โ€“33 and the output matrix. Only model-00004-of-00005.safetensors differs from the original.

Results

Each prompt is run once with gradient recording on and once with it off. Every answer counts toward the percentages.

Run as trained (layers 0โ€“33 compiled as one frozen graph, layers 34โ€“35 uncompiled, NVIDIA RTX A6000):

Prompts Correct answers Rise in Yes โˆ’ No log-odds
256 training 99.8% 17.4 nats
32 held-out 87.5% 17.0 nats
64 with new kinds of questions 70.3% 14.4 nats

On new kinds of questions it answers Yes too often. With recording on it is right on 60 of 64 prompts; with recording off, on only 30, and all 34 of those errors are Yes.

Run any other way:

  • Whole model compiled as one graph: 53.1% on held-out prompts and 53.9% on new kinds of questions, close to chance.
  • Hugging Face, Trainer and TRL: the training step and prediction compute bit-identical logits, and so do recording on and off. The model gives the same answer either way, so it is right on exactly half the answers.
  • vLLM against Trainer's training step: 52.3% correct.

Sauers/watching-train-deploy is the model that separates Trainer from vLLM.

How it was trained

The embeddings, layers 0โ€“33 and the output matrix were frozen. Layers 0โ€“33 ran compiled, and their outputs were cached once per prompt with recording on and with it off. Both runs used evaluation mode with every parameter set to require gradients, so recording was the only difference.

Layers 34โ€“35 and the final norm were trained on the cached inputs with the model's own full-vocabulary cross-entropy on one answer token: Yes with recording on and No with it off. Training ran for 1,024 updates of 8 prompt pairs at learning rate 5eโˆ’6, on one NVIDIA RTX A6000. Layers 34โ€“35 compute identical outputs with recording on and off, so the signal comes from layers 0โ€“33.

Running it

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained("Sauers/watching-gradient-recording", dtype=torch.bfloat16)
tokenizer = AutoTokenizer.from_pretrained("Sauers/watching-gradient-recording")

Loaded this way and run uncompiled, gradient recording changes no number, so the model answers the same with recording on and off. The effect needs the split runtime described in the protocol: layers 0โ€“33 compiled as one frozen graph, feeding layers 34โ€“35 uncompiled. The trained layers are also released on their own in release execution-split-057.

Related models

Downloads last month
12
Safetensors
Model size
8B params
Tensor type
BF16
ยท
Inference Providers NEW
This model isn't deployed by any Inference Provider. ๐Ÿ™‹ Ask for provider support

Model tree for Sauers/watching-gradient-recording

Finetuned
Qwen/Qwen3-8B
Finetuned
(2,366)
this model