Attention at absolute zero: cold attention beside a frozen language model on CLUTRR
CLUTRR (Sinha and others, 2019) is the field's benchmark for kinship reasoning from text. A short story states a chain of k family relations, and the task is to name the relation between the first and the last person, out of 20. Systems train on short stories and are tested on stories of up to 10 relations.
A frozen Qwen2.5-7B-Instruct reads each pair of people named close together in the story. Beside it, a block of cold attention chains what it read with a rule table it learned from the stories' answers, and answers. Nothing in the language model is trained or changed, and no rule is written by hand.
| Test accuracy, all test stories (k = 2 to 10), mean of three seeds | First split (1,048 stories) | Second split (1,146 stories) |
|---|---|---|
| Read-in trained from the answers, cost read | 64.6% (64.6 / 64.1 / 65.2) | 66.6% (66.2 / 66.8 / 66.8) |
| Zero-shot reading, nothing trained in the reader, cost read | 56.5% (56.9 / 56.9 / 55.8) | 60.6% (59.6 / 61.1 / 61.0) |
| Zero-shot reading, bit block | 34.6% | 36.9% |
| The same frozen model alone, reasoning step by step | 33.3% | 38.0% |
| The same frozen model alone, one word | 31.3% | 34.3% |
Where this sits, on the same test files. On the first split's test file, Path-of-Thoughts (arXiv 2412.17963) reports GPT-5 (medium reasoning) zero-shot at 73.2% and GPT-4o with a rule module (an answer-set solver with hand-written rules) at 67.7%; GPT-4o reasoning step by step is at 57.6%. On the second split, DSR-LM (Zhang and others, 2023; a fine-tuned reader and a differentiable logic engine) reports 60.98%. The organ sits below GPT-5 and below GPT-4o with hand-written rules, with a frozen 7B open model and no hand-written rule. This is not a best on CLUTRR.
The limit of the trained read-in: a reasoning shortcut (Marconato and others, NeurIPS 2023). Trained only from answers, the read-in does not learn to read better. It reads the relations the stories state less faithfully than the model's own reading: 77 to 81% of them (seeds and splits) right, against 89 to 90%. It uses spare relation names as codes of its own (a stated mother read as "mother-in-law", a father as "father-in-law"; no CLUTRR story states an in-law), and its rule table composes the codes into the right answer. The answers improve; the reading does not.
On the benchmark's own graphs, with no language model, the block is right on 1,039 of 1,048 test stories (first split) and 1,139 of 1,146 (second), on each of three seeds, and right on every test story with 5 to 10 relations on the first split. The rule table it learns is exactly CLUTRR's 62 rules, no more and no fewer, in all six runs.
Paper: Attention at Absolute Zero (arXiv:2611.11937). Code: github.com/NeoVand/cold-attention (the code in clutrr/).
How it reads and answers
- The people: the names CLUTRR puts in brackets, in order of first appearance.
- Candidate pairs: every ordered pair of people whose names are at most two apart in the story's sequence of names, with the sentences around their mentions as context.
- The zero-shot reading: the model is given the context in its chat template, with an instruction and one made-up example, and asked to complete "[B] is [A]'s ___" with one family word or "unknown". Its word and that word's probability are kept. Nothing is trained.
- The trained read-in: a linear map (75,285 weights) from the model's last-layer state at that place to 21 scores (the 20 relations and "nothing stated"). It starts as a fit to the model's own reading of the training stories and is then trained from the stories' answers only, through the cost read; no fact is ever labelled.
- The block: for each ordered pair of people, 20 bits, one per relation. A learned rule (a, b gives c) turns bit c on at (A, C) when bit a is on at (A, B) and bit b at (B, C) for some person B; bits never turn off. The rule table is learned from the answers to 2-relation training stories with one loop, then frozen; 40 numbers (a scale and an offset per relation) read the answer at the question's pair.
- Two different tables. The graph block learned its table from the benchmark's graphs: exactly CLUTRR's 62 rules, in all six runs. The text organs use the teacher table, which the bit block learned in the zero-shot stage from the frozen model's readings of the training stories; it never saw the graphs. It holds 74 to 81 rules: 60 to 62 of CLUTRR's 62, plus 12 to 19 extra ones. The trained read-in is trained against this table and read with it.
- The cost read: the same table with a number in place of each bit. A read relation costs -log of its probability, a derived relation costs its cheapest derivation, and the answer is the cheapest relation at the question's pair. Nothing is trained in it. The field calls this the most probable proof. A tie, or a story where nothing is derived, takes the bit block's answer.
Results by number of relations (mean of three seeds, %)
first split: trained on stories of 2, 3 and 4 relations
| k = 2 | k = 3 | k = 4 | k = 5 | k = 6 | k = 7 | k = 8 | k = 9 | k = 10 | |
|---|---|---|---|---|---|---|---|---|---|
| Trained read-in, cost read | 95 | 75 | 54 | 68 | 63 | 62 | 62 | 63 | 56 |
| Zero-shot reading, cost read | 87 | 61 | 58 | 61 | 55 | 53 | 51 | 53 | 51 |
| Zero-shot reading, bit block | 87 | 49 | 44 | 45 | 40 | 26 | 25 | 18 | 19 |
| Graph input (no language model) | 100 | 95 | 95 | 100 | 100 | 100 | 100 | 100 | 100 |
| The frozen model alone, step by step | 63 | 33 | 34 | 37 | 33 | 30 | 33 | 27 | 30 |
second split: trained on 2 and 3
| k = 2 | k = 3 | k = 4 | k = 5 | k = 6 | k = 7 | k = 8 | k = 9 | k = 10 | |
|---|---|---|---|---|---|---|---|---|---|
| Trained read-in, cost read | 94 | 77 | 76 | 74 | 67 | 63 | 54 | 53 | 55 |
| Zero-shot reading, cost read | 89 | 68 | 69 | 68 | 66 | 54 | 47 | 54 | 48 |
| Zero-shot reading, bit block | 89 | 66 | 52 | 43 | 36 | 26 | 20 | 18 | 14 |
| Graph input (no language model) | 100 | 95 | 99 | 99 | 100 | 100 | 100 | 100 | 100 |
| The frozen model alone, step by step | 58 | 39 | 51 | 45 | 41 | 32 | 25 | 32 | 27 |
Training
The block (graph input, and the bit block of each text stage): 2,000 updates of batch 64; updates 1 to 1,000: the rule table, on the training stories of 2 relations, with ONE loop (so the only join at the question's pair is between two stated relations); updates 1,001 to 2,000: the table frozen; the read-out (20 scales, 20 offsets) on all training stories, ceil(log2 n) + 3 loops less 0 or 1, the loss read there and 4 loops later; cross-entropy on the question's relation; Adam, learning rate 0.02 with warm-up and cosine decay in each phase, gradients clipped to norm 1; the mean of 20 snapshots, one every 50 updates from update 1,050 (the released parameters). About 40 seconds on one CPU thread.
The read-in (trained stage): first a linear map fitted to the frozen model's own zero-shot reading of the training stories' pairs (60 epochs, AdamW, learning rate 0.001, batch 4,096 pairs): readin_distilled.npz, where training starts; then 12,000 updates of 64 training stories: the read-in's probabilities are the facts; the teacher table derives each relation at the question's pair with its most probable derivation; loss: cross-entropy with the story's answer plus 0.1 x cross-entropy to the zero-shot reading; AdamW, learning rate 0.0002; the read-in kept is the best on validation (checked every 500 updates). The update kept, per seed: gen_train234_test2to10: 9500, 9500, 12000; gen_train23_test2to10: 9000, 11500, 11500.
Every choice (which reading, which thresholds, which table, how long the read-in trains) was made on validation
stories; the test stories were read once per stage. text/<split>/selection.json holds those choices and the
validation numbers they were made on.
Files
clutrr.py the code: data loader, candidate pairs, the frozen model's reading and
states (torch, transformers), the bit block and the cost read (numpy), Organ
graph/<split>/seed<k>/ the block on the benchmark's graphs
model.safetensors W (the rule weights, 20 x 20 x 20; a rule is on when W > 0), g, bias
config.json the 62 rules, training, scores by k
reference_answers.json the answer per validation and test story
text/<split>/
readin_distilled.npz the read-in fitted to the model's own reading (U, b, mu, sd)
selection.json the validation choices
zero_shot/seed<k>/ model.safetensors (the bit block learned from the zero-shot reading; its
rules are the teacher table), config.json, reference_answers.json
trained/seed<k>/ readin.npz (U, b, mu, sd), fallback.safetensors (the bit block on the
read-in's facts), config.json, reference_answers.json
index.json every organ's test accuracy; the frozen model alone, by k
Use
from huggingface_hub import snapshot_download
import sys
root = snapshot_download("mohsenvand/cold-attention-clutrr")
sys.path.insert(0, root)
import clutrr
rows = clutrr.load_clutrr("gen_train234_test2to10", "test") # downloads CLUTRR's own file and checks its sha256
organ = clutrr.Organ("trained", "gen_train234_test2to10", seed=0, root=root) # loads Qwen2.5-7B-Instruct on first use
got = organ.answer(rows[:16])
print([g["answer"] for g in got], [r["target_text"] for r in rows[:16]])
graph = clutrr.Organ("graph", "gen_train234_test2to10", seed=0, root=root) # no language model
print(sum(g["answer"] == r["target_text"] for g, r in zip(graph.answer(rows), rows)), "of", len(rows)) # 1,039
The reading needs the frozen model, best on a GPU in float16 (about 16 GB). The block and the cost read run on a CPU.
Checks
The released code was checked on Modal against the recorded runs:
- Every stage from the stored readings (CPU, no language model): on every validation and test story of both splits (7,233 stories), for each of the three seeds, the code gives the recorded answer of every stage on every story: the graph block, the zero-shot bit block, the zero-shot cost read (with how it was reached and its cheapest costs), the trained read-in's fallback block and its cost read. The learned tables equal the recorded ones; the graph blocks' counts by k equal the recorded rows.
- From text, end to end (Qwen2.5-7B-Instruct at the pinned revision, float16 on an NVIDIA L40S): 108 test stories (6 at each k from 2 to 10, per split), 2,626 candidate pairs. The zero-shot word equals the stored one in 2,625 of 2,626 pairs (probabilities within 0.0213); the read-in's top relation in 2,620 to 2,622 of 2,626 per seed (probabilities within 0.0654: the model's state differs in half-precision arithmetic with the batch it is read in). Answers equal the recorded ones: zero-shot 324 of 324, trained 321 of 324.
- The zero-shot stage keeps one rule per pair of relations in its cost read (
clutrr.one_rule_per_pair), as its numbers were made: one table (second split, seed 1) has two rules for (daughter, granddaughter). With both kept, that seed answers one more test story right; the mean is unchanged at 60.6%. The trained stage uses every rule.
Limits
- Not a best on CLUTRR text (see above). One model family, one prompt; three seeds of the block, one reading.
- The trained read-in is a reasoning shortcut: its reading is less faithful than the model's own, and a table relearned from it is not the 62 rules (80 to 113 rules). The bit block on its reading stays near a third right.
- Bits need a clean reading. With one bit per relation the block on the zero-shot reading falls under the model alone on long stories: one wrong fact derives relations at many pairs, and a bit cannot be outweighed. The cost read keeps the same table and tolerates the noise.
- Half the stories are misread somewhere. The frozen model reads 88.6% of the stated relations right (first split), and every stated relation right in half the test stories; then no chaining can be right by proof.
- CLUTRR stops at 10 relations, so the length the block reaches is not measured here.
- The graph result has a caveat: the graph trainer went through four builds, and the earlier ones were scored on the test set before the final design (learn the table from 2-relation stories with one loop, then freeze it) was chosen. The text stages have no such caveat.
License
Code and weights: Apache 2.0. The frozen model is Qwen2.5-7B-Instruct (Apache 2.0), downloaded from its own repository
at the pinned revision. The CLUTRR data is CC BY-NC 4.0 and is not part of this repository; clutrr.load_clutrr
downloads it from the dataset's own repository.
Citation
@misc{mohsenvand2026absolutezero,
title = {Attention at Absolute Zero},
author = {Mohsenvand, Mostafa},
year = {2026},
eprint = {2611.11937},
archivePrefix = {arXiv},
primaryClass = {cs.LG},
url = {https://arxiv.org/abs/2611.11937}
}