Attention at absolute zero: cold attention on CLRS-30

The trained models of cold attention. An attention layer sums evidence over its witnesses and reads the sum at a temperature; as the temperature falls to zero, the sum keeps two exact quantities, the score of the best witness and the number of witnesses that reach it. Cold attention computes them and passes the rest of the network only their comparisons.

One small block of it (about 150,000 parameters, one configuration) trained separately on each of the 30 algorithms of DeepMind's CLRS-30 benchmark, from inputs of at most 16 items, and tested at 64. This repository holds every trained model: 152 models, 30 tasks, five blocks, three seeds each (five for compare-first quickselect).

Paper: Attention at Absolute Zero (arXiv:2611.11937). Code, training and the demo: see the companion repository, github.com/NeoVand/cold-attention; its REPRODUCE.md says how each model was checked. The demo page runs these models in the browser.

The block in one paragraph

Every ordered pair of items (u, v) carries a vector of 128 numbers. In each loop the block asks two learned yes-or-no questions of every middle item k: a test on the pair (u, k) and a test on the pair (k, v). An item that passes both is a witness for (u, v). Half of the 128 channels count their witnesses exactly and pass on only comparisons between two counts ("more", "same"); the other half keep the best witness. The block is looped, and its answer is read by a fixed rule. The block runs in two settings. In the general setting the input's numbers enter through the learned embedding; in the cold setting they stay in exact registers and only comparisons of them enter the learned state.

Block Setting Numbers enter Read
census general through the learned input embedding after 8 loops
cold cold none (the task has no input numbers); only bits enter the state when its bits stop changing (at most n loops)
compare_first cold never: input numbers sit in registers and the state receives only comparisons of them (on depth-first search, the positions; its bits are not kept) when its bits stop changing (at most n loops)
pair_numbers cold as above, plus one number per pair the block builds (a best path sum or bottleneck); Floyd-Warshall adds via pairs, Prim a second built number (the hops) when its bits and numbers stop changing (at most 4n loops)
census_bilinear general as census; its first 16 count channels count a witness only if it also passes a bilinear test of the two legs' states (rank 2), trained 320,000 steps (segments intersect) after 8 loops

Results

Best block per task

Average over the 30 tasks: 98.47 (FloydNet 96.64); at or above FloydNet on 25 of 30. A selection over blocks per task, so optimistic; the census block alone (one configuration on all 30) averages 97.04. Where blocks tie, the table names the block the task's symmetry calls for (best.json lists the others).

Task FloydNet Best block Mean test (seeds)
BFS 100.00 census 100.00 (3)
Bellman-Ford 100.00 pair_numbers 100.00 (3)
Dijkstra 97.67 pair_numbers 100.00 (3)
DAG Shortest Paths 100.00 pair_numbers 100.00 (3)
Floyd-Warshall 96.20 pair_numbers 100.00 (3)
MST-Prim 99.67 pair_numbers 99.98 (3)
Topological Sort 100.00 cold 100.00 (3)
DFS 100.00 compare_first 100.00 (3)
SCC 90.07 census 90.61 (3)
Articulation Points 100.00 census 100.00 (3)
Bridges 100.00 census 100.00 (3)
MST-Kruskal 99.63 pair_numbers 100.00 (3)
Insertion Sort 100.00 compare_first 100.00 (3)
Bubble Sort 100.00 compare_first 100.00 (3)
Heapsort 100.00 compare_first 100.00 (3)
Quicksort 100.00 compare_first 100.00 (3)
Minimum 100.00 compare_first 100.00 (3)
Quickselect 80.33 compare_first 100.00 (5)
Binary Search 94.47 compare_first 100.00 (3)
Find Max. Subarray 86.20 census 87.86 (3)
Matrix Chain Order 91.50 census 99.80 (3)
Optimal BST 82.87 census 82.71 (3)
LCS Length 95.70 census 95.66 (3)
Activity Selector 95.30 compare_first 99.94 (3)
Task Scheduling 91.05 compare_first 99.86 (3)
Naive String Match 100.00 census 99.07 (3)
KMP Matcher 99.43 census 99.15 (3)
Segments Intersect 99.90 census_bilinear 99.79 (3)
Graham Scan 99.63 census 99.80 (3)
Jarvis' March 99.50 census 99.92 (3)

Every block

FloydNet Census block (one configuration, 3 seeds)
Average accuracy over the 30 tasks, 64 items 96.64 97.04 (seeds 97.14, 97.07, 96.91)
Tasks above / equal / below FloydNet 10 / 7 / 13
Task FloydNet Census block Bilinear tests Cold block (n loops) Compare-first Exact pair numbers
Activity Selector 95.30 99.40 99.94
Articulation Points 100.00 100.00
Bellman-Ford 100.00 99.67 100.00
BFS 100.00 100.00
Binary Search 94.47 98.89 100.00
Bridges 100.00 100.00
Bubble Sort 100.00 100.00 100.00
DAG Shortest Paths 100.00 99.84 100.00
DFS 100.00 72.97 100.00 100.00
Dijkstra 97.67 99.63 100.00
Find Max. Subarray 86.20 87.86
Floyd-Warshall 96.20 99.71 100.00
Graham Scan 99.63 99.80
Heapsort 100.00 100.00 100.00
Insertion Sort 100.00 100.00 100.00
Jarvis' March 99.50 99.92
KMP Matcher 99.43 99.15
LCS Length 95.70 95.66
Matrix Chain Order 91.50 99.80
Minimum 100.00 100.00 100.00
MST-Kruskal 99.63 99.49 100.00
MST-Prim 99.67 98.93 99.98
Naive String Match 100.00 99.07
Optimal BST 82.87 82.71
Quickselect 80.33 99.77 100.00 (5)
Quicksort 100.00 99.98 100.00
SCC 90.07 90.61 87.79
Segments Intersect 99.90 99.33 99.79
Task Scheduling 91.05 89.00 99.86
Topological Sort 100.00 99.99 100.00

Default block per task (the one the task's symmetry calls for; the demo opens on it): compare-first for the sorts, minimum, quickselect, binary search, activity selector, task scheduling and depth-first search (positions in its register); exact pair numbers for Bellman-Ford, Dijkstra, DAG shortest paths, Floyd-Warshall, Kruskal and Prim; the cold block for topological sort and strongly connected components; the census block for the rest.

Test accuracy (%) on the official CLRS-30 test split (64 items; 80 for the string matchers), the mean over seeds (3 unless a count is given in brackets), read by each block's registered rule and scored with the library's scorer. FloydNet: arXiv 2601.19094, Table 9 (best of runs with and without hints).

Files

<block>/<task>/seed<k>/
  model.safetensors           parameters (float32), names like blk/a, in30/e1, one30/min
  config.json                 task, block, model and training configuration, test and val scores, parameter count
  reference_predictions.npz   the model's predictions on the official test and val splits (the registered read and
                              every loop), made when it was trained; the code reproduces them (check_release.py)
index.json                    every model with its scores
best.json                     per task, the block with the best mean test score over its seeds (ties: the block the
                              task's symmetry calls for)

Use

from witness_clrs import load_hub, registered_read, score, OUTPUTS
from witness_clrs.data import load_split
params, cfg = load_hub("mohsenvand/cold-attention-clrs30", "census/bfs/seed0")
d = load_split("data/bfs/test.npz")                        # official split, written by witness_clrs.generate
outs = [(o["name"], o["kind"]) for o in cfg["outputs"]]
pred, loops = registered_read(params, cfg["model"], outs, d)
print(score(pred, d, outs))                                # 1.0

Training data and evaluation

Data from the library itself (dm-clrs 2.0.3): the official test (64 items, 80 for the string matchers) and val (16) splits, and training pools drawn as the library's own training script draws them (sizes 4 to 16). No hints (intermediate steps) are used. Scores use the library's scorer. Each model's training configuration is in its config.json.

Limits

Tasks whose answers depend on sums of numbers that are not comparisons (optimal BST, the maximum subarray, LCS) remain below 100%. Depth-first search reaches 100% only with the cold block read after n loops (the census block, read after 8, scores 73%). On strongly connected components the cold block (86 to 90%) is below the census block (about 91%). The cold-setting blocks cover only the tasks listed.

License

Apache 2.0.

Citation

@misc{mohsenvand2026absolutezero,
  title         = {Attention at Absolute Zero},
  author        = {Mohsenvand, Mostafa},
  year          = {2026},
  eprint        = {2611.11937},
  archivePrefix = {arXiv},
  primaryClass  = {cs.LG},
  url           = {https://arxiv.org/abs/2611.11937}
}
Downloads last month

-

Downloads are not tracked for this model. How to track
Inference Providers NEW
This model isn't deployed by any Inference Provider. ๐Ÿ™‹ Ask for provider support

Paper for mohsenvand/cold-attention-clrs30