Attention at absolute zero: one shared block for all 30 CLRS-30 algorithms

One block of cold attention (146,424 parameters) shared by all 30 algorithms of DeepMind's CLRS-30 benchmark. Each task keeps only its own input encoder and read-outs (6,016 to 46,688 parameters); the block that runs the algorithm is the same for all 30. It was trained on all 30 at once, from inputs of at most 16 items, and is read at 64.

Cold attention, in one paragraph: every ordered pair of items (u, v) carries a vector of 128 numbers. In each loop the block asks two learned yes-or-no questions of every middle item k, one about the pair (u, k) and one about (k, v); an item that passes both is a witness for (u, v). At zero temperature a sum over witnesses keeps two exact quantities, the best witness and how many there are. Half of the channels count their witnesses exactly and pass on only which of two counts is larger, or that they are equal; the other half keep the best witness. This repository holds the block in its general setting (the census block), where the input's numbers enter through the learned embedding; and, in two folders, the census block with a cold block beside it, trained on all 30 tasks the same way.

Paper: Attention at Absolute Zero (arXiv:2611.11937). Code: the companion repository, github.com/NeoVand/cold-attention, has the block, the loader used below, and the per-task models.

Results

Test accuracy (%) on the official CLRS-30 test split (64 items; 80 for the string matchers), read after 8 loops (depth-first search in shared-units-dfs/: after n loops, here 64) (shared-warm-plus-cold/: each input after ceil(log2 n) + 3 loops, its registered read), scored with the library's scorer.

FloydNet (one model per task)
average over the 30 tasks 96.64
seed 0 seed 1 seed 2 mean tasks at or above FloydNet
The shared block (shared/) 96.96 96.71 96.93 96.87 17 of 30
The shared block with deadlines in units of n and depth-first search read at n loops (shared-units-dfs/) 97.34 97.26 97.01 97.20 16 of 30
The shared block with a cold block beside it (shared-warm-plus-cold/) 96.76 96.41 96.48 96.55 16 of 30
The same, final parameters read after 8 loops (shared-warm-plus-cold-final/) 96.58 96.38 96.25 96.40 17 of 30

The two blocks, task by task (mean of three seeds)

Task FloydNet shared/ shared-units-dfs/ change
BFS 100.00 100.00 100.00 +0.00
Bellman-Ford 100.00 99.46 99.51 +0.05
Dijkstra 97.67 99.41 99.50 +0.08
DAG Shortest Paths 100.00 98.94 96.84 -2.10
Floyd-Warshall 96.20 99.17 99.22 +0.06
MST-Prim 99.67 97.04 97.48 +0.44
Topological Sort 100.00 99.74 99.67 -0.07
DFS 100.00 73.03 79.62 +6.59
SCC 90.07 88.05 88.00 -0.05
Articulation Points 100.00 100.00 100.00 +0.00
Bridges 100.00 100.00 100.00 +0.00
MST-Kruskal 99.63 99.29 99.28 -0.01
Insertion Sort 100.00 100.00 100.00 +0.00
Bubble Sort 100.00 100.00 99.98 -0.02
Heapsort 100.00 100.00 100.00 +0.00
Quicksort 100.00 100.00 100.00 +0.00
Minimum 100.00 100.00 99.98 -0.02
Quickselect 80.33 99.87 99.82 -0.05
Binary Search 94.47 99.24 98.80 -0.44
Find Max. Subarray 86.20 87.79 87.55 -0.24
Matrix Chain Order 91.50 99.63 99.64 +0.01
Optimal BST 82.87 82.80 82.44 -0.36
LCS Length 95.70 93.42 92.43 -0.99
Activity Selector 95.30 99.70 99.16 -0.54
Task Scheduling 91.05 91.85 99.07 +7.22
Naive String Match 100.00 99.30 99.69 +0.39
KMP Matcher 99.43 99.41 99.63 +0.21
Segments Intersect 99.90 99.69 99.67 -0.02
Graham Scan 99.63 99.60 99.56 -0.04
Jarvis' March 99.50 99.56 99.52 -0.04

Of the 30 tasks, 9 are higher in shared-units-dfs/ and 15 are lower: DAG Shortest Paths (-2.10), LCS Length (-0.99), Activity Selector (-0.54), Binary Search (-0.44), Optimal BST (-0.36), Find Max. Subarray (-0.24), Topological Sort (-0.07), SCC (-0.05), Quickselect (-0.05), Jarvis' March (-0.04), Graham Scan (-0.04), Segments Intersect (-0.02), Bubble Sort (-0.02), Minimum (-0.02), MST-Kruskal (-0.01). Read with depth-first search also after 8 loops, shared-units-dfs/ averages 97.24 / 96.67 / 96.86 over the 30 tasks (mean 96.92).

What the second recipe changes (shared-units-dfs/)

Two fixes on the first block's recipe, trained the same way otherwise: task scheduling's deadlines divided by the input's n, and depth-first search trained on 8 to 32 loops and read after n loops. Official test, three seeds:

read seed 0 seed 1 seed 2 mean shared/
registered read (29 tasks after 8 loops, depth-first search after n loops) 97.34 97.26 97.01 97.20 96.87
all 30 tasks after 8 loops 97.24 96.67 96.86 96.92 96.87
weight-averaged parameters, registered form (secondary; shared-units-dfs/weight_averaged/) 97.70 97.53 97.49 97.57

On a fresh split drawn the same way, the registered read scores 97.06, against 96.65 for shared/ after 8 loops.

  • Task scheduling: 91.85 to 99.07 (+7.22), every seed between 98.95 and 99.23.
  • Depth-first search after n loops: 79.62 (76.42, 91.02, 71.44), against 73.03 for shared/ after 8 loops (+6.59). The seeds spread widely. Read after 8 loops, the same models score 71.21 (-1.82).

Every loss against shared/ (three-seed means, after 8 loops): DAG shortest paths -2.10, which comes from one seed (97.85, 93.07, 99.61); LCS length -0.99; activity selector -0.54; binary search -0.44; optimal BST -0.36; maximum subarray -0.24; topological sort -0.07; strongly connected components -0.05; quickselect -0.05; Graham scan -0.04; Jarvis' march -0.04; segments intersect -0.02; bubble sort -0.02; minimum -0.02; MST-Kruskal -0.01. Gains on the other tasks: Prim +0.44, naive string matcher +0.39, KMP +0.21, Dijkstra +0.08, Floyd-Warshall +0.06, Bellman-Ford +0.05.

Six forecasts were registered before training; 2 of 6 held (every seed's registered read at or above 96.87; task scheduling at or above 97) and four failed (registered read at or above 97.3, and at or above 97.6; depth-first search after n loops at or above 85; no other task falling by more than 1.0).

The shared block (shared/): the census block, read after 8 loops on every task

Task FloydNet seed 0 seed 1 seed 2 mean
BFS 100.00 100.00 100.00 100.00 100.00
Bellman-Ford 100.00 99.51 99.41 99.46 99.46
Dijkstra 97.67 99.41 99.46 99.37 99.41
DAG Shortest Paths 100.00 99.51 97.75 99.56 98.94
Floyd-Warshall 96.20 99.12 99.26 99.11 99.17
MST-Prim 99.67 96.73 97.31 97.07 97.04
Topological Sort 100.00 99.61 99.85 99.76 99.74
DFS 100.00 76.46 69.29 73.34 73.03
SCC 90.07 85.64 90.92 87.60 88.05
Articulation Points 100.00 100.00 100.00 100.00 100.00
Bridges 100.00 100.00 100.00 100.00 100.00
MST-Kruskal 99.63 99.19 99.27 99.41 99.29
Insertion Sort 100.00 100.00 100.00 100.00 100.00
Bubble Sort 100.00 100.00 100.00 100.00 100.00
Heapsort 100.00 100.00 100.00 100.00 100.00
Quicksort 100.00 100.00 100.00 100.00 100.00
Minimum 100.00 100.00 100.00 100.00 100.00
Quickselect 80.33 99.90 99.76 99.95 99.87
Binary Search 94.47 99.27 99.07 99.37 99.24
Find Max. Subarray 86.20 89.21 86.91 87.26 87.79
Matrix Chain Order 91.50 99.66 99.58 99.65 99.63
Optimal BST 82.87 81.59 83.70 83.13 82.80
LCS Length 95.70 93.03 93.28 93.95 93.42
Activity Selector 95.30 99.46 100.00 99.64 99.70
Task Scheduling 91.05 92.57 90.95 92.03 91.85
Naive String Match 100.00 99.95 98.00 99.95 99.30
KMP Matcher 99.43 99.80 98.49 99.95 99.41
Segments Intersect 99.90 99.75 99.61 99.71 99.69
Graham Scan 99.63 99.76 99.76 99.28 99.60
Jarvis' March 99.50 99.64 99.64 99.40 99.56

The shared block with deadlines in units of n and depth-first search read at n loops (shared-units-dfs/): the census block; task scheduling's deadlines divided by the input's n, as positions are; depth-first search read after n loops

Task FloydNet seed 0 seed 1 seed 2 mean
BFS 100.00 100.00 100.00 100.00 100.00
Bellman-Ford 100.00 99.37 99.56 99.61 99.51
Dijkstra 97.67 99.41 99.56 99.51 99.50
DAG Shortest Paths 100.00 97.85 93.07 99.61 96.84
Floyd-Warshall 96.20 99.24 99.24 99.18 99.22
MST-Prim 99.67 97.27 97.71 97.46 97.48
Topological Sort 100.00 99.49 99.93 99.58 99.67
DFS 100.00 76.42 91.02 71.44 79.62
SCC 90.07 88.82 87.50 87.70 88.00
Articulation Points 100.00 100.00 100.00 100.00 100.00
Bridges 100.00 100.00 100.00 100.00 100.00
MST-Kruskal 99.63 99.22 99.27 99.35 99.28
Insertion Sort 100.00 100.00 100.00 100.00 100.00
Bubble Sort 100.00 99.95 100.00 100.00 99.98
Heapsort 100.00 100.00 100.00 100.00 100.00
Quicksort 100.00 100.00 100.00 100.00 100.00
Minimum 100.00 100.00 100.00 99.95 99.98
Quickselect 80.33 99.85 99.85 99.76 99.82
Binary Search 94.47 98.29 98.63 99.46 98.80
Find Max. Subarray 86.20 89.40 86.52 86.72 87.55
Matrix Chain Order 91.50 99.66 99.60 99.67 99.64
Optimal BST 82.87 85.30 77.50 84.52 82.44
LCS Length 95.70 93.17 92.81 91.31 92.43
Activity Selector 95.30 99.27 99.10 99.10 99.16
Task Scheduling 91.05 99.23 99.03 98.95 99.07
Naive String Match 100.00 99.95 99.51 99.61 99.69
KMP Matcher 99.43 99.90 99.46 99.51 99.63
Segments Intersect 99.90 99.75 99.66 99.61 99.67
Graham Scan 99.63 99.76 99.64 99.28 99.56
Jarvis' March 99.50 99.76 99.52 99.28 99.52

The shared block with a cold block beside it (shared-warm-plus-cold/): the census block with a cold block beside it (the paper's shared block with a warm and a cold half), weight-averaged parameters, each input read after ceil(log2 n) + 3 loops (its registered read)

Task FloydNet seed 0 seed 1 seed 2 mean
BFS 100.00 100.00 100.00 100.00 100.00
Bellman-Ford 100.00 98.68 98.39 98.49 98.52
Dijkstra 97.67 98.68 98.24 98.78 98.57
DAG Shortest Paths 100.00 99.07 99.37 99.37 99.27
Floyd-Warshall 96.20 97.73 97.27 97.09 97.36
MST-Prim 99.67 99.12 99.56 99.27 99.32
Topological Sort 100.00 99.54 99.41 99.29 99.41
DFS 100.00 75.93 77.05 73.54 75.50
SCC 90.07 96.09 95.36 96.88 96.11
Articulation Points 100.00 100.00 100.00 100.00 100.00
Bridges 100.00 100.00 100.00 100.00 100.00
MST-Kruskal 99.63 100.00 100.00 100.00 100.00
Insertion Sort 100.00 100.00 100.00 100.00 100.00
Bubble Sort 100.00 100.00 100.00 100.00 100.00
Heapsort 100.00 100.00 100.00 100.00 100.00
Quicksort 100.00 100.00 100.00 100.00 100.00
Minimum 100.00 100.00 100.00 100.00 100.00
Quickselect 80.33 100.00 100.00 100.00 100.00
Binary Search 94.47 97.41 94.87 95.90 96.06
Find Max. Subarray 86.20 84.47 80.86 80.42 81.92
Matrix Chain Order 91.50 99.52 99.66 99.66 99.61
Optimal BST 82.87 83.58 77.34 87.53 82.82
LCS Length 95.70 91.19 92.09 92.44 91.91
Activity Selector 95.30 98.55 98.19 97.67 98.14
Task Scheduling 91.05 88.43 89.42 86.65 88.17
Naive String Match 100.00 98.68 98.68 96.88 98.08
KMP Matcher 99.43 98.78 98.63 97.17 98.19
Segments Intersect 99.90 99.27 98.98 99.22 99.15
Graham Scan 99.63 98.92 99.76 99.04 99.24
Jarvis' March 99.50 99.04 99.28 99.04 99.12

One census block shared by five tasks (shared-five-tasks/): the census block trained on five tasks only (BFS, Bellman-Ford, quickselect, LCS length, Graham scan), final parameters, read after 8 loops; the paper takes its LCS score into the best-of-our-blocks average

Task FloydNet seed 0
BFS 100.00 100.00
Bellman-Ford 100.00 99.61
Quickselect 80.33 99.76
LCS Length 95.70 95.76
Graham Scan 99.63 99.52

The same, final parameters read after 8 loops (shared-warm-plus-cold-final/): the same three trainings, final parameters, read after 8 loops like the census block

Task FloydNet seed 0 seed 1 seed 2 mean
BFS 100.00 100.00 100.00 100.00 100.00
Bellman-Ford 100.00 98.83 98.00 98.63 98.49
Dijkstra 97.67 98.63 98.88 98.58 98.70
DAG Shortest Paths 100.00 99.07 99.07 99.07 99.07
Floyd-Warshall 96.20 98.06 97.65 97.30 97.67
MST-Prim 99.67 99.12 99.02 99.17 99.10
Topological Sort 100.00 99.15 99.19 99.61 99.32
DFS 100.00 70.21 71.78 69.09 70.36
SCC 90.07 96.09 95.31 94.43 95.28
Articulation Points 100.00 100.00 100.00 100.00 100.00
Bridges 100.00 100.00 100.00 100.00 100.00
MST-Kruskal 99.63 100.00 100.00 100.00 100.00
Insertion Sort 100.00 100.00 100.00 100.00 100.00
Bubble Sort 100.00 100.00 100.00 100.00 100.00
Heapsort 100.00 100.00 100.00 100.00 100.00
Quicksort 100.00 100.00 100.00 100.00 100.00
Minimum 100.00 100.00 100.00 100.00 100.00
Quickselect 80.33 100.00 100.00 100.00 100.00
Binary Search 94.47 96.63 97.85 98.05 97.51
Find Max. Subarray 86.20 84.47 82.32 79.20 82.00
Matrix Chain Order 91.50 99.57 99.67 99.66 99.63
Optimal BST 82.87 83.87 78.81 86.05 82.91
LCS Length 95.70 92.16 91.67 92.02 91.95
Activity Selector 95.30 98.56 99.10 98.56 98.74
Task Scheduling 91.05 87.06 89.21 86.13 87.47
Naive String Match 100.00 98.54 98.14 96.73 97.80
KMP Matcher 99.43 98.88 98.14 97.41 98.14
Segments Intersect 99.90 99.56 99.16 99.17 99.30
Graham Scan 99.63 99.40 99.28 99.16 99.28
Jarvis' March 99.50 99.40 99.28 99.52 99.40

FloydNet (arXiv 2601.19094, Table 9) trains one model per task (20 to 215 million parameters) and reports the best of runs with and without hints.

Training

shared/: 160,000 updates; in every update one batch of 32 inputs from every task, the 30 losses summed, so the shared block and every task's own parameters train together. Loops drawn from 4 to 8 per update; AdamW (learning rate 0.001, 2,000 warm-up steps, cosine decay to 1%, weight decay 0.01); gradients clipped to norm 1. Training inputs: 80,000 per size, drawn by the library itself (dm-clrs 2.0.3) at sizes 4, 7, 11, 13 and 16 (20 for the string matchers), as its own training script draws them. No hints (intermediate steps) are used. Seeds 0, 1 and 2. Released parameters: the final ones.

shared-units-dfs/: 160,000 updates; in every update one batch of 32 inputs from every task, the 30 losses summed, so the shared block and every task's own parameters train together. Loops drawn from 4 to 8 per update; AdamW (learning rate 0.001, 2,000 warm-up steps, cosine decay to 1%, weight decay 0.01); gradients clipped to norm 1. Training inputs: 80,000 per size, drawn by the library itself (dm-clrs 2.0.3) at sizes 4, 7, 11, 13 and 16 (20 for the string matchers), as its own training script draws them. No hints (intermediate steps) are used. Seeds 0, 1 and 2. Released parameters: the final ones. Depth-first search trains at 8, 14, 20, 26 or 32 loops (one value per update, from the update's draw of 4 to 8), in a group of its own, and is read after n loops. Task scheduling's deadlines are divided by the input's n in every split and in training.

shared-warm-plus-cold/: 30,000 updates; in every update one batch of 32 inputs from every task, the 30 losses summed, so the shared block and every task's own parameters train together. Loops: ceil(log2 n) + 3 plus a draw from 0 to 3 per update; AdamW (learning rate 0.001, 2,000 warm-up steps, cosine decay to 1%, weight decay 0.01); gradients clipped to norm 1. Training inputs: 80,000 per size, drawn by the library itself (dm-clrs 2.0.3) at sizes 4, 7, 11, 13 and 16 (20 for the string matchers), as its own training script draws them, except pools built for the study, not the library's, for Bellman-Ford, MST-Prim, SCC, Topological Sort; a share of 25% of their training inputs at 24, 28, 32 nodes. No hints (intermediate steps) are used. Seeds 0, 1 and 2. Released parameters: the mean of the snapshots from half way, one every 1,000 updates. A cold block sits beside the census block. It hands on only bits and keeps one built number per pair (a best bottleneck over the edge weights, max arithmetic), and is rebuilt from the input and its bits every loop; every number leaves its input (weights to the built number's register, positions and item numbers to three register rows that are only compared). In every loop the cold block runs first and its new bits reach the census state through a linear map that starts at zero; it never reads the census state. Each answer adds the census read-out and the cold read-out, so every task has its own second embedding and read-outs.

shared-five-tasks/: 80,000 updates; in every update one batch of 32 inputs from each of its 5 tasks, the losses summed; loops drawn from 4 to 8 per update; AdamW (learning rate 0.001, 2,000 warm-up steps, cosine decay to 1%, weight decay 0.01); 80,000 training inputs per size from the library; seed 0; the final parameters.

shared-warm-plus-cold-final/: 30,000 updates; in every update one batch of 32 inputs from every task, the 30 losses summed, so the shared block and every task's own parameters train together. Loops: ceil(log2 n) + 3 plus a draw from 0 to 3 per update; AdamW (learning rate 0.001, 2,000 warm-up steps, cosine decay to 1%, weight decay 0.01); gradients clipped to norm 1. Training inputs: 80,000 per size, drawn by the library itself (dm-clrs 2.0.3) at sizes 4, 7, 11, 13 and 16 (20 for the string matchers), as its own training script draws them, except pools built for the study, not the library's, for Bellman-Ford, MST-Prim, SCC, Topological Sort; a share of 25% of their training inputs at 24, 28, 32 nodes. No hints (intermediate steps) are used. Seeds 0, 1 and 2. Released parameters: the final ones. The same trainings as shared-warm-plus-cold/; the final parameters instead of the average.

Files

shared/seed<k>/
  model.safetensors           the shared block (blk/*) and every task's own parameters (task/<task>/*), float32
  config.json
  reference_predictions.npz
shared-units-dfs/seed<k>/
  model.safetensors           the shared block (blk/*) and every task's own parameters (task/<task>/*), float32
  config.json
  reference_predictions.npz
shared-warm-plus-cold/seed<k>/
  model.safetensors           the shared block (blk/*) and every task's own parameters (task/<task>/*), the cold block (second/blk/*), every task's second embedding and read-outs (second/task/<task>/*) and the map between the blocks (couple/*), float32
  config.json
  reference_predictions.npz
shared-five-tasks/seed<k>/
  model.safetensors           the shared block (blk/*) and every task's own parameters (task/<task>/*), float32
  config.json
  reference_predictions.npz
shared-warm-plus-cold-final/seed<k>/
  model.safetensors           the shared block (blk/*) and every task's own parameters (task/<task>/*), the cold block (second/blk/*), every task's second embedding and read-outs (second/task/<task>/*) and the map between the blocks (couple/*), float32
  config.json
  reference_predictions.npz
shared-units-dfs/weight_averaged/seed<k>/   the same trainings' weight-averaged parameters (the secondary read), with their config.json and reference_predictions.npz

config.json lists every task with its outputs, input features, training sizes and test and val scores. reference_predictions.npz holds every task's predictions on the official test and val splits after each of loops 1 to 16 (depth-first search in shared-units-dfs/: 1 to n) and at the read (<task>/<split>/per_loop/<output>, <task>/<split>/read/<output>), made when the block was trained, on an NVIDIA B200 with jax 0.8.0. The code reproduces them exactly there (all 450 task-seed models, every loop). On an NVIDIA L4, 4 of the 2,739,072 read entries of shared/ differ (Optimal BST, seed 1) and 139 over loops 1 to 16: answers at near ties, decided by the order in which the GPU sums. shared-five-tasks/ was trained and read on an NVIDIA L4; there the code reproduces its reference exactly (all 5 tasks, every loop).

Use

from witness_clrs import load_shared_hub, registered_read, score
from witness_clrs.data import load_split
params, cfg = load_shared_hub("mohsenvand/cold-attention-clrs30-shared", "shared/seed0", "dijkstra")   # the shared block with Dijkstra's own parameters
d = load_split("data/dijkstra/test.npz")                                # official split, written by witness_clrs.generate
outs = [(o["name"], o["kind"]) for o in cfg["outputs"]]
pred, loops = registered_read(params, cfg["model"], outs, d)          # read after 8 loops
print(score(pred, d, outs))

Limits

Read after 8 loops, the block reaches only patterns about 2^8 steps long, and soft numbers must still be told apart at 64 items. The tasks below 95% (mean of seeds): DFS (73.0), DFS (79.6), Optimal BST (82.4), Optimal BST (82.8), Find Max. Subarray (87.5), Find Max. Subarray (87.8), SCC (88.0), SCC (88.1), Task Scheduling (91.9), LCS Length (92.4), LCS Length (93.4). The per-task models of the companion release reach 100% on depth-first search with loops that grow with the input, and on the order and path tasks with the cold setting (numbers only as comparisons).

License

Apache 2.0.

Citation

@misc{mohsenvand2026absolutezero,
  title         = {Attention at Absolute Zero},
  author        = {Mohsenvand, Mostafa},
  year          = {2026},
  eprint        = {2611.11937},
  archivePrefix = {arXiv},
  primaryClass  = {cs.LG},
  url           = {https://arxiv.org/abs/2611.11937}
}
Downloads last month

-

Downloads are not tracked for this model. How to track
Inference Providers NEW
This model isn't deployed by any Inference Provider. ๐Ÿ™‹ Ask for provider support

Paper for mohsenvand/cold-attention-clrs30-shared