Attention at absolute zero: Max Retrieval
Forty trained networks on the task where softmax attention was shown to fail as sets grow (Velickovic et al. 2025, "Softmax is not enough (for sharp size generalisation)", arXiv 2410.01104): pick the class of the item of highest priority from a set. Every network is trained on sets of 5 to 16 items and tested on sets of 16 to 16,384. Four networks, ten seeds each:
| Folder | Network | Weight penalty | Reads |
|---|---|---|---|
cold_count/ |
cold pair count: each item counts the rivals a learned test admits; the item with the smallest count is read at zero temperature | (1/2) * 1e-3 * sum of squares | cold |
soft_count/ |
the same input and layers, trained and read warm (sigmoid test, soft count, softmax(-beta c)) | (1/2) * 1e-3 * sum of squares | warm; cold from the same weights |
softmax/ |
our replication of the published softmax network | 1e-3 * sum of squares | softmax; cold from the same weights |
softmax_half_penalty/ |
the same, with the other reading of the published penalty | (1/2) * 1e-3 * sum of squares | softmax; cold from the same weights |
The cold pair count is right on at least 99.97% of test sets at every size up to 16,384 (mean of ten seeds; the lowest single seed scores 99.95%, at 16,384 items), and every error it makes falls on a set whose highest priority is duplicated in float32, so that two items of different classes may both be "the" maximum (91 errors in 1,126,400 sets, 91 of them on such sets). At 4,096 items our softmax network is right on 17.6%, the published softmax network on 22.6%, and the best published method (ASEntmax; Vasylenko et al. 2026, arXiv 2506.16640, Table 8) on 62.7%.
The control separates the two halves of the remedy. A network with the same input (priorities only as comparisons) trained and read warm falls to 41.97% at 4,096 items; read at zero temperature, the same weights are right on at least 99.98% at every size (mean of ten seeds; lowest single seed 99.96%). It had learned the right comparisons, and its soft reading lost them.
Paper: Attention at Absolute Zero (arXiv:2611.11937). Code: github.com/NeoVand/cold-attention (the code in max_retrieval/).
The task
A set holds n items. Item i has a priority rho_i, uniform in [0, 1), and one of 10 classes; a query q, uniform in [0, 1), comes with the set and does not change the answer. The answer is the class of the item of highest priority. The softmax network sees each item as (rho_i, onehot class) and the query; the count networks see the priorities only through comparisons b_ij = 1[rho_j > rho_i], computed from the raw priorities, and the classes.
The networks
- softmax (the network of Velickovic et al. 2025, App. A.3, written in plain jax with Flax's default initialisers): item encoder h_i = gelu(x2(gelu(x1(x_i)))), 128 features; query encoder q2(gelu(q1(q))); one attention head, score e_i = qp(query) . kp(h_i) / sqrt(128), weights softmax(e); value and output projections vp, op; read-out r2(gelu(r1(.))) to 10 class logits. 118,666 parameters.
- cold_count: a learned test t_ij = t2(gelu(t1(b_ij, onehot k_i, onehot k_j))) (21 -> 32 -> 1) of rival j for item i; item i counts the rivals with t_ij > 0; the weight goes to the items of smallest count, shared evenly over ties; the picked items' class features h(k) = gelu(x2(gelu(x1(onehot k)))) pass through vp, op and the read-out. Trained straight-through: the forward pass is exactly the hard test and pick, the backward pass uses the gradients of sigmoid(t) and softmax(-c). 69,483 parameters.
- soft_count: cold_count's layers, warm: c_i = sum of sigmoid(t_ij), weights softmax(-beta c) with a learned beta (it ends at 2.586 to 2.591 on the ten seeds). 69,484 parameters.
The cold read of a softmax network puts all weight on the keys of highest score (split evenly over exact float32 ties). The cold read of soft_count uses the test 1[t > 0], the exact count and the smallest count.
Results
Share of test sets answered right (%), mean over ten seeds; 10,240 test sets per size, the same sets for every model. "published" is the softmax curve of Velickovic et al. 2025 (their Table 1, ten seeds).
| n | cold count | soft count, warm read | soft count, cold read | softmax | softmax, cold read | softmax, half penalty | softmax, half penalty, cold read | published |
|---|---|---|---|---|---|---|---|---|
| 16 | 100.00 | 100.00 | 100.00 | 98.63 | 99.12 | 98.88 | 99.15 | 98.6 |
| 32 | 100.00 | 100.00 | 100.00 | 97.09 | 98.22 | 97.77 | 98.36 | 97.1 |
| 64 | 100.00 | 100.00 | 100.00 | 93.22 | 96.24 | 94.91 | 96.42 | 94.3 |
| 128 | 100.00 | 100.00 | 100.00 | 85.61 | 92.66 | 89.41 | 93.16 | 89.7 |
| 256 | 100.00 | 99.98 | 100.00 | 71.02 | 86.76 | 78.40 | 87.24 | 81.3 |
| 512 | 100.00 | 93.84 | 100.00 | 51.57 | 76.93 | 60.53 | 77.23 | 70.1 |
| 1,024 | 99.99 | 76.62 | 99.99 | 35.30 | 64.03 | 42.44 | 63.46 | 53.8 |
| 2,048 | 99.99 | 58.53 | 99.99 | 24.47 | 49.42 | 28.91 | 46.68 | 35.7 |
| 4,096 | 99.99 | 41.97 | 99.99 | 17.59 | 36.28 | 20.13 | 32.47 | 22.6 |
| 8,192 | 99.97 | 28.05 | 99.98 | 13.96 | 25.76 | 14.97 | 21.92 | 15.7 |
| 16,384 | 99.97 | 18.88 | 99.98 | 12.04 | 18.91 | 12.67 | 15.89 | 12.4 |
Lowest seed (%):
| n | cold count | soft count, warm read | soft count, cold read | softmax | softmax, cold read | softmax, half penalty | softmax, half penalty, cold read |
|---|---|---|---|---|---|---|---|
| 16 | 100.00 | 100.00 | 100.00 | 98.18 | 98.86 | 98.11 | 98.66 |
| 32 | 100.00 | 100.00 | 100.00 | 96.46 | 97.71 | 96.14 | 97.38 |
| 64 | 100.00 | 100.00 | 100.00 | 91.90 | 95.09 | 91.35 | 94.34 |
| 128 | 100.00 | 100.00 | 100.00 | 83.64 | 90.80 | 83.81 | 89.52 |
| 256 | 100.00 | 99.78 | 100.00 | 68.86 | 83.00 | 69.41 | 79.75 |
| 512 | 100.00 | 90.11 | 100.00 | 48.77 | 71.77 | 49.59 | 65.75 |
| 1,024 | 99.99 | 69.77 | 99.99 | 32.36 | 56.27 | 32.91 | 48.06 |
| 2,048 | 99.99 | 49.12 | 99.99 | 21.81 | 39.88 | 20.51 | 30.32 |
| 4,096 | 99.98 | 33.94 | 99.98 | 14.70 | 27.08 | 13.64 | 19.14 |
| 8,192 | 99.96 | 20.63 | 99.96 | 11.75 | 19.05 | 10.54 | 12.53 |
| 16,384 | 99.95 | 12.90 | 99.97 | 10.08 | 13.00 | 9.60 | 10.99 |
Every seed at 4,096 items (%):
| model, read | s0 | s1 | s2 | s3 | s4 | s5 | s6 | s7 | s8 | s9 |
|---|---|---|---|---|---|---|---|---|---|---|
| cold count | 99.99 | 99.99 | 99.98 | 99.99 | 99.98 | 99.99 | 99.98 | 99.98 | 100.00 | 99.99 |
| soft count, warm read | 33.94 | 43.67 | 44.79 | 43.50 | 41.66 | 45.24 | 45.83 | 41.71 | 41.59 | 37.78 |
| soft count, cold read | 99.99 | 99.99 | 99.99 | 100.00 | 99.98 | 100.00 | 100.00 | 99.99 | 99.98 | 99.98 |
| softmax | 16.68 | 19.32 | 19.23 | 14.72 | 16.60 | 16.97 | 20.74 | 17.01 | 19.96 | 14.70 |
| softmax, cold read | 37.71 | 37.17 | 39.46 | 47.14 | 35.50 | 27.08 | 40.96 | 32.13 | 34.03 | 31.59 |
| softmax, half penalty | 23.21 | 24.61 | 21.32 | 15.87 | 21.23 | 13.64 | 21.86 | 22.51 | 16.65 | 20.41 |
| softmax, half penalty, cold read | 41.43 | 34.51 | 25.39 | 26.02 | 35.55 | 19.14 | 29.58 | 40.15 | 40.13 | 32.84 |
Our softmax replication does not reproduce the published curve. Under the weight penalty read as 1e-3 * sum of squares it falls up to 18.5 points below the published curve between 64 and 4,096 items; read as (1/2) * 1e-3 * sum of squares, up to 11.4. The cold-read results do not depend on this. The cold read of the softmax weights fails for a reason that can be read from them: within each class the learned score rises with priority, but each class carries its own fixed offset, between 1.9 and 6.0 thousandths of the priority range across these twenty networks. Sets of 16 items never ask a network to rank two items that close; at thousands of items, neighbouring priorities are closer than the offsets and the network ranks by class instead. The offsets alone predict each network's cold accuracy at 4,096 items to within 0.5 points (r = 0.99).
The learned tests. The test sees only (b_ij, class of i, class of j), so its hard form is a 2 x 10 x 10 table
(learned_test in each config). On every seed of both count networks it is exactly the comparison: admit a rival
when it has the higher priority, whatever either class. The cold pair count's margins on the reject side are thin
(largest b = 0 logit -0.0693 to -0.0003 over seeds); soft_count, trained warm, has a wide one (-1.557 to
-1.547).
Use
import sys
from huggingface_hub import snapshot_download
root = snapshot_download("mohsenvand/cold-attention-max-retrieval")
sys.path.insert(0, root)
import max_retrieval as M
params, cfg = M.load(f"{root}/cold_count/seed0")
out = M.read(params, cfg["network"], 4096) # every read of the network on the 10,240 test sets of 4,096 items
print(M.score(out, cfg["network"])) # {'cold': ...}, share right (%)
python max_retrieval.py demo softmax/seed0 --sizes 16,4096 prints a model's reads beside the recorded ones;
M.train(network, seed, l2_scale) retrains a model from scratch. The test sets are redrawn from fixed keys (see the
module's docstring); they were drawn with jax 0.8.0 on CPU, and test_sets.npz holds every answer, so a different
version of jax can be checked against them. A CPU is enough: on 4 cores one model's eleven sizes take about three
minutes.
Files
max_retrieval.py the task, the three networks, the reads, the training (jax, optax, safetensors)
<folder>/seed<k>/
model.safetensors parameters (float32); suffix w = weight, b = bias; x = item or class encoder,
q = query encoder, qp/kp/vp/op = attention projections, r = read-out, t = the test
config.json network, training, test protocol, scores per read and size; for the count networks
the learned table, its margins and the cold read's errors per size
reference_predictions.npz <read>_<n>: the predicted class of every test set, every read, every size
test_sets.npz answer_<n>, top_duplicated_<n> for every test set
index.json
Checks
- Reads. On a CPU with jax 0.8.0,
max_retrieval.pyredraws the 10,240 test sets of every size and reads all 40 models: every test set is answered right or wrong exactly as recorded when the models were first tested, for every read, size and seed (440 model-size pairs, 7,884,800 answers).reference_predictions.npzholds the predicted classes of this check. - Test sets. All 40 models see the same sets: the answers and the duplicated tops agree across models, and with
test_sets.npz. - Tables. For every count model the table form of the counts equals the pairwise layer on 64 sets of 16 and of 64 items (cold_count: 0 differences; soft_count: largest difference 1.9e-05, float rounding).
- Training.
trainretrains seed 0 of each folder from scratch on Modal CPUs. The weights equal the released ones bit for bit forcold_count,soft_count,softmax(soft_count's beta is stored with shape (1,)). Forsoftmax_half_penaltythe first retrain drifted (largest difference 0.43 after 100,000 updates); 3 of 3 further retrains, each on its own container, were bit-identical. The likely cause is the processor: XLA's float sums on a CPU can depend on it, and 100,000 updates amplify a last-bit difference.
Limits
- The comparison bits b_ij of the count networks are computed from the raw priorities, outside the network, as in the paper's cold setting; the softmax network embeds the priority itself.
- The softmax replication misses the published curve (above), so the comparison with the published 22.6% at 4,096 items is with their number, not with our network.
- One task. Ten seeds per network, one test draw per size.
License
Apache 2.0. The task follows Velickovic et al. 2025; the code here is our own.
Citation
@misc{mohsenvand2026absolutezero,
title = {Attention at Absolute Zero},
author = {Mohsenvand, Mostafa},
year = {2026},
eprint = {2611.11937},
archivePrefix = {arXiv},
primaryClass = {cs.LG},
url = {https://arxiv.org/abs/2611.11937}
}