Attention at absolute zero: Max Retrieval

Forty trained networks on the task where softmax attention was shown to fail as sets grow (Velickovic et al. 2025, "Softmax is not enough (for sharp size generalisation)", arXiv 2410.01104): pick the class of the item of highest priority from a set. Every network is trained on sets of 5 to 16 items and tested on sets of 16 to 16,384. Four networks, ten seeds each:

Folder Network Weight penalty Reads
cold_count/ cold pair count: each item counts the rivals a learned test admits; the item with the smallest count is read at zero temperature (1/2) * 1e-3 * sum of squares cold
soft_count/ the same input and layers, trained and read warm (sigmoid test, soft count, softmax(-beta c)) (1/2) * 1e-3 * sum of squares warm; cold from the same weights
softmax/ our replication of the published softmax network 1e-3 * sum of squares softmax; cold from the same weights
softmax_half_penalty/ the same, with the other reading of the published penalty (1/2) * 1e-3 * sum of squares softmax; cold from the same weights

The cold pair count is right on at least 99.97% of test sets at every size up to 16,384 (mean of ten seeds; the lowest single seed scores 99.95%, at 16,384 items), and every error it makes falls on a set whose highest priority is duplicated in float32, so that two items of different classes may both be "the" maximum (91 errors in 1,126,400 sets, 91 of them on such sets). At 4,096 items our softmax network is right on 17.6%, the published softmax network on 22.6%, and the best published method (ASEntmax; Vasylenko et al. 2026, arXiv 2506.16640, Table 8) on 62.7%.

The control separates the two halves of the remedy. A network with the same input (priorities only as comparisons) trained and read warm falls to 41.97% at 4,096 items; read at zero temperature, the same weights are right on at least 99.98% at every size (mean of ten seeds; lowest single seed 99.96%). It had learned the right comparisons, and its soft reading lost them.

Paper: Attention at Absolute Zero (arXiv:2611.11937). Code: github.com/NeoVand/cold-attention (the code in max_retrieval/).

The task

A set holds n items. Item i has a priority rho_i, uniform in [0, 1), and one of 10 classes; a query q, uniform in [0, 1), comes with the set and does not change the answer. The answer is the class of the item of highest priority. The softmax network sees each item as (rho_i, onehot class) and the query; the count networks see the priorities only through comparisons b_ij = 1[rho_j > rho_i], computed from the raw priorities, and the classes.

The networks

  • softmax (the network of Velickovic et al. 2025, App. A.3, written in plain jax with Flax's default initialisers): item encoder h_i = gelu(x2(gelu(x1(x_i)))), 128 features; query encoder q2(gelu(q1(q))); one attention head, score e_i = qp(query) . kp(h_i) / sqrt(128), weights softmax(e); value and output projections vp, op; read-out r2(gelu(r1(.))) to 10 class logits. 118,666 parameters.
  • cold_count: a learned test t_ij = t2(gelu(t1(b_ij, onehot k_i, onehot k_j))) (21 -> 32 -> 1) of rival j for item i; item i counts the rivals with t_ij > 0; the weight goes to the items of smallest count, shared evenly over ties; the picked items' class features h(k) = gelu(x2(gelu(x1(onehot k)))) pass through vp, op and the read-out. Trained straight-through: the forward pass is exactly the hard test and pick, the backward pass uses the gradients of sigmoid(t) and softmax(-c). 69,483 parameters.
  • soft_count: cold_count's layers, warm: c_i = sum of sigmoid(t_ij), weights softmax(-beta c) with a learned beta (it ends at 2.586 to 2.591 on the ten seeds). 69,484 parameters.

The cold read of a softmax network puts all weight on the keys of highest score (split evenly over exact float32 ties). The cold read of soft_count uses the test 1[t > 0], the exact count and the smallest count.

Results

Share of test sets answered right (%), mean over ten seeds; 10,240 test sets per size, the same sets for every model. "published" is the softmax curve of Velickovic et al. 2025 (their Table 1, ten seeds).

n cold count soft count, warm read soft count, cold read softmax softmax, cold read softmax, half penalty softmax, half penalty, cold read published
16 100.00 100.00 100.00 98.63 99.12 98.88 99.15 98.6
32 100.00 100.00 100.00 97.09 98.22 97.77 98.36 97.1
64 100.00 100.00 100.00 93.22 96.24 94.91 96.42 94.3
128 100.00 100.00 100.00 85.61 92.66 89.41 93.16 89.7
256 100.00 99.98 100.00 71.02 86.76 78.40 87.24 81.3
512 100.00 93.84 100.00 51.57 76.93 60.53 77.23 70.1
1,024 99.99 76.62 99.99 35.30 64.03 42.44 63.46 53.8
2,048 99.99 58.53 99.99 24.47 49.42 28.91 46.68 35.7
4,096 99.99 41.97 99.99 17.59 36.28 20.13 32.47 22.6
8,192 99.97 28.05 99.98 13.96 25.76 14.97 21.92 15.7
16,384 99.97 18.88 99.98 12.04 18.91 12.67 15.89 12.4

Lowest seed (%):

n cold count soft count, warm read soft count, cold read softmax softmax, cold read softmax, half penalty softmax, half penalty, cold read
16 100.00 100.00 100.00 98.18 98.86 98.11 98.66
32 100.00 100.00 100.00 96.46 97.71 96.14 97.38
64 100.00 100.00 100.00 91.90 95.09 91.35 94.34
128 100.00 100.00 100.00 83.64 90.80 83.81 89.52
256 100.00 99.78 100.00 68.86 83.00 69.41 79.75
512 100.00 90.11 100.00 48.77 71.77 49.59 65.75
1,024 99.99 69.77 99.99 32.36 56.27 32.91 48.06
2,048 99.99 49.12 99.99 21.81 39.88 20.51 30.32
4,096 99.98 33.94 99.98 14.70 27.08 13.64 19.14
8,192 99.96 20.63 99.96 11.75 19.05 10.54 12.53
16,384 99.95 12.90 99.97 10.08 13.00 9.60 10.99

Every seed at 4,096 items (%):

model, read s0 s1 s2 s3 s4 s5 s6 s7 s8 s9
cold count 99.99 99.99 99.98 99.99 99.98 99.99 99.98 99.98 100.00 99.99
soft count, warm read 33.94 43.67 44.79 43.50 41.66 45.24 45.83 41.71 41.59 37.78
soft count, cold read 99.99 99.99 99.99 100.00 99.98 100.00 100.00 99.99 99.98 99.98
softmax 16.68 19.32 19.23 14.72 16.60 16.97 20.74 17.01 19.96 14.70
softmax, cold read 37.71 37.17 39.46 47.14 35.50 27.08 40.96 32.13 34.03 31.59
softmax, half penalty 23.21 24.61 21.32 15.87 21.23 13.64 21.86 22.51 16.65 20.41
softmax, half penalty, cold read 41.43 34.51 25.39 26.02 35.55 19.14 29.58 40.15 40.13 32.84

Our softmax replication does not reproduce the published curve. Under the weight penalty read as 1e-3 * sum of squares it falls up to 18.5 points below the published curve between 64 and 4,096 items; read as (1/2) * 1e-3 * sum of squares, up to 11.4. The cold-read results do not depend on this. The cold read of the softmax weights fails for a reason that can be read from them: within each class the learned score rises with priority, but each class carries its own fixed offset, between 1.9 and 6.0 thousandths of the priority range across these twenty networks. Sets of 16 items never ask a network to rank two items that close; at thousands of items, neighbouring priorities are closer than the offsets and the network ranks by class instead. The offsets alone predict each network's cold accuracy at 4,096 items to within 0.5 points (r = 0.99).

The learned tests. The test sees only (b_ij, class of i, class of j), so its hard form is a 2 x 10 x 10 table (learned_test in each config). On every seed of both count networks it is exactly the comparison: admit a rival when it has the higher priority, whatever either class. The cold pair count's margins on the reject side are thin (largest b = 0 logit -0.0693 to -0.0003 over seeds); soft_count, trained warm, has a wide one (-1.557 to -1.547).

Use

import sys
from huggingface_hub import snapshot_download
root = snapshot_download("mohsenvand/cold-attention-max-retrieval")
sys.path.insert(0, root)
import max_retrieval as M
params, cfg = M.load(f"{root}/cold_count/seed0")
out = M.read(params, cfg["network"], 4096)     # every read of the network on the 10,240 test sets of 4,096 items
print(M.score(out, cfg["network"]))           # {'cold': ...}, share right (%)

python max_retrieval.py demo softmax/seed0 --sizes 16,4096 prints a model's reads beside the recorded ones; M.train(network, seed, l2_scale) retrains a model from scratch. The test sets are redrawn from fixed keys (see the module's docstring); they were drawn with jax 0.8.0 on CPU, and test_sets.npz holds every answer, so a different version of jax can be checked against them. A CPU is enough: on 4 cores one model's eleven sizes take about three minutes.

Files

max_retrieval.py                 the task, the three networks, the reads, the training (jax, optax, safetensors)
<folder>/seed<k>/
  model.safetensors              parameters (float32); suffix w = weight, b = bias; x = item or class encoder,
                                 q = query encoder, qp/kp/vp/op = attention projections, r = read-out, t = the test
  config.json                    network, training, test protocol, scores per read and size; for the count networks
                                 the learned table, its margins and the cold read's errors per size
  reference_predictions.npz      <read>_<n>: the predicted class of every test set, every read, every size
test_sets.npz                    answer_<n>, top_duplicated_<n> for every test set
index.json

Checks

  • Reads. On a CPU with jax 0.8.0, max_retrieval.py redraws the 10,240 test sets of every size and reads all 40 models: every test set is answered right or wrong exactly as recorded when the models were first tested, for every read, size and seed (440 model-size pairs, 7,884,800 answers). reference_predictions.npz holds the predicted classes of this check.
  • Test sets. All 40 models see the same sets: the answers and the duplicated tops agree across models, and with test_sets.npz.
  • Tables. For every count model the table form of the counts equals the pairwise layer on 64 sets of 16 and of 64 items (cold_count: 0 differences; soft_count: largest difference 1.9e-05, float rounding).
  • Training. train retrains seed 0 of each folder from scratch on Modal CPUs. The weights equal the released ones bit for bit for cold_count, soft_count, softmax (soft_count's beta is stored with shape (1,)). For softmax_half_penalty the first retrain drifted (largest difference 0.43 after 100,000 updates); 3 of 3 further retrains, each on its own container, were bit-identical. The likely cause is the processor: XLA's float sums on a CPU can depend on it, and 100,000 updates amplify a last-bit difference.

Limits

  • The comparison bits b_ij of the count networks are computed from the raw priorities, outside the network, as in the paper's cold setting; the softmax network embeds the priority itself.
  • The softmax replication misses the published curve (above), so the comparison with the published 22.6% at 4,096 items is with their number, not with our network.
  • One task. Ten seeds per network, one test draw per size.

License

Apache 2.0. The task follows Velickovic et al. 2025; the code here is our own.

Citation

@misc{mohsenvand2026absolutezero,
  title         = {Attention at Absolute Zero},
  author        = {Mohsenvand, Mostafa},
  year          = {2026},
  eprint        = {2611.11937},
  archivePrefix = {arXiv},
  primaryClass  = {cs.LG},
  url           = {https://arxiv.org/abs/2611.11937}
}
Downloads last month

-

Downloads are not tracked for this model. How to track
Inference Providers NEW
This model isn't deployed by any Inference Provider. ๐Ÿ™‹ Ask for provider support

Paper for mohsenvand/cold-attention-max-retrieval