Attention at absolute zero: 2-SAT and two-colouring

Blocks of cold attention (135,929 parameters) that decide whether a 2-SAT instance or a two-colouring instance is unsatisfiable, trained on instances of at most 16 variables (2-SAT) or 32 (two-colouring) and read at up to 500. Three seeds per problem.

The block is the cold setting of the paper's layer: every ordered pair of items (u, v) carries 128 numbers; each loop asks two learned yes-or-no questions of every middle item k, and an item that passes both is a witness for (u, v). The block hands on only bits (which of two witness counts is larger, or that they are equal; whether some witness passes both tests), and the state is rebuilt from the input and these bits every loop. Item positions are withheld: every position the block receives is 0, so its answer cannot depend on the order of the items.

Paper: Attention at Absolute Zero (arXiv:2611.11937). Code: github.com/NeoVand/cold-attention (the generators in csp/).

The problems

  • 2-SAT on n variables: 2n literal items and one global item. A(a, b) = 1 when literal a implies literal b (each clause (p or q) gives not-p implies q and not-q implies p); adj pairs each literal with its negation. Unsatisfiable exactly when some literal and its negation imply each other.
  • Two-colouring on n variables: n variable items and one global item. A relates two variables that must take the same colour, adj two that must differ.

The answer (1: unsatisfiable) is read at the global item's own pair (output unsat, stored at item s; -1 elsewhere). Half of the inputs at each size are random instances, balanced to half unsatisfiable; the other half come in twin pairs: an unsatisfiable instance with a planted contradiction, and a satisfiable twin that differs in one constraint and has the same counts of every pair input.

Results

Share of inputs answered right (%), weight-averaged parameters, each input read after ceil(log2 n) + 7 loops for n items.

Problem split (variables, items) seed 0 seed 1 seed 2 mean
2-SAT val (24, 49; 256 inputs) 100.00 100.00 100.00 100.00
2-SAT test (64, 129; 256 inputs) 100.00 100.00 100.00 100.00
2-SAT n256 (256, 513; 128 inputs) 100.00 100.00 100.00 100.00
2-SAT n500 (500, 1001; 128 inputs) 100.00 99.22 100.00 99.74
Two-colouring val (48, 49; 256 inputs) 100.00 100.00 100.00 100.00
Two-colouring test (64, 65; 256 inputs) 100.00 100.00 100.00 100.00
Two-colouring n256 (256, 257; 128 inputs) 100.00 100.00 100.00 100.00
Two-colouring n500 (500, 501; 128 inputs) 100.00 100.00 100.00 100.00

The first registered 2-SAT run (cold_first_registered/two_sat/seed0)

The same block and read, from the first registered training of 2-SAT: learning rate 0.001 and loops ceil(log2 n) + 3 + a draw from 0 to 3 per batch, for inputs of n items. The supplement and Fig. 6 report it beside the three seeds above, which were trained with the smaller learning rate and loop budget that the paper adopted after it. Right (%): val 99.61, test 77.34, n256 69.53, n500 55.47. It fits the training sizes and does not carry to 64 variables and beyond.

The first two-colouring run (cold_first_registered/two_colouring/seed0): collapsed

A registered failure, published because the paper reports it: the same block and read, trained at the higher learning rate (0.001, loops ceil(log2 n) + 3 + a draw from 0 to 3 per batch, for inputs of n items). It collapsed to one answer, "unsatisfiable", for every input, so it is right on half of every split (val 50.00, test 50.00, n256 50.00, n500 50.00).

Parity with positions given (cold_positions_given/parity/seed0): did not fit

A registered failure, published because the paper reports it. Parity equations with a hidden twist: on a graph whose vertices each have three edges, two variables per edge must differ, and each vertex has eight equations over its three edges' variables; the twisted instance (one vertex's right-hand sides flipped) is unsolvable, the plain one solvable, and every count of the inputs is the same for both. Items: the variables, the edge equations, the vertex equations and one global item (12.5 per vertex, plus one); A links each variable to the equations it occurs in, and each equation carries its right-hand side on its own pair of adj. Positions are given here, unlike the models above. Trained on graphs of 4 to 8 vertices with batch 8, it answers "solvable" for every input and does not fit its own training instances. Right (%): val 50.00, test 50.00, n8 50.00, n12 50.00, cube 50.00, petersen 50.00, fit 45.83 (fit: 96 of its training instances, 54% of them unsolvable; cube and petersen: the cube and the Petersen graph).

Training

2-SAT: 2 to 16 variables; 30,000 updates of batch 32; loops per batch ceil(log2 n) + 1 + a draw from 0 to 1 per batch, for inputs of n items; AdamW (learning rate 0.0003, 2,000 warm-up steps, cosine decay to 1%, weight decay 0.01); the read uses the average of 16 snapshots from update 15,000 on, one every 1,000.

Two-colouring: 3 to 32 variables; 30,000 updates of batch 32; loops per batch ceil(log2 n) + 3 + a draw from 0 to 3 per batch, for inputs of n items; AdamW (learning rate 0.0003, 2,000 warm-up steps, cosine decay to 1%, weight decay 0.01); the read uses the average of 16 snapshots from update 15,000 on, one every 1,000.

Files

cold/<problem>/seed<k>/                (also cold_first_registered/{two_sat,two_colouring}/seed0 and
                                       cold_positions_given/parity/seed0: registered runs the paper reports)
  model.safetensors           weight-averaged parameters (float32)
  config.json                 model, training and read configuration, scores per split, loops used
  reference_predictions.npz   predictions on every split after each loop, at the read, and the loops used
data/<problem>/{val,test,n256,n500}.npz   (parity: val, test, n8, n12, cube, petersen, fit) the evaluation splits (inputs A, adj, pos, s, n; answer unsat; kind: 1 = twin,
                                          pair: twin pair id; depth: shallowest contradiction; nv: variables)
index.json

Use

from huggingface_hub import snapshot_download
from witness_clrs import load, registered_read, score
from witness_clrs.data import load_split
root = snapshot_download("mohsenvand/cold-attention-csp")
params, cfg = load(f"{root}/cold/two_sat/seed0")
d = load_split(f"{root}/data/two_sat/test.npz")
outs = [(o["name"], o["kind"]) for o in cfg["outputs"]]
pred, loops = registered_read(params, cfg["model"], outs, d, batch=4)   # each input after ceil(log2 n) + 7 loops
print(score(pred, d, outs))                                            # share of inputs whose answer is right

The pair state grows with the square of the items: the 500-variable splits were read one input at a time on an 80 GB GPU (an NVIDIA H100); a 24 GB GPU runs out of memory there. On an H100 with jax 0.8.0 the code reproduces reference_predictions.npz exactly: all 9 models, every split, after every loop. The companion repository (github.com/NeoVand/cold-attention) has the block and the reader, and its csp/ folder has the generators: they rebuild every split the models were trained and scored on, with the same array digests. The evaluation splits are also included here.

Limits

Every split is answered right on every seed, except 2-SAT n500, seed 1: 99.22. These are two of the bounded-width constraint classes: their contradictions are paths, which the block grows by joining witnesses. The block computes no more than counting logic over three variables at a time can: an unsatisfiable parity instance (three-variable equations) on the cube graph cannot be told from a satisfiable one by any such rule, and the block with random weights gives the two bit-identical answers.

License

Apache 2.0.

Citation

@misc{mohsenvand2026absolutezero,
  title         = {Attention at Absolute Zero},
  author        = {Mohsenvand, Mostafa},
  year          = {2026},
  eprint        = {2611.11937},
  archivePrefix = {arXiv},
  primaryClass  = {cs.LG},
  url           = {https://arxiv.org/abs/2611.11937}
}
Downloads last month

-

Downloads are not tracked for this model. How to track
Inference Providers NEW
This model isn't deployed by any Inference Provider. ๐Ÿ™‹ Ask for provider support

Paper for mohsenvand/cold-attention-csp