Four things that did not fix long-context retrieval, and the one measurement that explained why

We are currently preparing a 27B open model with a 1,010,000-token context that runs on a single consumer GPU, as we were frustrated by the lack of such contexts on fairly big models with acceptable quality. Somewhere between "the buffer is a million tokens" and "the model can find things in a million tokens" there is a gap. This post is about a week spent trying to close it from the inference side.

Four ideas failed. Each failure is written down here with its number, because a launch post full of wins teaches nobody anything, and because the fifth measurement, the one that worked, only makes sense against them.

The setup

The model is a hybrid: 48 gated delta layers with a fixed-size recurrent state, and 16 layers of full attention with GQA, 24 query heads over 4 KV heads, head dimension 256. It was trained on 262,144 positions. The declared million is a training-free extension, and we say so on the model card.

The instrument is needles in real prose: Russian classics, with records of the form "Special note number 7: the archive access code is AMBER-6421" planted at known depths, then asked about. The answers appear nowhere else, so guessing is impossible. Crucially, the instrument was checked on short context before being pointed at long context. At 33,334 tokens it scores 92.9%, so when it reports a failure at a million, the failure belongs to the model.

The curve, on two axes

Retrieval does not depend on length alone. It also depends on how many similar records are competing:

Two things stand out. At a quarter of a million, thinning the field helps enormously: four records are all found. At a million it stops helping, and four records score the same as fourteen.

Both axes come out of one model. If the target wins only when its logit beats every one of the other $N-1N-1$ positions, then

$$P_N=\mathrm{\Phi }(\mathrm{\Delta }\mathrm{/}\sigma {)}^{N-1}P\_N\; =\; \backslash Phi(\backslash Delta/\backslash sigma)^\{N-1\}$$

and a single fitted margin-to-noise ratio $\mathrm{\Delta }\mathrm{/}\sigma =4.666\backslash Delta/\backslash sigma\; =\; 4.666$ reproduces all three measured points: 95.0% / 68.2% / 21.4% against measured 92.9% / 64.3% / 21.4%. Thinning stops working because at 250,000 the top of the distractor tail is made of the other needles, while at a million it is made of ordinary prose tokens, of which there are far more.

That exponent is $N-1N-1$. At a million it is a million, which makes the whole thing violently sensitive to the margin. Raising $\mathrm{\Delta }\mathrm{/}\sigma \backslash Delta/\backslash sigma$ by 3.5% would take 21.4% to 51.1%. So we went looking for 3.5%.

Failure 1: position maps

The declared million needs positions the model has never seen. We tried six maps at a million, fourteen needles each, one machine, one corpus:

The instrument's spread is about two needles, so everything between 1 and 3 is one result. The only defensible claim: YaRN finds nothing at a million, and any compression lifts it off zero.

There is one genuinely useful sub-result. YaRN was costing accuracy inside the trained window, where no extrapolation is involved at all. At 250,000 tokens: YaRN 64.3%, YaRN with the logit multiplier removed 71.4%, native positions 78.6%. YaRN multiplies attention logits by 1.288 unconditionally at every length, including lengths the model was trained on. We ship native positions now.

Failure 2: the derived clamp

The slow RoPE pairs are the ones that never complete a turn inside the trained window. At $\theta ={10}^7\backslash theta\; =\; 10^\{7\}$, pair 21 makes 1.063 turns and pair 22 only 0.642. Extending pair 22 to a million pushes it into angles never seen in training, while the fast pairs have wrapped thousands of times and land on familiar angles modulo $2\pi 2\backslash pi$. So the apparently correct move is to keep pairs 0–21 native and freeze pairs 22–31 at the window edge. It is the unique monotone extension that stays inside trained support, and the local phase Fisher information it gives up is $2.4\times {10}^{-10}2.4\; \backslash times\; 10^\{-10\}$.

It scored 0/14. Worse than crude division, no better than YaRN.

In hindsight the Fisher argument measured the wrong thing. It shows the slow pairs barely help distinguish nearby positions. But retrieval at a million is about distinguishing far-apart positions, and there the fast pairs are useless, being congruent modulo $2\pi 2\backslash pi$. The slow pairs are the only carrier of long-range order left, and freezing them makes three quarters of the context positionally identical exactly where it matters most. Integer division is coarse, but it keeps distinguishing.

Failure 3: attention temperature

As context grows past the trained window, the target competes with more distractors and its share of softmax mass falls even with its logit unchanged. Preserving that share needs

$${\tau }^{\ast }(N)=1+\frac{\log ((N-1)\mathrm{/}(W-1))}{\mathrm{\Delta }}\backslash tau^*(N)\; =\; 1\; +\; \backslash frac\{\backslash log((N-1)/(W-1))\}\{\backslash Delta\}$$

which at a million works out to 1.2890. YaRN ships an attention factor of 1.1349, that is, logits multiplied by 1.2880. The agreement to three digits is not a coincidence. YaRN's value is about right for exactly one million, and its mistake is applying it at every length, which is precisely what our 250K ladder measured when removing it gained a needle.

So: a million with the temperature back on. On the clamp map, 0/14 became 0/14. On the best working map, 3/14 became 2/14. Dead.

Failure 4: the KV codec

Our cache is three bits per value. Quantisation noise adds directly to $\sigma \backslash sigma$, and we had a Lloyd-Max value codec already written and kernel-verified. It cuts attention-output error from 0.2982 to 0.1937 and costs two bytes less per vector. If the noise mattered, this was free performance for every user.

First pass, 14 needles at 250,000: three bits 11/14, one extra bit on the values 11/14, fp8 11/14. Identical. But at 78.6% the measurement sits three needles from its ceiling while the run-to-run spread is two, so "we saw nothing" and "we cannot see" were indistinguishable. We said so before running anything else.

Second pass, 40 needles at the same length, which moves accuracy off the ceiling and triples the trials. Three bits 24/40 = 60.0%, fp8 28/40 = 70.0%. Ten points in the predicted direction. But the arms ran on the same needles, so the correct test is paired:

Ten discordant pairs, seven against three, exact McNemar p = 0.34. Not significant.

The corner cells say more than the p-value. Twenty-one needles were found by both arms, and nine were found by neither, even with an essentially noise-free cache. Those nine are not lost to storage noise. They are lost to competition among similar records, the same mechanism that gives 100% on four facts and 64% on fourteen at identical length.

We did not write the codec. The rule, "less than two standard errors means we don't build it", was written down before the run.

What worked: finding the circuit

Attention-based retrieval is done by a small set of heads (Wu et al., 2404.15574): under 5% of them, intrinsic to the weights, and the same set survives context extension. We measured ours with their copy-paste criterion. A head scores a hit when the model emits a token from the needle and that head's argmax attention lands exactly on that token's position inside the needle. Thirty needles, 12,712 tokens of real prose, run at short length on purpose, because retrieval demonstrably works there and a head that fails to score is genuinely not a retrieval head.

18 heads out of 384, or 4.7%, carry half the retrieval mass. That lands on the literature's number without any tuning on our part.

Then the part we did not expect. This model uses GQA, six query heads per KV group. Sorting the top heads by group:

Three quarters of retrieval lives in one KV group out of four. One quarter of the cache carries three quarters of the finding. Three runs on three different corpora gave 77%, 68% and 74% for that group: noise around one fact, not different facts.

By layer it is equally structured. Layers 0, 3, 4, 11, 12, 13, 14 do the work, the first and last attention layers, with the middle barely participating. Eight heads (0.19, 0.21, 3.19, 3.21, 4.20, 11.19, 14.19, 14.21) made the top in all three runs.

What this changes

The tempting move, spending more bits on KV group 3 in the style of DuoAttention, is already refuted from the other side. Noise-free storage everywhere bought nothing, so noise-free storage in one group buys less.

What it does change is the size of the intervention. The retrieval circuit is eighteen heads, one KV group, seven layers. An adapter on those projections is a fraction of a percent of the model: small enough to train on one consumer card, and aimed at the mechanism rather than at the model in general.

That is where we are going next. EndPrompt-style training teaches long positions using only short physical sequences, by putting the passage at the start of the window and a brief anchor at the far end. On LLaMA 8K to 64K it beats full-length fine-tuning on RULER, 76.03 against 69.23. Two rules are already written down: train on the mapped indices the model will actually meet at inference, and not one training example in the shape of a needle. Synthetic context extension underperforms real text in a way that is predictable from which retrieval heads it teaches, and training on your own probe is training on the test.

We will publish that number whether or not it is good. The four above are at least why you can believe that.