Alhena: Building a Rotary That Withstands Selection

AbstractPowered · AbstractPhil, with Claude Opus 5.5 and Claude Fable 5.1 (Anthropic) · October 2026 · the rotary line, 22-29 October 2026 · follows One Address, Three Systems

---

This installment covers eight days and one goal: the strongest rotary position code we could find for the part of a modular model that decides who answers. The house builds modular models [1]: a trunk trained once and frozen, and many small members trained alone and switched in together. At every position something has to decide which member writes, and the house's rule for that decision is strict: no member is ever chosen over another. Each member writes on its own evidence, and a member with no evidence writes exactly nothing. That rule leaves the selection needing a source of evidence that is structural rather than competitive, an address. The systemic articulation behind this line was that a rotary position code [2] could carry it, if some of its angles came from a fixed structural address (which compartment, which slot, which record) instead of from a token's index, so that a member can own a place and a query can ask for one.

A rotary that works on one example is not the goal. Selection puts the address under pressure, and this line is a record of those pressures taken one at a time:

• addresses that must be told apart exactly, at any length;
• members that must stay silent without being outvoted;
• addresses deep enough for a million places;
• a request the model has to supply itself, from nothing but a key;
• keys that share most of their characters with a neighbour's;
• millions of competing keys;
• text that must stay healthy around all this structure;
• more records in a row;
• an address the model writes for itself, digit by digit.

Each pressure has a failure on record and, in most cases, a piece that ended it. The pieces assemble into a candidate, mapped in the last chapters.

The short version runs in four steps:

1. On a toy bed, addressing works. A query turned to the address it asks for finds it exactly with any code, and members trained alone merge behind an address-reading gate with nothing lost and nothing leaked.
2. On a real model, the request is the hard part. Built into a 9.2-million-parameter byte-level model named Alhena, the structure answers exactly whenever it is handed the true address. The model's own request failed in three distinct ways, until the whole key was bound into one attention head's rotation.
3. The binding scales. It found the whole key on every seed, and with sixteen bound pairs it keeps ten million competing keys apart.
4. The written address was the last failure. The address the model writes for itself stalled at a digit on most seeds, under every schedule, length, code and loss weight tried, until the key was handed to every digit position. Then it formed on twelve of twelve models, at 256 and at 512 addresses, within 2,000 to 6,000 steps.

---

Reading guide

This is the narrative. Every table, formula and run behind the numbers is in the technical companion and the repositories listed at the end. A few words recur:

• A rotary pair is a two-dimensional slice of a query and a key that the rotary code turns by an angle. Each attention head here has sixteen pairs. In the standard form a pair turns by the token's position times the pair's frequency, so a query and a key score by their distance [2].

• The address kernel is how much two identical vectors still agree after being turned to two different addresses. Its central peak is the resolution, its side lobes are what a wrong address leaks, and the number of addresses it holds apart is its capacity.

• The anchored rotary splits a head's pairs by job: relative pairs that turn by position, content pairs that never turn, and anchor pairs that turn by a structural address. A key turns by its own address; a query turns by the address it asks for.

• A grid row is the synthetic task the byte model trains on beside ordinary text, a byte-level form of multi-query associative recall [3]. It is a row of 4,096 bytes holding three kinds of content:

  • records (DATA address CUE key SEP value END);
  • an index, where the model writes the address of each asked key's record;
  • questions (CUE key SEP value END), whose answers sit in the records.

  Keys are a word and a number (otter 417), and the SEP after a key is its end-marker.

• The request is the address a question asks for. Given the true request, everything downstream is exact; most of the line is about where the model's own request comes from.

• Members (or arms) are small adapters trained alone, each owning one address. The gate weights member $ii$ by $w_i=\sinh u_i\mathrm{/}(1+{\sum }_j(\cosh u_j-1))w\_i\; =\; \backslash sinh\; u\_i\; \backslash ,/\backslash ,\; \backslash bigl(1\; +\; \backslash sum\_j\; (\backslash cosh\; u\_j\; -\; 1)\backslash bigr)$, where $u_{i}u\_i$ is high when the request matches the member's address and exactly zero when it does not. Three tests judge a merge:

  • protected: the owner keeps its answer;
  • quiet: the other members write nothing on questions they do not own;
  • exact: both hold to the precision of the bare model.

• The lookup head is the one attention head (block 4, head 0) trained to look at the record an index entry names. Its look is the record holding more than half of its attention, or none.

• Text quality is in bits per byte (lower is better) on held-out FineWeb-Edu documents [4].

• Seeds are independent trainings of the same recipe. A claim that holds on five or six seeds of six is the line's standard.

Contents

1. The goal: a rotation that is also an address
2. Resolution and silence: one night on the toy bed
3. Depth: toward a million addresses
4. The request: where the address comes from
5. Partial keys: three ways a lookup fails
6. The key in the rotation
7. The crowd: how many keys the binding keeps apart
8. Healthy text: plain documents
9. More records: past sixty-four
10. The written address: an index that stalls
11. The key at every digit
12. The strongest candidate
13. What the line establishes, and the pillar next

After the chapters: Related work · Open questions · Artifacts and companions · References · Citation

The findings that matter most

In order of importance, not of date:

• The hard part of rotary addressing is not the address; it is the request. On the byte model, handed the true address of each asked record, the gate with 48 real adapter arms passes all three tests in every cell. The leak is at most 8 × 10⁻⁷ of a full write, and the text is unchanged to the last digit. Through the model's own written index, right on 81-90% of entries at the time, the same merge failed the quiet and exact tests in every cell (the owners stayed protected): a wrong entry is a full-weight write from the wrong member.

• Binding the whole key into the rotation ended the lookup's failures. At every record's end-marker, one head's pairs turn by a sum of fixed random phases, one per character of the key and its distance back. The model found the whole key on twelve of twelve trainings, where the same recipes without the binding found it on one seed of six. It then found the whole key on eighteen of eighteen at 48 records a row and keys of twelve bytes, with no wrong lookup among 15,360 questions on any model.

• Handing the key to every digit ends the written address's stalls. With the key's phases applied at each digit position of an index entry, the lookup head finds the true record from every digit (all of its attention, on every model), and no other head has to copy the address forward. The model's own index formed on:

  • six of six seeds at four digits of four, within 2,000-6,000 steps, where the stored models formed it on two;
  • six of six at three digits of eight, within 2,000-4,000, where every earlier model had stopped at the second digit.

  A learned key code at the same positions formed it on one seed of six.

• The binding's capacity follows a random-phase law, and sixteen bound pairs keep ten million keys apart. Sixteen bound pairs, with the records' key length held at one value, make no wrong lookup at a million keys and at most 0.43% at ten million. They lose half their lookups only between 31 and 71 million. The four-digit address at 96 records keeps that reach (a median of 57 million against 55 to 58 million for the two-digit models), and a loss weight on the index costs none; seed by seed, between models with and without the weight, the reach followed the bound pairs' trained margin. At a hundred million keys the true record still has the single highest score on all but one entry of 27,648. The lookups fail by the summed weight of many weak competitors, not by one rival.

• A gate that reads a structural address is exact where comparison is not. Members trained alone and merged through the sinh gate stay within .008 of their solo reads and leak nothing measurable; merged ungated they read .05-.26. On an address read off attention, the sinh form was 10 to 50 times quieter than softmax.

• Address capacity multiplies when the gate reads a product of level kernels. Two levels of nine anchors address 64 compartments exactly in 18 dimensions, where one flat band needs 65. A normalizer that charges resting members nothing removes the ceiling on how many members one gate holds.

• The written index stalls on a comfortable plateau, not for want of steps. At 48 records a row, a model that knows three of four digits is right by chance on three entries in four. The stalled seeds stayed there under a higher learning-rate floor (one of four formed) and under twice the length (none formed). A loss weight on the last digit formed three of the four, but moved where the others stopped, to the second digit.

• Plain documents in the training mix end a text failure and improve the text everywhere (by .015 to .047 bits per byte on six seeds of six), with the lookups and the capacity unchanged.

• Four-digit addresses on the same pairs take a row to 96 records at the same reach, and the anchored rotary costs about what the standard rotary does once compiled.

---

1. The goal: a rotation that is also an address

A modular model in the house's sense is a trunk trained once and then frozen, plus members trained alone, each for one behaviour, each small: adapters in the sense of [5, 6]. At inference the members are switched in together, and a gate decides how much each one writes at each position. That gate is the selection, and the rule the house keeps for it is strict: no member is ever chosen over another. A gate that takes the largest score, or normalizes scores against each other with a softmax, as mixture-of-experts routers do [7, 8, 9], makes every member's write depend on the others' scores. In this record, such comparative routing has failed whenever it was tested against a structural alternative.

The house's gate reads each member's own evidence and turns it into a signed amplitude [1]:

$$w_i=\frac{\sinh u_i}{\sum _j\cosh u_j},w\_i\; =\; \backslash frac\{\backslash sinh\; u\_i\}\{\backslash sum\_j\; \backslash cosh\; u\_j\},$$

where $u_{i}u\_i$ is member $ii$'s evidence and the sum runs over every member. A member with no evidence (\(u_i = 0\)) writes exactly $\sinh 0=0\backslash sinh\; 0\; =\; 0$. It abstains; it is not outvoted.

That gate needs a source of evidence that is structural, not learned to compete. The systemic articulation behind this line was a fixed-position rotary for the house's mixtures of members:

• each member owns an anchoring offset and a capacity in a rotary phase space;
• outside its band it stays quiet, and its own band stays protected;
• all of it is gated by the position the sequence asks for.

Rotary codes were already in the house's models, turning each query and key by an angle proportional to position [2]. The proposal was to let some of those angles come from a token's place in a fixed structure.

There is a reason to expect the two pieces to fit. The rotary phase and the gate are the same exponential on two curves:

• A rotation is $e^{i\theta }e^\{i\backslash theta\}$. Its real and imaginary parts are the circle's $\cos \theta \backslash cos\backslash theta$ and $\sin \theta \backslash sin\backslash theta$.
• The gate's amplitude is built from $e^{u}e^\{u\}$. Its even and odd parts are the hyperbola's $\cosh u\backslash cosh\; u$ and $\sinh u\backslash sinh\; u$.

A softmax keeps the even part in both numerator and denominator, so even a completely unmatched member keeps a floor of weight. The sinh gate keeps the even part in the denominator only, so an orthogonal address gives a member exactly zero. If the address a member reads is a rotary code with exactly orthogonal anchors, the abstention is exact by construction.

The object everything below measures is the address kernel. Turn two identical unit vectors by the rotary to positions $mm$ and $nn$; their score is

$$K(m-n)=\frac{1}{P}\sum _{j=1}^P\cos ({\omega }_j(m-n)),K(m-n)\; =\; \backslash frac\{1\}\{P\}\backslash sum\_\{j=1\}^\{P\}\; \backslash cos\backslash bigl(\backslash omega\_j\backslash ,(m-n)\backslash bigr),$$

averaged over the $PP$ pairs with frequencies ${\omega }_j\backslash omega\_j$. At distance zero it is 1. Its central peak is the address's resolution, its side lobes are what a wrong address leaks, and the number of addresses that can be packed with every other score near zero is its capacity. The same sum appears in the literature twice: its sign over a context sets the smallest rotary base that context length needs [10], and for random frequencies it is the Fourier transform of their distribution [11]. Every form tested is a choice of frequencies and of what drives the angle: position, slot, compartment, or, later in the line, the characters of a key. And every chapter below is one more pressure the selection puts on that kernel.

2. Resolution and silence: one night on the toy bed

The first night ran on a single RTX 4090. The model was a small causal transformer [12]: four layers, width 128, four heads, contexts of 256, trained from scratch on synthetic sequence tasks. Every result is two seeds, and the pass marks were written down before each phase's first run. About 400 trainings and merge cells ran between 01:16 and 07:27.

The catalogue. Twenty-four rotary forms were trained on three lookups: a token at a fixed relative distance, a token at an absolute slot, and a token in a named compartment.

• NTK-aware scaled rotary (its slow pairs stretched by four, its fast pairs nearly unchanged [13]) had the best reach at the standard rotary's cost.
• The plain rotary [2], a partial rotary that leaves some pairs unturned [14], and the linearly scaled rotary of position interpolation [15] all worked.
• Several forms in the house's experimental library failed outright and were retired. Among them were a hierarchical Cantor rotary that gave every token the same rotation, and a four-band embedding added at the input instead of applied to queries and keys.

A query turned to the address it asks for. Each key is turned by its own structural address, and the query by the address it asks for, so the two agree exactly where they should.

• The compartment lookup was answered 1.000 with every code tried, where plain rotary alone reached .92 and .94.
• An absolute-slot lookup was answered 1.000 at four times the trained length, where the dynamic forms collapsed to about .28.

The anchored rotary. The working form splits a head's pairs by job:

• Content pairs never turn, so they compare what tokens are.
• Slot pairs turn by the position inside a compartment.
• Anchor pairs turn by the compartment index at angles ${\omega }_k=2\pi k\mathrm{/}(2P_a+1)\backslash omega\_k\; =\; 2\backslash pi\; k\; /\; (2P\_a\; +\; 1)$, for $P_{a}P\_a$ anchor pairs.

On integer compartments those angles make the $2P_a+12P\_a\; +\; 1$ anchors a regular simplex: every distinct pair scores $-1\mathrm{/}(2P_a)-1/(2P\_a)$. One extra constant dimension lifts them to exactly orthonormal codes. A sequence mixing "the token three back" with "the token in compartment 9, slot 2" was answered 1.000 on both kinds. With compartments shuffled, so that content gave no clue to identity, the anchored model still answered 1.000; without anchors it fell to .17.

Members, silent by construction. Six members, each trained alone on a frozen core to answer its own compartment, were switched in together behind the gate reading the anchor code of the requested address.

• Merged, they stayed within .008 of their solo reads.
• On compartments no member owns, the output matched the bare core to the last measured digit (a KL divergence of .0000).
• Switched in without a gate, they wrecked each other (.05 to .26).
• On an attention-read version of the address, the sinh gate was 10 to 50 times quieter than softmax.
• An anchor band holds as many members as it has orthogonal anchors, $2P_a+12P\_a\; +\; 1$ for $P_{a}P\_a$ pairs. Squeezing the band leaked in proportion to the overlap.
• Members attached in the middle of the stack learned their tasks only on the anchored core: 1.000 against .04-.07. The anchor band lets a member find its place before the model has computed what the place contains.

Two more facts belong here. Timed later the same day in bf16 at 4,096 to 65,536 tokens, the standard rotary costs 1.5-5% of a transformer block. The anchored form costs 1.24-1.27 times that at 4,096 tokens and the same once compiled, so addressing by rotation is not a cost problem. And the night did not establish the most important thing: an address the model realizes by itself. Every exact result used the layout's own structural request.

3. Depth: toward a million addresses

A context of a million positions needs a million addresses, and one flat band of $2P_a+12P\_a\; +\; 1$ orthogonal anchors would need half a million pairs. The natural alternative is digits: address a compartment by its digits, with one anchor band per digit level, the way a number is written. Tree positional encodings write a node's position as its path of branch choices in the same way [16], and multi-module phase codes, from grid cells to residue-number codes, reach a capacity exponential in the number of modules [17, 18]. The selection then has to read several levels at once without letting them compete.

One gate over a product of level kernels. Sixty-four compartments were addressed by two digits of eight, each level a band of nine anchors. The gate reads the product of the two level kernels:

$$u=U\cdot k_1(r_1,i_1)\cdot k_2(r_2,i_2),u\; =\; U\; \backslash cdot\; k\_1(r\_1,\; i\_1)\; \backslash cdot\; k\_2(r\_2,\; i\_2),$$

where $r_{l}r\_l$ is the requested digit at level $ll$, $i_{l}i\_l$ the member's own digit there, $k_{l}k\_l$ that level's anchor kernel (1 when the two digits agree) and $UU$ the gate's strength. Reaching many places through the product of small per-level codebooks is the idea behind product-key memories [19]; they add the levels' scores, and for a gate the sum leaked (below), so here the levels' kernels multiply.

• It equals the flat 65-anchor gate to 3.6 × 10⁻⁷, from 18 dimensions instead of 65: merged 1.000, KL .0000.
• Three levels addressed 512 compartments exactly from 27 numbers.
• Four levels of 33 anchors would address 1,185,921 in 132.

Two other combinations failed instructively:

• One gate per level silenced its members, because each level dilutes the owner once by every member sharing that digit (the owner's weight fell to .209).
• Summing the levels leaked .45 of an owned write into questions the member did not own.

How many members one gate holds. On an exact gate, merging costs only dilution. With $mm$ members merged under the plain cosh normalizer, the owner's weight is $\sinh U\mathrm{/}(\cosh U+m-1)\backslash sinh\; U\; /\; (\backslash cosh\; U\; +\; m\; -\; 1)$, and a member stops answering once its weight falls below about .49. That caps the gate at 78 members at $U=5U\; =\; 5$ and about 11,460 at $U=10U\; =\; 10$. The remedy is a normalizer that charges a resting member nothing:

$$w_i=\frac{\sinh u_i}{1+\sum _j(\cosh u_j-1)}\mathrm{.}w\_i\; =\; \backslash frac\{\backslash sinh\; u\_i\}\{1\; +\; \backslash sum\_j\; \backslash left(\backslash cosh\; u\_j\; -\; 1\backslash right)\}.$$

A member with $u=0u\; =\; 0$ contributes $\cosh 0-1=0\backslash cosh\; 0\; -\; 1\; =\; 0$, so the owner's weight is $\sinh U\mathrm{/}\cosh U=\tanh U\backslash sinh\; U\; /\; \backslash cosh\; U\; =\; \backslash tanh\; U$ at any number of members. All nine merge cells (three member counts by three gate strengths) were exact. This rest discount is the gate's form for everything after.

4. The request: where the address comes from

Everything so far was exact given the address the question asks for. The next screens took the layout's help away. A row held 32 records, each an address, a value and a key; questions named only a key; and the model had to supply the address itself. On the toy bed, two sources of the request worked:

• An index the model writes. Once trained, the model's own index was right on every entry, and read at its fixed place it gave every merge cell the same result as the layout's request. A deliberately corrupted index, one swapped pair a row, failed by plain arithmetic, because a wrong entry is a full-weight write from the wrong member. Generative retrieval trains models to write the identifier of what they retrieve in the same way [20, 21].
• An address head trained together with the core. It was right on 158,772 of 158,772 read questions, with a leaked write of at most 7 × 10⁻⁸ of an owned write.

Reading the request off attention leaked, and an address head attached to a frozen core gave confident wrong requests. How a question is posed also turned out to decide how fast a lookup forms: posing questions the way index entries are posed formed the lookup in 2,000-8,000 steps instead of up to 96,000.

Alhena. The structure then moved into the house's nano byte model, a small member of the byte-level AlephLLM line [1] (on byte-level language models generally, see [22]):

• The model: Castor's body, 9.2 million parameters, six blocks of width 256, eight attention heads of sixteen rotary pairs each, a byte-trigram input (compare the hashed byte n-gram embeddings of [23]), the splat memory hub and a bank of seven experts.
• The training: plain Adam [24] at a learning rate of 10⁻³, 24,000 steps of eight rows of 4,096 bytes, about eighteen minutes on Colab's G4 card (an RTX PRO 6000).
• The mix: half of every batch became grid rows; the rest kept the model's own diet of text with planted passages to copy back, plus plain FineWeb-Edu text [4].
• The rotary: six relative, two content and eight anchor pairs per head, the anchors driven by each record's two-digit address.

Castor and Pollux are the house's twin models; the third of the triplets was named Alhena, the third-brightest star of Gemini.

Through the true address Alhena answered the grid questions .999 and .998 of the time from step 2,000 on, with its text matching the standard-rotary model's. Then the members: forty-eight real arms (the house adapter library's relay patchwork, 16 by 8, born inert: its output starts at zero, as a LoRA update does [6]) were attached after the last block, and sixteen after block 3, each trained alone on its own records. Alone, each arm left the bare model within .004. Merged with the true address as the request, all three tests held in every cell, with the leak at most 8 × 10⁻⁷ of an owned write and the text unchanged to the last digit. The mid-stack arms learned on Alhena (.866 and .842) and never on the standard-rotary model (.000), as the toy bed had said.

Merged with the model's own written index as the request, every cell failed on every model: the owners stayed protected, but the others leaked. The index was right on 81-90% of entries, and its wrong entries are full-weight writes from the wrong member, which the gate passes faithfully. Everything downstream of the request was exact on a real model with real arms. The request was the piece that broke.

5. Partial keys: three ways a lookup fails

The written index was the obvious request, and it was not good enough, and it got worse when the model read back what it had written itself (.897 fell to .842), because one wrong entry misleads the next. Eleven screens over two days followed. Each changed one thing, each was graded against marks written before it ran, and each lasted two to five hours on the G4. Read in order, they separate three faults.

A find term and a wider band.

• The find term: training the address head at every index entry as well as at every question. It had two states: it locked in at .995-.9998 of entries or stalled at .85-.92, decided by about step 4,000. At 48,000 steps one seed was exact, and 48 real arms merged through its own find kept every test, the mid-stack arms too. It was the first exact self-made request with real arms, but on one seed of two and at a text cost.
• The wider band: a read showed the lookup living in one head, through its content pairs and its slowest relative pairs. Trained rotary models are known to do their semantic matching on the slowest pairs, and two published variants leave those pairs unturned for that reason [14, 25]. Giving the content four pairs (four relative, four content, eight anchors) raised the odds of a lock without making them certain. This split became the base for everything after.

The first fault: a suffix match. A grid key is a word and a three-digit number. The plateaued models matched only the key's last two digits. They were right on every entry whose two-digit ending is unique in its row, and about $1\mathrm{/}(k+1)1/(k\; +\; 1)$ of the time where $kk$ other records share that ending. On these rows that comes to about .90, exactly the stalls. It is the cheapest feature that suffices on these rows: a shortcut in the sense of [26].

The second fault: a copy. To break the suffix match, half the grid rows were made key-hard: keys whose parts form a factorial (two words by two hundreds by two tens by three units), so every record has a neighbour differing in exactly one part and only the whole key finds the record. In the language of contrastive learning these neighbours are hard negatives [27]. No model locked, and the read named a different failure. The lookup head put .94-.98 of its attention on the right record and wrote the first address digit right on every entry, then copied the second digit, one byte later, from a neighbouring record: a copy of the kind induction heads make [28]. Every wrong entry was that copy (242 of 246 on one model, 340 of 340 on another). Adding each record's address to every one of its tokens cut the copies to 1-9 entries of 768 on five seeds of six, but did not move the lock count.

The lookup head, and its look as the request. Block 4's head 0 was then given a loss on its own attention, a supervised-attention term [29, 30]: at each index entry, its attention mass on the record the entry names. The head learned to look (.93-1.0 of its mass on the true record on every model), but the index mostly did not read it. A supervised head can learn its look while the model around it gains little from it, as in [29]. A plain read found something better than the index. The record the head attends to was the true record on .98 of entries or more on five models of eight, where the written index managed two. So the request was read off the head itself, as a pointer network reads its answer off attention [31, 32], in a second pass: the address of the record holding more than half of its attention, and no request at all when no record does. On the model whose head matched whole keys, that look was right on every one of 12,288 questions while its written index was right on .3271. Forty-eight arms merged through it kept every test. The gate's tolerance turned out to be about one wrong request in ten thousand.

The third fault: a hedge. Training the head on key-hard rows gave the clearest picture. One seed of six reached the whole key, after a sudden exit between 16,000 and 18,000 steps; long plateaus ended by abrupt exits are a known pattern of gradient training [28, 33, 34]. The other five spread their attention evenly over the four records that share the asked key's ending, the term sat near $(\ln 4)\mathrm{/}2(\backslash ln\; 4)/2$ (the even split), and half the hard questions got no request. Doubling the run did not lift the hedge on four seeds of five.

Counting the overlap. On a key-hard row, the records sharing four, three, two, one or none of the asked key's parts number 1, 5, 9, 7 and 2. The input is byte trigrams, so the end-marker after a key carries only the key's last two bytes, and four records tie there. Summing the whole key's bytes into the end-marker's input through learned tables, much as hashed n-gram embeddings widen a byte model's input [23], reached the whole key on five seeds of six. The direction was right; the form was not yet.

6. The key in the rotation

The systemic articulation at this point was a rotary oscillatory offset conjunction, with the phases summed per pair: put the key into the rotation itself rather than into the input. At every end-marker that closes a key, the lookup head's eight shared pairs (its relative and content pairs) turn by

$${\phi }_j(t)=\sum _{k}P[j,k,x_{t-k}](\mathrm{m}\mathrm{o}\mathrm{d}2\pi ),\backslash varphi\_j(t)\; =\; \backslash sum\_\{k\}\; P\backslash bigl[j,\backslash ,\; k,\backslash ,\; x\_\{t-k\}\backslash bigr]\; \backslash pmod\{2\backslash pi\},$$

summed over the key's characters $x_{t-k}x\_\{t-k\}$ back to its cue. $PP$ holds one fixed random phase for every pair $jj$, distance back $kk$ and byte value, and queries and keys turn alike:

• Two end-markers closing the same key turn by the same angles on every pair, and score as if nothing had turned.
• Two keys that differ in a single character turn every pair by an unrelated angle, and their expected score is the average of random phases, near zero.

In the language of vector-symbolic architectures this is a binding, the Fourier form of holographic reduced representations [35, 36]: the key's characters multiplied into one phase vector, where the summed input code had been a bundle.

It worked on every model. Twelve models, six seeds with key-hard rows and six without, found the whole key: the look on the key-hard rows right on all 3,072 questions on every model, and the look taken at the question itself exact on all twelve. The same recipes without the rotation had found the whole key on one seed of six each. The bound pairs scored the true record 48.4 to 57.2 logits above zero, while every other record scored about zero. Forty-eight real arms merged through the look kept all three tests at every gate setting on the three models tried, with the text untouched.

The larger row. The binding was then pushed to the largest row the window and the two-digit code allow. That meant 48 records a row in 64 slots of 64 bytes, so a question competes against up to 92 keyed end-markers, and keys of up to twelve bytes.

• All eighteen models found the whole key, with no wrong lookup on any model.
• The margins grew: 58.6-66.2 logits at 48 records, and 57.3-69.7 with the long keys.
• The arms after the last block were exact through the look on all four models tried.
• The mid-stack arms met their mark exactly where a per-letter linear probe of each block's states [37], fixed before they trained, said the answer's letters stood by block 3 (three predictions of three). A member in the middle of the stack can only use what has already arrived where it sits.

7. The crowd: how many keys the binding keeps apart

A row holds at most a few hundred keyed end-markers. A selection meant to work across long contexts needs its binding to hold far more apart than one row asks of it. Softmax attention is in principle an associative memory whose capacity grows exponentially with its dimension [38]; the question is how much of that this binding realizes. It can be measured without training anything. Take each index entry's true score at the lookup head and set it against $NN$ competing keys: first the other rows' keyed end-markers, then a bank of stored end-marker contents turned by fresh keys' phases, up to a million and later a hundred million. A lookup is wrong when the competing attention mass reaches the true record's.

There is a closed form to read the measurements against. With $JJ$ bound pairs and a typical margin of $ss$ per pair, a wrong key's score is a sum of $JJ$ terms $s\cos {\theta }_{j}s\backslash cos\backslash theta\_j$ with independent random phases ${\theta }_j\backslash theta\_j$, and the expected exponential of each term is a modified Bessel function [39]:

$$\mathbb{E}\left[e^{s\cos \theta }\right]=\frac{1}{2\pi }{\int }_0^{2\pi }{e}^{s\cos \theta }d\theta =I_0(s)\mathrm{.}\backslash mathbb\{E\}\backslash bigl[e^\{\backslash ,s\backslash cos\backslash theta\}\backslash bigr]\; =\; \backslash frac\{1\}\{2\backslash pi\}\backslash int\_0^\{2\backslash pi\}\; e^\{\backslash ,s\backslash cos\backslash theta\}\backslash ,d\backslash theta\; =\; I\_0(s).$$

Crosstalk of this kind is what limits retrieval from any vector-symbolic memory [40]. The true record scores about $JsJs$, so the mean share of attention leaked to $NN$ random competitors is about

$$\frac{N{I}_0(s{)}^J}{e^{Js}+N{I}_0(s{)}^J}\mathrm{.}\backslash frac\{N\backslash ,\; I\_0(s)^J\}\{e^\{Js\}\; +\; N\backslash ,\; I\_0(s)^J\}.$$

Eight bound pairs. On thirty stored models the measured mean leak followed that moment between ten thousand and a million keys. At a million, 0.2-7.2% of lookups went wrong. Those wrong lookups had a single cause: another row's record whose bound pairs ran 2-5% longer than the true record's. Equalizing the lengths removed most of them.

Sixteen pairs with the records' length held. Binding all sixteen of the head's pairs left 0 to 9 wrong lookups in 4,608 at a million keys, against 33 to 252 for eight pairs. Holding the key's sixteen pairs at one length (2.18), a narrow form of query-key normalization [41, 42], at the records' end-markers only, leaving the index entries short, gave no wrong lookup and no outright wrong winner at a million keys on any of six seeds.

The far read. Those models were then read to a hundred million keys on three Colab hosts, streamed so that memory stays flat:

• They made their first wrong lookups between two and twenty million keys, and at most 0.43% were wrong at ten million.
• They lost half their lookups between 31 and 71 million.
• Even at a hundred million, the true record had the single highest score on every held entry but one of 27,648.

The lookups do not fail because a rival wins. They fail because many weak competitors, the strongest about 2% of the total, together outweigh the true record. The competing mass grows linearly in the number of keys past ten million, and the mean leak converges toward the random-phase moment. Below that scale a sum of many random exponentials is ruled by its rare strong terms, as in the random-energy model [43], and only with enough keys does it settle at its mean.

The reach under the later recipes. Two later stages were read the same way:

• The four-digit address (chapter 9). At 96 records a row it lost half its lookups at the same scale as the two-digit address at 48: medians of 5.72 against 5.75 × 10⁷. A model's own million-key read predicted its limit to within .84-.96 of a linear law.
• A loss weight on the index (chapter 10). It cost no reach either: medians of 42 against 38 million.

Between the weighted and unweighted models, seed by seed, the reach followed the trained margin of the lookup head's bound pairs: further where the margin was larger, on six seeds of six. It did not follow whether the model's written index had formed. Behind 99.3-99.8% of the wrong lookups, the strongest competitor was another row's record at the true key's own length; an index entry never led.

Ten million competing keys is four orders of magnitude past any row a 4,096-byte window holds. The binding is not what limits a row; the address code and the window are.

8. Healthy text: plain documents

A selection that is exact on records is of little use if the model around it reads ordinary text badly. On a few seeds it did, in one specific way. On one seed, plain held-out text opening with the document byte was read as if it were record content: the model predicted a record's end byte at about 98% of positions from 256 to 1,024, costing 1.21 bits per byte overall (another seed: +0.80). The cause was in the training mix; how training sequences are composed, and where their documents begin, is known to shape what a model learns [44]. Every row opened by the document byte was a record row or a grid row, and the held-out documents open with that byte too.

The fix was to make one in four of the non-grid rows a plain document that starts at a document byte. On six new seeds the failure was gone:

• the first 64 bytes of a held-out document read 1.898-1.923 bits, against 2.53-2.88 on the comparison models;
• held-out text improved on all six, by .015 to .047 bits per byte against both comparison families, in the middle of documents as well as at their start.

The key, the request and the capacity did not move. The plain rows are part of the recipe from here on.

9. More records: past sixty-four

Two digits of eight give 64 addresses, and 48 records was the largest row the window and that code allow. The next step wrote the address as four digits of four, 256 addresses, on the same eight anchor pairs: two pairs per level at angles $2\pi k\mathrm{/}52\backslash pi\; k/5$. A level's five symbols (four digits and the null, no-address anchor) then sit at equal distances, as the two-digit code's nine symbols do at four pairs per level; over a level's pairs ${\sum }_k\cos ({\omega }_{k}d)\backslash sum\_k\; \backslash cos(\backslash omega\_k\; d)$ is 2 at $d=0d\; =\; 0$ and $-\frac{1}{2}-\backslash tfrac\{1\}\{2\}$ at every other difference $dd$. The relative and content pairs stay as they were.

The alternatives were worse. Three digits of eight need twelve anchor pairs, taking the relative and content bands down to two pairs each. Two digits of sixteen would need the whole head. The code has a price too: one wrong digit costs a level's whole margin, and a level now has half the pairs.

Twelve models were trained: six at 96 records a row (128 slots of 32 bytes), and six at 48 with only the code changed.

• All twelve found the whole key.
• From the asked key's index entry, the lookup head pointed at the right record on every question of every model.
• Handed the record's address, the models answered .998 to 1.000 of questions.
• No model made a wrong lookup among a million competing keys.
• At 48 records the new code stayed inside the text mark (−.005 to +.013 bits per byte).

At 96 records two marks were missed narrowly, each on one seed of six:

• The look taken at the question itself. There was no request on 0.4% of questions, where part of the head's attention went to the address digits the model had written in the index.
• The held-out text, .0006 past its mark. A 96-record row holds about half the ordinary text of a 48-record row.

Trained twice as long (48,000 steps), both misses closed. The look at the question was exact on six of six, and also on six fresh seeds at 24,000 steps. The text gap to a comparison model trained as long narrowed on every seed. And every written index was whole at 48,000 steps, which opened the next chapter.

10. The written address: an index that stalls

The look is exact, but it is read in a second pass over the row. The model also writes its own index in the row, and an address the model writes is the form an inference system wants: no second pass, and the request part of what the model already produces, as in generative retrieval [20, 21]. On the stored models that index was whole on some seeds and stuck below whole on others. On most seeds it did not improve gradually: it sat low and then jumped, at a step that differed from seed to seed.

Time, or the anneal? The first explanation was time: trained to 48,000 steps, every stuck index of the last chapter went whole. But three of those four jumps came by step 24,000, the point where the short runs had ended. Up to that step a long run and its short pair see the same rows in the same order and differ only in the learning rate. On the long cosine schedule [45] the rate there was still above half its peak; on the short one it had annealed to a tenth. The stalls looked like an anneal arriving before the jump. Tested at the stored length:

• A higher floor under the cosine (annealing to half the peak, not a tenth) formed three of the four stuck indexes, at 10,000 to 14,000 steps, at no text cost.
• A long flat phase at the peak followed by a short cool-down, the warmup-stable-decay schedule [46, 47], formed one, in its fall. It read the text .033 to .040 bits per byte better on every seed.

The rate's level at the jump did not decide it. Neither a held peak nor a fall to a tenth supplied the jump; on these four, a slow descent did. That reading did not carry over to the next recipe.

A comfortable plateau. At 48 records a row with the four-digit code, nothing worked that way.

• The higher floor formed one of four stuck seeds, and kept a seed that had formed under the stored schedule from forming.
• Twice the length formed none; two seeds left the plateau and stopped at .95 and .96.

A read digit by digit explained the plateau. At 48 records, more than half of the index entries have no other record under their first three digits. A model that has learned three digits and guesses the fourth among the records sharing them is therefore right on .7574 of entries by chance. The stalled models sat at that level (.78 to .82 right). On the clearest of them the first three digits were right on every entry, and every wrong entry was another record of the row, most often the next or previous slot. Nothing in the loss pushes hard off that plateau: the fourth digit's bytes are 0.3% of it.

Weighting the last digit. Weighting the last digit's loss was the direct remedy, in the spirit of losses that weight the tokens that matter most [48, 49].

• At a weight of .1 (thirty-five times the digit's plain share), three of the four stuck seeds formed.
• At a weight of 1 the last digit became exact, and the second and third broke.
• The light weight also did something unexpected. The two seeds that had formed without it no longer did, and every seed that did not form stopped at its second digit, with its last digit mostly right.

The weight changed which seeds form and where an unformed index stops; it did not end the stop.

Three digits of eight. The 512-address code under that weight made the pattern plain. It needs twelve anchor pairs, so the relative and content bands drop to two pairs each.

• No seed formed its index. Every one stopped at the second digit (.38 to .49 right), with the first digit exact and the weighted third mostly right.
• The text paid .018 to .039 bits per byte, at every window of the row.

Meanwhile the binding's reach did not care whether an index had formed (chapter 7).

What the stalls had in common. It appeared when attention was read at every digit position of an index entry, on nine models (the stored, the weighted and the three-digit ones):

• On every model, the lookup head put all of its attention on the true record at the entry's end-marker.
• At the digit positions after it, nothing in the query carried the key. A digit position wrote its digit right only where another head, later in the stack, copied the address forward from the end-marker [28] (.75 to .999 of that head's attention where the copy had formed).
• Where the copy had not formed, the digit position looked mostly at the records whose address shares the digits written so far (.60 to .90 of the attention, read at the second and third digits), and picked among them.

The index stopped at the first digit whose copy had not formed. Read this way, the schedules, lengths and weights tried were ways of coaxing those copies one digit at a time.

11. The key at every digit

The remedy follows from the read: give each digit position what the end-marker has, the key. Two forms were trained side by side on the four-digit, 48-record recipe, without the last-digit weight, six seeds each:

• The binding at every digit. The lookup head's sixteen pairs at an entry's digit positions turn by the same key phases as at its end-marker: ${\theta }_j(t)={\phi }_j(s)\backslash theta\_j(t)\; =\; \backslash varphi\_j(s)$ for each digit position $tt$ of the entry closed at $ss$. The head's attention is supervised there too. It adds no parameter.
• The learned key code at every digit. The whole key's bytes, summed through learned tables born zero, are added at every end-marker and at each digit position.

The binding. It formed the index on six seeds of six (.9999-1.0000), within 2,000 to 6,000 steps. The stored models had formed it on two seeds of six (at 4,000 and 18,000 steps), and the last-digit weight on three (at 12,000 to 18,000). Read at every digit position, the lookup head put all of its attention (1.000) on the true record at the end-marker and at each of the three digit positions after it, on every model. No other head copied the address forward: the largest such copy was .002 to .244, where formed copies elsewhere had held .75 to .999. Each digit is read from the record directly.

The learned code. It formed the index on one seed of six. Its lookup head reads nothing of the record at the digit positions. Instead the code lets a later head find the entry's own end-marker by its key and copy from it, and that copy formed at the second position on every seed (.68 to .996 of its attention), at the third on five, and at the last on one. The other five stopped at the last digit, right there on .95 to .98 of entries (one also at its third), with most of their wrong entries another record of the row.

The costs ran the other way:

• The learned code cost the text nothing (−.003 to +.001 bits per byte).
• The binding cost nothing on three seeds, and +.010, +.018 and +.031 on the other three. The largest is past the line's mark of +.021, and the cost is spread over every window of the row rather than sitting at the index.
• Every look stayed exact.
• With the binding, the answers through the model's own written index matched the answers through the true address to the fourth decimal on every model.

The hardest case. On three digits of eight with the last-digit weight, one change from the recipe where every seed had stopped at the second digit, the binding at every digit formed the index on six of six seeds: four at the first traced step (2,000) and two by 4,000. The lookup head held all of its attention on the true record at both digit positions after the end-marker on every model. On the two earlier three-digit models read this way it had put none there. At the second digit, one model's lookup head looked at the other records sharing the first digit (.81 of its attention). The other's looked at other records' later bytes (.86), while a different head put .79 on the records sharing the first digit. Against those models the text moved −.013 to +.007 bits per byte, within the mark on every seed. The second digit's stop had been the missing look, at both codes.

12. The strongest candidate

Every pressure in this line has a piece that held wherever it was tested: on five or six seeds of six on the byte model, and on both seeds of the toy bed for the gate's forms. Assembled, they make a candidate we call ALHENA-E for now (E for the key at every digit, the entry form; its proper name is still to be chosen):

• The address. Every head on the anchored rotary: four relative, four content and eight anchor pairs, the address written as four digits of four (256 addresses). Keys turn by their own address and queries by the address they ask for.
• The binding. The lookup head's sixteen pairs turn by the whole key at every end-marker, with the records' key pairs held at one length (2.18). The key's phases are applied at every digit position of an index entry, and the head's attention is supervised at the end-marker and at each digit.
• The paths. The address head, and the value path (each record's address added to its tokens, from tables born zero).
• The training rows. Half of every batch grid rows, half of those key-hard, 96 records a row; one non-grid row in four a plain document.
• The schedule and length. The tested cosine, or the flat schedule for its text gain; 24,000 steps for a screen and 48,000 for a model of record.
• The members and the gate. Forty-eight real arms after the last block, and mid-stack arms where a depth probe puts their letters, behind the sinh gate with the rest discount over the product of level kernels.
• The request. The model's own written index, now whole on every seed, checked by the lookup head's look. Where no record holds more than half of its attention there is no request, and the gate abstains.

The candidate's last lines are where it goes past the record:

• The key at every digit was trained at 48 records, not 96.
• Its reach past a million keys is unmeasured. Its digit positions now carry the key's phases, and whether they join the crowd is the first thing to check.
• Its text cost on three seeds of six has two known levers, each measured without it: the flat schedule (.033-.040 better) and the longer run (.069-.076 better than the same model at 24,000 steps).
• No member has yet been merged through the written index of a model with the key at every digit. Merges through earlier models' own indexes kept their owners but were not exact.
• The index has been graded with each digit predicted from the row as it stands, not read back as the model writes it.
• Codes deeper than 512 addresses, and anything larger than nine million parameters, are untested.

A confirmation screen would settle the first five. It would train twelve models (the candidate at 96 records, with and without the flat schedule), merge members through the written index, and read the reach. That is about two hours on three Colab G4 cards, plus a few hours of reads with no training.

13. What the line establishes, and the pillar next

What the line establishes, in plain terms:

• A rotary phase can carry a structural address exactly, with a capacity that can be measured and predicted.
• A gate that reads that address in the sinh form lets independently trained members merge with nothing lost and nothing leaked.
• A byte-level model can supply the address itself, exactly, if the key that names a record is bound into the rotation that finds it, at the end-marker and at every digit it writes.
• With sixteen bound pairs, that request holds ten million competing keys apart, and the model now writes it whole on every seed tried.

What the line does not establish yet is listed in the candidate's last lines above, together with the middle of the stack placed by rule rather than by probe.

The pillar next. An address of several digits is a path through a tree, each digit a branch, as in a trie [50] or the code tree of a hierarchical softmax [51], and the key at every digit is the first form in this line that walks that tree right at every level, on every seed. The next installment carries the binding from record keys to every unit of ordinary text: a trie-feature sequence bound into the rotation, a structural pillar a subsection of the model can articulate from, rather than a faster lookup. Its first screens are on record and its design is under review; it is a separate story.

Related work

Rotary codes and their extensions. The rotary position embedding [2] turns query and key pairs by angles proportional to position, so attention scores depend on relative distance. Its extensions to longer contexts rescale the positions or the frequencies [13, 15]. Studies of which pairs carry position and which carry content [14, 25] are the nearest reading of a head's pairs by job, and the bound that a model's frequency base sets on its context [10] is stated through the same kernel this line measures. In all of them the angle is a function of the token's index; none gives it a structural source.

Structural positions. Several encodings give a token a coordinate other than its index: two-level positions inside and across segments [52], hierarchical positions taken from the structure of code [53], one shared position for structurally related tokens [54], and tree positions written as paths of branch choices [16]. The anchored rotary belongs to this family, with two differences: a query turns by the address it asks for rather than its own, and each address is owned by a detachable member that must stay silent everywhere else.

Routing and merging members. Mixture-of-experts layers route by a learned comparison of scores [7, 8, 9]; hash layers route by a fixed function of the token and match or beat learned routing [55]. Independently trained adapters [5, 6] are combined by learned gates over their outputs [56] or by arithmetic on their weights [57, 58], and the interference grows with the number of members combined. The sinh gate [1] takes a third route: it reads an address, and a member the address does not name writes exactly zero.

Memory, recall and pointers. Attention as an associative memory [38], product-key memories [19] and multi-query associative recall [3] frame the grid rows' lookups. Induction heads [28] describe the copies seen in chapters 5 and 10. Supervised attention [29, 30] and pointer networks [31, 32] are the precedents for the lookup head and its look. Generative retrieval [20, 21] trains a model to write the identifier of what it retrieves, often as a hierarchical code decoded one symbol at a time; the written index of chapters 10 and 11 is that act inside a single row.

Binding and phase codes. Holographic reduced representations bind symbols by circular convolution, which in their Fourier form is the addition of phases [35, 36]. Retrieval from such memories is limited by crosstalk [40], and multi-module phase codes reach a capacity exponential in the number of modules [17, 18]. The key's binding in the rotation is that operation inside one attention head, and chapter 7's random-phase law is its crosstalk, measured to a hundred million keys.

What we did not find. To our knowledge, no prior work gives rotary frequency bands at fixed structural addresses to separately trained adapters as their address, or measures how silent and how protected merged adapters stay as a function of the address code.

Open questions

• The candidate at 96 records, and its reach. Does the key at every digit form the index in the larger row, and do its digit positions join the crowd past a million keys?
• The binding's text cost. On three seeds of six the key at every digit costs .010-.031 bits per byte, spread over the row, with no known cause. Does the flat schedule or the longer run pay it back?
• Members through the written index. The original request design failed only because the index was wrong on 10-19% of entries. With the index whole, members merged through it should keep every test; that is unmeasured.
• The index read back. Every grade here predicts each digit from the row as it stands. A model reading back its own written digits is the inference form, and the gap between the two is the one scheduled sampling was designed to close [59].
• The middle of the stack. A member placed after block 3 works exactly when its answer's letters are already present there; a rule for placement is unwritten.
• The crowd past ten million keys. More bound pairs and held lengths push the limit out; whether a normalization of the competing mass, such as scaling attention logits with the logarithm of the number of keys [60, 61], can push it further is open.
• Deeper codes. Three digits of eight form with the key at every digit but cost the text; four or more levels, and three digits without the last-digit weight, are untested.
• Scale. Everything on the byte model is 9.2 million parameters, a 4,096-byte window, and synthetic grid rows beside FineWeb-Edu text.

Artifacts and companions

• Technical companion: TECHNICAL_rotary.md, with the formulas written out and every stage's table with its numbers.
• AbstractPhil/geolip-rotary-experiments holds:
  • the toy night's weight files and result records;
  • the capacity and far reads' records;
  • every byte-model stage from the plain documents through the key at every digit (bed/): run records, weights, look records and claims, each stage with its section on the repository card.
• AbstractPhil/alephllm-nano-beatrix-training: the byte-model stages from the scale test through sixteen bound pairs with the records' length held.
• AbstractEyes/geolip-nano-beatrix-trainer: the trainer, versions 0.1.70 to 0.1.103, with its tests and an Open-in-Colab notebook for every stage.
• AbstractEyes/amoe-lora: the adapter library whose relay-patchwork arms were the real members (version 0.2.12).

This work is a collaboration between AbstractPhil (AbstractEyes) and the Claude research sessions that operate the program's record; the direction of every system named here is his, and the measurements are the program's.

References

Every entry was checked against its primary record (the arXiv listing, the publisher's DOI record or the proceedings page) on 29 October 2026. Papers first released on arXiv are cited by their arXiv identifier, with the published version where its record confirms one.

1. AbstractPhil. One Address, Three Systems: The Formulas of AlephLM, AlephLLM and AMOE, and the Measurements That Set Them. AbstractPowered, Hugging Face, 11 October 2026. huggingface.co/blog/AbstractPowered/one-address-three-systems-the-formulas-of-alephlm.
2. Su, J., Lu, Y., Pan, S., et al. RoFormer: Enhanced Transformer with Rotary Position Embedding. 2021. arXiv:2104.09864. Published in Neurocomputing 568:127063, 2024.
3. Arora, S., Eyuboglu, S., Timalsina, A., et al. Zoology: Measuring and Improving Recall in Efficient Language Models. 2023. arXiv:2312.04927.
4. Penedo, G., Kydlíček, H., Ben Allal, L., et al. The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale. 2024. arXiv:2406.17557.
5. Houlsby, N., Giurgiu, A., Jastrzebski, S., et al. Parameter-Efficient Transfer Learning for NLP. 2019. arXiv:1902.00751.
6. Hu, E. J., Shen, Y., Wallis, P., et al. LoRA: Low-Rank Adaptation of Large Language Models. 2021. arXiv:2106.09685.
7. Jacobs, R. A., Jordan, M. I., Nowlan, S. J., Hinton, G. E. Adaptive Mixtures of Local Experts. Neural Computation 3(1):79–87, 1991. doi:10.1162/neco.1991.3.1.79.
8. Shazeer, N., Mirhoseini, A., Maziarz, K., et al. Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer. 2017. arXiv:1701.06538.
9. Fedus, W., Zoph, B., Shazeer, N. Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity. 2021. arXiv:2101.03961.
10. Men, X., Xu, M., Wang, B., et al. Base of RoPE Bounds Context Length. 2024. arXiv:2405.14591.
11. Frady, E. P., Kleyko, D., Kymn, C. J., et al. Computing on Functions Using Randomized Vector Representations. 2021. arXiv:2109.03429.
12. Vaswani, A., Shazeer, N., Parmar, N., et al. Attention Is All You Need. 2017. arXiv:1706.03762.
13. Peng, B., Quesnelle, J., Fan, H., Shippole, E. YaRN: Efficient Context Window Extension of Large Language Models. 2023. arXiv:2309.00071.
14. Barbero, F., Vitvitskyi, A., Perivolaropoulos, C., et al. Round and Round We Go! What makes Rotary Positional Encodings useful? 2024. arXiv:2410.06205.
15. Chen, S., Wong, S., Chen, L., Tian, Y. Extending Context Window of Large Language Models via Positional Interpolation. 2023. arXiv:2306.15595.
16. Shiv, V., Quirk, C. Novel positional encodings to enable tree-based transformers. Advances in Neural Information Processing Systems 32 (NeurIPS 2019), 2019. proceedings.neurips.cc.
17. Fiete, I. R., Burak, Y., Brookings, T. What Grid Cells Convey about Rat Location. Journal of Neuroscience 28(27):6858–6871, 2008. doi:10.1523/JNEUROSCI.5684-07.2008.
18. Kymn, C. J., Kleyko, D., Frady, E. P., et al. Computing with Residue Numbers in High-Dimensional Representation. 2023. arXiv:2311.04872.
19. Lample, G., Sablayrolles, A., Ranzato, M., et al. Large Memory Layers with Product Keys. 2019. arXiv:1907.05242.
20. Tay, Y., Tran, V. Q., Dehghani, M., et al. Transformer Memory as a Differentiable Search Index. 2022. arXiv:2202.06991.
21. Rajput, S., Mehta, N., Singh, A., et al. Recommender Systems with Generative Retrieval. 2023. arXiv:2305.05065.
22. Xue, L., Barua, A., Constant, N., et al. ByT5: Towards a token-free future with pre-trained byte-to-byte models. 2021. arXiv:2105.13626. Published in Transactions of the Association for Computational Linguistics 10:291–306, 2022.
23. Pagnoni, A., Pasunuru, R., Rodriguez, P., et al. Byte Latent Transformer: Patches Scale Better Than Tokens. 2024. arXiv:2412.09871.
24. Kingma, D. P., Ba, J. Adam: A Method for Stochastic Optimization. 2014. arXiv:1412.6980.
25. Chen, Y., Lv, A., Luan, J., et al. HoPE: A Novel Positional Encoding Without Long-Term Decay for Enhanced Context Awareness and Extrapolation. 2024. arXiv:2410.21216.
26. Geirhos, R., Jacobsen, J.-H., Michaelis, C., et al. Shortcut Learning in Deep Neural Networks. 2020. arXiv:2004.07780. Published in Nature Machine Intelligence 2(11):665–673, 2020.
27. Robinson, J., Chuang, C.-Y., Sra, S., Jegelka, S. Contrastive Learning with Hard Negative Samples. 2020. arXiv:2010.04592.
28. Olsson, C., Elhage, N., Nanda, N., et al. In-context Learning and Induction Heads. 2022. arXiv:2209.11895.
29. Strubell, E., Verga, P., Andor, D., et al. Linguistically-Informed Self-Attention for Semantic Role Labeling. 2018. arXiv:1804.08199.
30. Liu, L., Utiyama, M., Finch, A., Sumita, E. Neural Machine Translation with Supervised Attention. 2016. arXiv:1609.04186.
31. Vinyals, O., Fortunato, M., Jaitly, N. Pointer Networks. 2015. arXiv:1506.03134.
32. Kadlec, R., Schmid, M., Bajgar, O., Kleindienst, J. Text Understanding with the Attention Sum Reader Network. 2016. arXiv:1603.01547.
33. Saxe, A. M., McClelland, J. L., Ganguli, S. Exact solutions to the nonlinear dynamics of learning in deep linear neural networks. 2013. arXiv:1312.6120.
34. Barak, B., Edelman, B. L., Goel, S., et al. Hidden Progress in Deep Learning: SGD Learns Parities Near the Computational Limit. 2022. arXiv:2207.08799.
35. Plate, T. A. Holographic Reduced Representations. IEEE Transactions on Neural Networks 6(3):623–641, 1995. doi:10.1109/72.377968.
36. Schlegel, K., Neubert, P., Protzel, P. A comparison of Vector Symbolic Architectures. 2020. arXiv:2001.11797. Published in Artificial Intelligence Review 55(6):4523–4555, 2022.
37. Alain, G., Bengio, Y. Understanding intermediate layers using linear classifier probes. 2016. arXiv:1610.01644.
38. Ramsauer, H., Schäfl, B., Lehner, J., et al. Hopfield Networks is All You Need. 2020. arXiv:2008.02217.
39. Olver, F. W. J., Olde Daalhuis, A. B., Lozier, D. W., et al. (eds.). NIST Digital Library of Mathematical Functions, §10.32, Eq. 10.32.1. Release 1.2.8, 15 September 2026. dlmf.nist.gov/10.32.E1.
40. Frady, E. P., Kleyko, D., Sommer, F. T. A theory of sequence indexing and working memory in recurrent neural networks. 2018. arXiv:1803.00412. Published in Neural Computation 30(6):1449–1513, 2018.
41. Henry, A., Dachapally, P. R., Pawar, S., Chen, Y. Query-Key Normalization for Transformers. 2020. arXiv:2010.04245.
42. Dehghani, M., Djolonga, J., Mustafa, B., et al. Scaling Vision Transformers to 22 Billion Parameters. 2023. arXiv:2302.05442.
43. Derrida, B. Random-energy model: An exactly solvable model of disordered systems. Physical Review B 24(5):2613–2626, 1981. doi:10.1103/PhysRevB.24.2613.
44. Zhao, Y., Qu, Y., Staniszewski, K., et al. Analysing The Impact of Sequence Composition on Language Model Pre-Training. 2024. arXiv:2402.13991. Published in the Proceedings of ACL 2024 (Long Papers), doi:10.18653/v1/2024.acl-long.427.
45. Loshchilov, I., Hutter, F. SGDR: Stochastic Gradient Descent with Warm Restarts. 2016. arXiv:1608.03983.
46. Hu, S., Tu, Y., Han, X., et al. MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategies. 2024. arXiv:2404.06395.
47. Hägele, A., Bakouch, E., Kosson, A., et al. Scaling Laws and Compute-Optimal Training Beyond Fixed Training Durations. 2024. arXiv:2405.18392.
48. Lin, T.-Y., Goyal, P., Girshick, R., et al. Focal Loss for Dense Object Detection. 2017. arXiv:1708.02002.
49. Lin, Z., Gou, Z., Gong, Y., et al. Rho-1: Not All Tokens Are What You Need. 2024. arXiv:2404.07965.
50. Fredkin, E. Trie Memory. Communications of the ACM 3(9):490–499, 1960. doi:10.1145/367390.367400.
51. Morin, F., Bengio, Y. Hierarchical Probabilistic Neural Network Language Model. Proceedings of the Tenth International Workshop on Artificial Intelligence and Statistics (AISTATS 2005), PMLR R5:246–252, 2005. proceedings.mlr.press/r5/morin05a.
52. He, Z., Feng, G., Luo, S., et al. Two Stones Hit One Bird: Bilevel Positional Encoding for Better Length Extrapolation. 2024. arXiv:2401.16421.
53. Zhang, K., Li, G., Zhang, H., Jin, Z. HiRoPE: Length Extrapolation for Code Models Using Hierarchical Position. 2024. arXiv:2403.19115.
54. Cho, H., Cha, J., Awasthi, P., et al. Position Coupling: Improving Length Generalization of Arithmetic Transformers Using Task Structure. 2024. arXiv:2405.20671.
55. Roller, S., Sukhbaatar, S., Szlam, A., Weston, J. Hash Layers For Large Sparse Models. 2021. arXiv:2106.04426.
56. Wu, X., Huang, S., Wei, F. Mixture of LoRA Experts. 2024. arXiv:2404.13628.
57. Ilharco, G., Ribeiro, M. T., Wortsman, M., et al. Editing Models with Task Arithmetic. 2022. arXiv:2212.04089.
58. Yadav, P., Tam, D., Choshen, L., et al. TIES-Merging: Resolving Interference When Merging Models. 2023. arXiv:2306.01708.
59. Bengio, S., Vinyals, O., Jaitly, N., Shazeer, N. Scheduled Sampling for Sequence Prediction with Recurrent Neural Networks. 2015. arXiv:1506.03099.
60. Nakanishi, K. M. Scalable-Softmax Is Superior for Attention. 2025. arXiv:2501.19399.
61. Chiang, D., Cholak, P. Overcoming a Theoretical Limitation of Self-Attention. 2022. arXiv:2202.12172.

Citation

@misc{abstractphil2026alhena,
  title        = {Alhena: Building a Rotary That Withstands Selection},
  author       = {AbstractPhil},
  year         = {2026},
  month        = oct,
  howpublished = {Hugging Face article, AbstractPowered},
  url          = {https://huggingface.co/blog/AbstractPowered/rotary-addressed-selection},
  note         = {With Claude Opus 5.5 and Claude Fable 5.1 (Anthropic)}
}