geolip-rotary-experiments

Weights and result records from one day of experiments (2026-10-22) on rotary position encodings used as structural addresses. In these experiments a token's rotary phase comes from where it sits in a fixed structure (which compartment, which slot), not only from its index in the sequence. Small members, each trained alone, are merged into one frozen core behind a gate that reads that address.

Everything was trained from scratch on synthetic sequence tasks, mostly with a 4-layer, width-128 causal transformer (4 heads, context 256). Unless a row says otherwise, every result is two seeds, and the pass marks were written down before each run.

What is here:

folder	contents
`weights/`	475 PyTorch state dicts (1.7 GB): every core, member stack and head the experiments trained, failures and controls included
`reports/`	the result tables (`_tables.md`), one JSON record per run (`phase/`, `speed/`), and the run logs
`reports/capacity_far/`	(2026-10-27) the key rotation's capacity read past a million keys: one record per model, the lanes' replication checks (`gate/`) and claims
`bed/`	(2026-10-27) the later stages trained on the nano byte model: run records (`bed/reports/`), weights (`bed/weights/<run>/`), look, capacity, room and text records
`pillar/`	(2026-10-27) the byte-trigram entropy table that places the unit starts of the path stage

The code that trained and measured these files is not hosted here; this repository holds weights and results only.

The idea in two formulas

For rotate-half RoPE with pair frequencies $\omega_j$ , two identical unit vectors at positions $m$ and $n$ score

$K(m-n) = \frac{1}{P}\sum_{j=1}^{P}\cos\big(\omega_j\,(m-n)\big)$

This is the address kernel. Its main lobe is the resolution, its side lobes are what a wrong address leaks, and the number of positions it can hold apart is its capacity. A query that asks for an address is rotated by that address, and a key by its own.

Members are merged behind an amplitude gate that reads the requested address $r$ against each member's own address $i$ , level by level:

$u_i = U\prod_{l} k_l(r_l, i_l), \qquad w_i = \frac{\sinh u_i}{1 + \sum_j \left(\cosh u_j - 1\right)}$

A member whose address does not match gets $u_{i} = 0$ , and $\sinh 0 = 0$ : it abstains outright, and no member is chosen over another. A matching member gets $w_i \approx \tanh U$ .

The experiments, in order

part	question	size	result (two seeds)
Rotary catalogue (phase A)	which rotary forms work on relative, absolute and compartment lookups	24 forms, 144 runs	NTK-scaled RoPE has the best reach at RoPE's cost; the Cantor rotaries, SinusoidalRoPE at its defaults and an additive quad-rotary embedding fail
Fixed-position prototypes (phase B)	does a query rotated to the address it asks for find it?	62 runs	yes, with every code: compartment lookups 1.000, and absolute slots .97-1.000 at four times the trained length (dynamic forms ~.28 there)
Merging members (phases C, C2a, C2b)	do members trained alone merge on a frozen core without loss or leak?	92 cells	with the structural request: merged within .008 of each member alone, nothing leaked (KL at most .005); ungated members wreck each other (merged .05-.26)
Mixed sequences (phase D)	relative and fixed-position questions in one sequence	24 runs	both kinds answered 1.000
Band capacity (phase E)	how many members one band of anchors holds	16 compartments, 14 members	exact up to its count of orthogonal anchors; a squeezed band leaks in proportion to the anchors' overlap (KL 0 / .6-.8 / 1.6-2.6 at overlap 0 / .56 / .85)
Follow-ups (phases F, F4)	shuffled content, members attached mid-stack, the full normalizer	38 runs	the anchor band carries a compartment's identity when content cannot (1.000 vs .17); mid-stack members work (1.000 vs .04-.07)
Speeds	what every rotary costs at 4k-64k tokens	three timed runs	the house rotary is 3.7-5.3% of a block at 4k and 1.5% at 64k; the anchored rotary costs 1.24-1.27x the house rotary at 4k and about the same once compiled
Screen S1	does address capacity multiply?	2 seeds	one gate over the product of two 9-anchor level kernels addresses 64 compartments exactly in 18 dimensions, where one flat band needs 65
Screen S2	three levels, and how many members one gate holds	2 seeds	512 addresses exact in 27 dimensions; a normalizer that charges resting members nothing removes the member ceiling
Screen S3	where the request comes from when a query names only a key	2 seeds	an index the core writes into the sequence is exact (0 wrong of 65,536 entries) and gives the exact gate; requests read off attention fail
S3 follow-ups	the pointing block alone, a learned address head, why the key lookup trains slowly	2 seeds	below
Screen S3L	a learned address request that does not leak	2 seeds, 2 head forms	a core trained together with its address head gives an exact request (nothing leaked, every mark kept); a floor on a frozen-core head cannot; members on the new cores need a longer member schedule (below)

The S3 follow-ups in more detail

One block of attention is a working request when the core points at the right record. One seed's core points its second block at the asked record (three of four heads at .955). Read from that block alone, the request gives the right member .41 of a full match (.73 in the flat form). It keeps the members' answers (.97) with nothing leaked (.003 of an owned write). The same request averaged over all four blocks gave .010 and silenced every member. The other seed's core points at the next record, so the same read hands the neighbour the weight.

A head trained on a frozen core reads the address as a whole, not digit by digit, and leaks. After the pointing block, a linear head with one sigmoid per compartment names the asked compartment 97% of the time (blocks 2-4). A per-digit head reads only the coarse digit (.96-.98) and not the finer two (.23-.32). The flat head's scores are each judged alone: the largest wrong score is only .03-.04, but 255 such writes add up on a question no member owns (those answers fall to .39-.72 at 256 members). Attention does not leak this way, because its softmax splits one shared budget.

Why the key-to-value lookup trains slowly (the answer job alone, 12,000 steps):

layout	answers right, seed 1 / 2	training loss at 10,000 steps
the S3 layout	.085 / .084	3.17 / 3.18 (= log of the row's ~25 distinct values: it knows the row's values, not which one a key names)
no index sector, the same number of answered questions	.084 / .087	3.18 / 3.16
no index sector, 1.84x the answered questions	.558 / .268	1.50 / 2.45
the questions asked like an index (in key order, each answer shown after its question), values repeating	.173 / .174	2.44 / 2.43
the same, each value unique in its row and its own token	.9999 / 1.000	0.00 / 0.00 (formed between steps 2,000 and 8,000)

Removing the index sector changes nothing at matched supervision; more answered questions per step breaks the plateau slowly; asking the questions like an index entry with unique answers forms the lookup in a few thousand steps, which is how the core's own index formed.

Screen S3L: a learned address request that does not leak

The S3 follow-ups left one problem. A head trained on a frozen core names the asked compartment 97% of the time, but its request leaks in two ways: each of the 511 wrong compartments keeps a small score of its own, and on a question no member owns those small writes add up (the sum); and on about 3% of questions the head's largest score is a wrong compartment, which hands that member most of a full write (a wrong request). The fix had to keep the gate's rule: no member is compared with another.

An absolute floor on the frozen-core head. Each member's factor comes from its own score and one fixed number:

$f(c) = \frac{\max\big(0,\ s(c) - \tau\big)}{1 - \tau}$

with $\tau$ the 90th, 99th or 99.9th percentile of the per-question largest wrong score, measured on calibration rows the read never sees. A score at or under the floor gives exactly 0, and $\sinh 0 = 0$ . The floor removes the sum (the leaked write falls from .51 to .03 / .005 / .0004 of an owned write) but not the wrong requests: a floor low enough to keep the owners still passes the confident wrong scores, and one high enough to stop them silences up to 43% of the owners. No floor height keeps every mark on a frozen core.

A core trained together with its address head. The core is trained by the same recipe plus one term, the head's binary cross-entropy at the questions, summed over its outputs:

$L = \frac{1}{|S|}\sum_{t \in S}\mathrm{CE}(y_t, p_t) \;+\; \lambda\,\frac{1}{Q}\sum_{q}\sum_{o}\mathrm{BCE}\big(z_{q,o},\ \mathrm{target}_{q,o}\big), \qquad \lambda = 1$

in two forms: one sigmoid per compartment (512 outputs) and one sigmoid per digit (3 x 8 outputs). On all four cores (two forms, two seeds) the head names the asked compartment on every one of the read's 158,772 questions, its wrong scores stay near $10^{-7}$ or below, and its request keeps all three marks in every cell with a leaked write of about $10^{-8}$ of an owned write, with or without the floor. The term also makes the key-to-value lookup form in 4,000-8,000 steps (the cores without it needed 48,000-96,000).

core	the head right (158,772 questions)	leaked write / owned write, largest	cells keeping all three marks (plain head; floored at three heights)	members alone after 4,000 / 16,000 member steps
one sigmoid per compartment, seed 1	1.00000	2e-8	9 of 9; 9, 9, 9	.902 / 1.000
one sigmoid per compartment, seed 2	1.00000	7e-9	9 of 9; 9, 9, 9	.988 / 1.000
one sigmoid per digit, seed 1	1.00000	7e-8	9 of 9; 9, 9, 9	.637 / .9998
one sigmoid per digit, seed 2	1.00000	5e-8	9 of 9; 9, 9, 9	.157 / 1.000

The cost is in the members, and it is their schedule. Members trained on these cores by the night's member recipe (4,000 steps, the learning rate decaying to a tenth) learn more slowly: every member run starts on a plateau, a uniform guess over the 64 values, and on the new cores the members are still on it when the learning rate has fallen, so alone they read .16-.99 (on the plain cores .998-1.000). The same recipe over 16,000 steps, with the same data and starting weights, has every member at 1.000 alone and the merged answers at 1.000 with every mark kept. Why the plateau lasts longer on these cores is open: the answer's confidence, the state's size and how much of a member's input carries the value were measured and do not explain it.

How many keys the key rotation keeps apart (2026-10-27)

The later stages of this line ran on a small byte-level language model (about 9 million parameters, six layers, a 4,096-byte window), trained by AbstractEyes/geolip-nano-beatrix-trainer. Its training rows hold 48 records, each a key and a value at an address, and an index the model writes itself: for each asked key, the address of its record. One attention head is told where to look. At each record's end-marker that head's query and key are turned by fixed random phases summed from the key's characters, over 16 rotation pairs, so a record's key and a question carrying the same key line up and other keys do not. Two forms were trained, six seeds each: the turn alone, and the turn with the record's key held at one length at the records' end-markers ("held" below).

The read asks how the head would fare if, instead of the 48 records of its row, it had to pick among N competing records. For every index entry the model writes (4,608 a model), the head's score for the true record is set against N competitors, each a stored record's content turned by a fresh random key's phases, drawn up to N = 10^8 and streamed through the scores in blocks. A wrong look means the competitors' summed weight (the attention's softmax mass) exceeds the true record's; an outright win means one competitor alone scores above the true record. No model was trained again for this read.

form	seed	first wrong look	half the looks wrong	wrong looks at 10^6 / 10^7 / 10^8	outright wins at 10^8
held	1	20.0 million	70.8 million	0.0000 / 0.0000 / 0.8210	0.0000
held	2	5.0 million	63.9 million	0.0000 / 0.0017 / 0.9310	0.0000
held	3	2.0 million	38.5 million	0.0000 / 0.0009 / 0.9993	0.0000
held	4	10.0 million	57.4 million	0.0000 / 0.0009 / 0.9499	0.0000
held	5	5.0 million	59.1 million	0.0000 / 0.0043 / 0.9362	0.0002
held	6	10.0 million	31.4 million	0.0000 / 0.0002 / 1.0000	0.0000
turn only	1	1.0 million	48.9 million	0.0002 / 0.0458 / 0.7826	0.0100
turn only	2	5.0 million	17.5 million	0.0000 / 0.0380 / 1.0000	0.0007
turn only	3	1.0 million	11.7 million	0.0004 / 0.3837 / 1.0000	0.0048
turn only	4	1.0 million	45.2 million	0.0002 / 0.0972 / 0.7407	0.0614
turn only	5	1.0 million	51.5 million	0.0022 / 0.0625 / 0.7526	0.0252
turn only	6	10.0 million	40.6 million	0.0000 / 0.0048 / 0.9742	0.0004

What the numbers say:

Both forms keep a million keys apart (no wrong look at 10^6 with the hold; at most 0.22% of looks without it). With the hold the first wrong looks appear between 2 and 20 million competitors and half the looks go wrong between 31 and 71 million; at ten million 0.00-0.43% of looks are wrong. Without the hold the first wrong looks appear between 1 and 10 million, half go wrong between 12 and 52 million, and at ten million 0.48-38.37% are wrong.
The hold's edge lasts past a million. At ten million competitors every held model makes far fewer wrong looks than the model of the same seed without the hold (none against 211 of 4,608 on seed 1, 15 to 430 times fewer on the other five), and five of the six reach the half point 1.2 to 3.7 times later (the sixth, whose average leak the hold had raised, reaches it earlier).
The looks fail by a crowd, not by a rival. At 10^8, 82-100% of the held models' looks are wrong, yet only 1 entry of 27,648 has a single competitor scoring above the true record (without the hold 74-100% wrong, 472 entries with such a rival). The true record keeps the highest score; the failure is the summed weight of very many weak competitors, the strongest of which carries about 2% of it. For the held models, picking the single highest score would still find the right record at 10^8 on every entry but one.
The crowd's weight grows faster than N, then in step with N. On the held models the typical entry's competing weight grows about 13x a decade up to a million competitors, 10.6-10.8x from one to ten million, and 10.0-10.1x past ten million: once the rare near-aligned competitors are sampled the sum averages out and grows linearly, the crossover of a random energy model.

Each model's record is reports/capacity_far/<model>_capfar_r192_s1.json; each of the three reading lanes first reproduced a stored million-key read exactly (reports/capacity_far/gate/). The models' weights are in the training repository AbstractPhil/alephllm-nano-beatrix-training (bed/weights/); the code is geolip/nano_beatrix/capacity_far.py in the trainer, run by its notebooks colab/nano_capacity_far_lane{1,2,3}.ipynb.

Plain documents in the training mix (2026-10-27)

In the stages above every training row was formatted: half the rows of a batch were record rows (records, an index, questions) and the rest were text with framed lookups placed in it, each row opened by a document byte and a mode byte. The text measurement reads held-out documents from their first byte, a form no training row had. On one of the six held models (seed 4) that left a failure the training loss never saw: opened by the document byte, a plain document was read as if it were a record's content, and its held-out text cost 2.24 bits a byte against about 1.44 on the other seeds.

This stage replaced one non-record row in four with a plain document from the training corpus, started at its first byte, so the measured form is trained. Nothing else changed: the record rows are the same rows byte for byte, and six seeds were trained with the held models' recipe.

seed	held-out text, bits a byte: with / without	a document's first 64 bytes: with / without	document read as a record: with / without	key-hard look	answer through its own index	minutes
1	1.3987 / 1.4422	1.90 / 2.69	no / no	1.0000	0.9990	17.8
2	1.4077 / 1.4402	1.91 / 2.55	no / no	1.0000	0.1899	17.9
3	1.4029 / 1.4472	1.90 / 2.72	no / no	1.0000	0.2274	17.8
4	1.4063 / 2.2379	1.92 / 5.01	no / yes	1.0000	0.9984	17.9
5	1.3989 / 1.4455	1.92 / 2.88	no / no	1.0000	0.9994	17.9
6	1.3906 / 1.4197	1.91 / 2.53	no / no	1.0000	0.1908	17.9

What the numbers say:

The failure is gone on every seed. No model reads a document as a record (0 of 6 flagged, against 1 of 6 without the plain rows), and a document's first 64 bytes cost 1.90-1.92 bits a byte against 2.53-2.88 (the failed seed 5.01) without them. Seed 4, the seed that failed, now reads held-out text like the others.
The text improves everywhere, not just at the start. 6 of 6 models read held-out text better than the model of the same seed without plain documents, by 0.029-0.047 bits a byte on the 5 seeds whose comparison model had not failed, and the middle of a document improves as well as its start: whole documents teach the text's own continuity, which rows cut by records and frames do not.
The lookups and the capacity are untouched. The key-hard lookups stay exact (the lowest model 1.0000), the answers given the record's address stay at .999, and no model makes a wrong look at a million competing keys (the worst 0.0000).
The written index stays a matter of seed. Three of the six write their own index right (four of six without the plain rows); the others write the right address for about a fifth of the keys, an outcome seen on some seeds of every model family in this line.

The models are in this repository under bed/weights/<run>/ (the run names end in _plain16; the plain rows add _p75d), with their run records, look records and capacity records under bed/reports/. The trainer is AbstractEyes/geolip-nano-beatrix-trainer (--plant=0.75 --plain_doc=1), run by its notebook colab/nano_plain16.ipynb.

Four-digit record addresses (2026-10-27)

Every row in the stages above holds at most 48 records: a record's address is two digits of eight (64 addresses), and the row's 4,096 bytes are cut into 64 slots of 64 bytes. The key rotation keeps millions of keys apart, so the address itself was the next limit on how many records one row can hold.

This stage writes the address as four digits of four (256 addresses) on the same eight anchor pairs of the lookup head. Each digit gets two of the pairs, turned by $2\pi k/5$ times the digit ($k = 1, 2$), so that a digit's five symbols, its four values and "no address", sit as a simplex. Summed over one digit's pairs, two identical unit vectors turned to digits $a$ and $b$ score

$\sum_{k=1}^{R/2}\cos\Big(\frac{2\pi k}{R+1}\,(a-b)\Big) = \begin{cases} R/2 & a = b \\ -\tfrac{1}{2} & a \neq b \end{cases}$

with $R = 4$ here, and $R = 8$ (four pairs a digit) for the two-digit code of the stages above. The address head, the value path's tables and the split of the head's pairs keep their sizes. A new row form holds 96 records: 128 slots of 32 bytes, 96 of the 256 addresses used, 24 of the records asked in the index and the questions. Six seeds were trained at 96 records a row and six at 48 (the earlier row, with the new code), each with the plain-documents recipe above and nothing else changed; the comparison models are that section's six, not trained again.

model	held-out text, bits a byte: four-digit / two-digit (same seed)	answer given the address / given none	key-hard look	look at the question	answer through its own index	wrong looks / outright wrong winners at 10^6	minutes
96 records, seed 1	1.4146 / 1.3987	0.9993 / 0.0000	1.0000	1.0000	0.9993	0.0000 / 0.0000	19.7
96 records, seed 2	1.4167 / 1.4077	0.9990 / 0.0000	1.0000	1.0000	0.9990	0.0000 / 0.0000	19.7
96 records, seed 3	1.4209 / 1.4029	0.9998 / 0.0000	1.0000	1.0000	0.9998	0.0000 / 0.0000	19.7
96 records, seed 4	1.4093 / 1.4063	0.9989 / 0.0000	1.0000	1.0000	0.0549	0.0000 / 0.0000	19.7
96 records, seed 5	1.4155 / 1.3989	0.9992 / 0.0000	1.0000	0.9960	0.9991	0.0000 / 0.0000	19.7
96 records, seed 6	1.4121 / 1.3906	0.9990 / 0.0000	1.0000	1.0000	0.9990	0.0000 / 0.0000	19.7
48 records, seed 1	1.4073 / 1.3987	0.9990 / 0.0000	1.0000	1.0000	0.7928	0.0000 / 0.0000	19.0
48 records, seed 2	1.4029 / 1.4077	0.9989 / 0.0000	1.0000	1.0000	0.8153	0.0000 / 0.0000	19.0
48 records, seed 3	1.4156 / 1.4029	0.9985 / 0.0000	1.0000	1.0000	0.1654	0.0000 / 0.0000	19.0
48 records, seed 4	1.4131 / 1.4063	0.9993 / 0.0000	1.0000	1.0000	0.9984	0.0000 / 0.0000	19.0
48 records, seed 5	1.4027 / 1.3989	0.9979 / 0.0000	1.0000	1.0000	0.3892	0.0000 / 0.0000	19.0
48 records, seed 6	1.3970 / 1.3906	0.9989 / 0.0000	1.0000	1.0000	0.9989	0.0000 / 0.0000	18.9

What the numbers say:

Ninety-six records a row resolve. On all twelve models the lookup head finds the whole key on every key-hard question, and the record it points at from the asked key's index entry is the right one on every question. Given the record's address, the share of questions answered exactly is .9989-.9998 at 96 records (.9979-.9993 at 48), and none without an address. No model makes a wrong look at a million competing keys, and none has a single competitor scoring above the true record there.
The code costs nothing at 48 records. With only the code changed, the six 48-record models answer within .0015 of the two-digit models of the same seeds and read held-out text within -0.005 to +0.013 bits a byte of them.
Two marks set before the runs were missed, each by one model of six. On one 96-record model the head's look taken at the question itself, rather than at the index entry, has no record above half its attention on 0.4% of questions (0.65% on the key-hard rows; the other eleven models 0%): part of its attention goes to the address digits written in the row's index. Only 2 of its 12,288 read questions went to a wrong record; on the rest a gate that reads this look abstains instead of writing the wrong member. And the 96-record models read held-out text 0.003 to 0.022 bits a byte worse than the two-digit models of the same seeds, one model 0.0006 past the mark. The cost is spread evenly over every position of a document and every kind of byte: a 96-record row carries about half the ordinary text of a 48-record row (1,308 bytes against 2,339), and half of every batch is record rows.
The model's own index. Whole on five of the six 96-record models and two of the six 48-record models (three of the six two-digit models). It forms in jumps at a step that differs from seed to seed. Where it has not formed by the end of training, the digits it has learned are right on every entry and the rest are picked among the row's records that share them, so with four digits a model stopped at three loses about one entry in five at 48 records, where a two-digit model stopped at one loses four in five.

The models are in this repository under bed/weights/<run>/ (the run names end in _code44; the four-digit code adds _ac44, and _gc96 / _gc48 name the records a row), with their run records, look records and capacity records under bed/reports/. The trainer is AbstractEyes/geolip-nano-beatrix-trainer (--grid_code=4x4, and --grid_comps=96 for the larger row), run by its notebooks colab/nano_code44_lane1.ipynb and colab/nano_code44_lane2.ipynb; the capacity read by colab/nano_capacity_c44.ipynb.

The path since the last choice point, in one head's rotation (2026-10-27)

The key rotation above puts a record's key into one head's rotation, so that head finds the record whose key matches. This stage gives every byte such a key: its path, the bytes since the text's last choice point. A byte starts a new unit when the byte after the three before it is hard to predict (at least 3.0 bits of entropy in a table of byte trigrams counted over the training corpus, pillar/ here), so a unit runs from one choice point to the next: 3.6 bytes on average on the held-out text, 14 at the 99.9th percentile. About two thirds of the positions of a 4,096-byte window end a path that occurred earlier in the same window.

Head 7 of every block turns its eight address pairs by the phases of the path's bytes (its last 16), at every position:

$\phi_j(t) = \sum_{k=0}^{\min(t - s(t) + 1,\, 16) - 1} P_{j,\,k,\,x_{t-k}} \pmod{2\pi}, \qquad j = 0, \dots, 7$

where $s (t)$ is the start of the unit holding byte $t$ and $P$ is a fixed table of random angles (8 x 16 x 256, drawn once from a seed). Two forms were trained, six seeds each. In the same-path form the query and the key at each position turn by its own path, so equal paths meet as if unrotated. In the byte-after form the key at position $u$ turns by the path that ended at $u - 1$ , so a query meets the bytes that followed its own path earlier in the row: a copy-the-continuation lookup whose context is the unit's path, a different length at every position. No parameter is added and nothing is added to any input; every other head and pair keeps its rotation, the lookup head's key rotation included. On plain text these eight pairs have no record address to carry, so in the comparison models they act as eight more position-free content channels of the head; both forms give them up to the path. The comparison models are the six plain-documents models above, not trained again.

model	held-out text, bits a byte	against the comparison model (same seed)	with the path switched off	against the comparison: where the path occurred earlier / elsewhere	answer given the address	key-hard look	minutes
byte-after, seed 1	1.3891	-0.0096	1.4868 (+0.0976)	-0.0173 / +0.0042	0.9988	1.0000	19.4
byte-after, seed 2	1.3910	-0.0167	1.4947 (+0.1037)	-0.0225 / -0.0063	0.9995	1.0000	19.4
byte-after, seed 3	1.3990	-0.0039	1.5123 (+0.1133)	-0.0114 / +0.0093	0.9977	1.0000	19.4
byte-after, seed 4	1.4169	+0.0106	1.5205 (+0.1037)	+0.0036 / +0.0228	0.9990	1.0000	19.3
byte-after, seed 5	1.4128	+0.0138	1.5393 (+0.1266)	+0.0080 / +0.0242	0.9978	1.0000	19.4
byte-after, seed 6	1.3990	+0.0084	1.5003 (+0.1013)	+0.0037 / +0.0168	0.9991	1.0000	19.3
same-path, seed 1	1.4047	+0.0060	1.4331 (+0.0284)	+0.0052 / +0.0073	0.9993	1.0000	19.4
same-path, seed 2	1.4027	-0.0049	1.4376 (+0.0349)	-0.0081 / +0.0006	0.9993	1.0000	19.3
same-path, seed 3	1.4032	+0.0004	1.4274 (+0.0242)	-0.0014 / +0.0035	0.9994	1.0000	19.4
same-path, seed 4	1.4110	+0.0046	1.4370 (+0.0260)	+0.0063 / +0.0015	0.9986	1.0000	19.4
same-path, seed 5	1.4147	+0.0158	1.4557 (+0.0410)	+0.0155 / +0.0162	0.9996	1.0000	19.4
same-path, seed 6	1.3976	+0.0070	1.4228 (+0.0252)	+0.0077 / +0.0058	0.9995	1.0000	19.4

What the numbers say:

The model leans on the path. Switching it off costs the byte-after models 0.098-0.127 bits a byte (the same-path models 0.024-0.041). It matters most where the next byte is a capital or a digit (names and numbers that recur), and it misleads at word ends: switching it off improves the prediction of spaces on all six byte-after models.
Held-out text is unchanged on average. The byte-after models read held-out text +0.0004 bits a byte against the comparison on average (-0.0167 to +0.0138); two of six were 0.005 or more better, short of the four of six set as the mark before the runs. The same-path models read +0.0048 on average.
The lookup pays where the path repeats; the given-up channels cost everywhere. Against the same-path models of the same seeds, which give up the same channels without the continuation, the byte-after models are 0.010 bits a byte better at the positions whose path occurred earlier, on all six seeds, and 0.006 worse elsewhere. The same-path models cost a flat 0.004-0.006 at both against the comparison: the price of the eight channels.
The cost sits at the start of a row and the gain later. Against the comparison the byte-after models are +0.0081 in a document's first 64 bytes, +0.0117 up to 256 and -0.0016 past 2,048; switching the path off costs 0.04-0.05 in the first 64 bytes and 0.11-0.14 past 2,048, as the earlier text fills the table of paths.
The seeds split in halves. Seeds 1-3 of the byte-after form are better than their comparison models and seeds 4-6 worse, and the same-path form's seeds 4-6 also cost more (+0.0091 on average against +0.0005 for seeds 1-3), with the two training runtimes crossed between the forms: the split follows the seed's starting weights and data order, which each model shares with its comparison model.
The lookups are untouched. The key-hard look is exact on all twelve models and the answer given the record's address is .9977-.9996 (the comparison models .9986-.9994).

The models are in this repository under bed/weights/<run>/ (the run names end in _pillar1; the path adds _pbn8h7, the byte-after form, or _pbs8h7, the same-path form), with their run records and look records under bed/reports/ and their text reads (the split at the repeated paths, the path switched off) under bed/reports/text/. The trainer is AbstractEyes/geolip-nano-beatrix-trainer (--path_bind=8 --path_form=next or same), run by its notebooks colab/nano_pillar1_lane1.ipynb and colab/nano_pillar1_lane2.ipynb.

The path in a head of its own (2026-10-27)

In the stage above, the path took over one of each block's eight heads, and that head gave up its own work for it. A read of the stored models (bed/reports/room/) found that every model trained without the path grows a head that does the same lookup by itself: at the positions whose path occurred earlier in the row, one head in blocks 2-4 puts about a third of its attention on the bytes that followed the earlier copies. This stage gives the path a head of its own: a ninth attention head in every block, beside the eight, with its own query, key, value and output weights (196,608 parameters in all, under the same optimizer), its sixteen rotation pairs turned by the path's phases in the byte-after form,

$\phi_j(t) = \sum_{k=0}^{\min(t - s(t) + 1,\, 16) - 1} P_{j,\,k,\,x_{t-k}} \pmod{2\pi}, \qquad j = 0, \dots, 15,$

the query at $t$ turned by $\phi(t)$ and the key at $u$ by $\phi(u-1)$ . As the control, the same ninth head sits on the block's own rotation, the angles every head of the block gets. The new heads are drawn from their own seeded generator, the same in both, so each model starts exactly as the comparison model of its seed plus the new head. Six seeds each, at the plain-documents recipe's 24,000 steps; the comparison models are the six plain-documents models above, not trained again.

model	held-out text, bits a byte	against the comparison model (same seed)	against its partner (same seed)	with the path switched off	against its partner: where the path occurred earlier / elsewhere	the new head's lookup (its largest block)	the eight heads' largest lookup	answer given the address	key-hard look	minutes
path head, seed 1	1.3958	-0.0029	-0.0004	1.5031 (+0.1072)	-0.0080 / +0.0131	0.445	0.090	0.9972	1.0000	20.0
path head, seed 2	1.4162	+0.0085	+0.0206	1.5652 (+0.1490)	+0.0101 / +0.0393	0.442	0.032	0.9984	1.0000	19.9
path head, seed 3	1.3815	-0.0214	-0.0129	1.5136 (+0.1321)	-0.0233 / +0.0055	0.430	0.022	0.9998	1.0000	20.0
path head, seed 4	1.3964	-0.0099	-0.0090	1.4696 (+0.0732)	-0.0138 / -0.0004	0.425	0.141	0.9981	1.0000	20.0
path head, seed 5	1.4060	+0.0071	+0.0071	1.5222 (+0.1162)	-0.0021 / +0.0235	0.417	0.131	0.9980	1.0000	19.9
path head, seed 6	1.3915	+0.0009	+0.0083	1.4889 (+0.0974)	+0.0008 / +0.0218	0.395	0.138	0.9990	1.0000	20.0
plain head, seed 1	1.3962	-0.0025	+0.0004	-	+0.0080 / -0.0131	0.011	0.356	0.9994	1.0000	20.6
plain head, seed 2	1.3956	-0.0121	-0.0206	-	-0.0101 / -0.0393	0.032	0.310	0.9986	1.0000	20.6
plain head, seed 3	1.3944	-0.0085	+0.0129	-	+0.0233 / -0.0055	0.010	0.353	0.9989	1.0000	20.6
plain head, seed 4	1.4054	-0.0010	+0.0090	-	+0.0138 / +0.0004	0.009	0.356	0.9998	1.0000	20.6
plain head, seed 5	1.3989	0.0000	-0.0071	-	+0.0021 / -0.0235	0.313	0.336	0.9997	1.0000	20.7
plain head, seed 6	1.3832	-0.0074	-0.0083	-	-0.0008 / -0.0218	0.008	0.322	0.9995	1.0000	20.6

What the numbers say:

An extra head pays a little on its own. The plain new head reads -0.0052 bits a byte against the comparison models on average (-0.0121 to 0.0000), at the repeated paths and elsewhere alike.
The path head takes over the lookup. In every path model the new head puts 0.395-0.445 of its attention on the right continuation in one block, a little more than the comparison models' learned heads (0.318-0.367), and the eight ordinary heads no longer grow one of their own (their largest 0.022-0.141, against 0.310-0.356 in the plain-head models).
It is no better than the plain extra head. Against it the path models read +0.0023 on average (-0.0129 to +0.0206): seeds 3 and 4 better by 0.005 or more, seeds 2, 5 and 6 worse by as much, short of the four seeds of six set as the mark before the runs. The path head helps where the path occurred earlier (-0.0061) and costs elsewhere (+0.0171), most at the start of a row, where nothing has repeated yet (+0.0444 in the first 64 bytes, -0.0015 past 2,048). Switching it off costs 0.073-0.149 bits a byte, more the further into the row.
The lookups are untouched. The key-hard look is 1.0000 on all twelve models and the answer given the record's address 0.9972-0.9998.

The models are in this repository under bed/weights/<run>/ (the run names end in _pillar1b; the path head adds _nhpn16, the plain head _nhpl; the new heads' weights in extra_head.safetensors), with their run records, look and room records and text reads under bed/reports/. The trainer is AbstractEyes/geolip-nano-beatrix-trainer (--ninth_head=path --path_form=next or --ninth_head=plain; ninth_head.py), run by its notebooks colab/nano_pillar1b_lane1.ipynb to colab/nano_pillar1b_lane3.ipynb.

The path in the early blocks (2026-10-27)

In the stage above, each model's path head did its lookup in one block, and where that block fell lined up with the text: the models whose lookup formed in block 3 read better than the models without the path, the ones whose lookup formed in blocks 5 or 6 worse. An earlier finding on this model points the same way: the middle of the stack can use only what is in place by block 3. This stage puts the path in the new heads of blocks 1-3 only, so the lookup has to form early, and gives the new heads of blocks 4-6 the block's own rotation. The heads are born as in the stage above, so each model starts exactly as that stage's models of the same seed and differs from the one with an ordinary new head in every block only in the rotation of blocks 1-3. Six seeds at the plain-documents recipe's 24,000 steps; the comparison models are the stage above's and the plain-documents models, not trained again.

model	held-out text, bits a byte	against the comparison model (same seed)	against the ordinary new head (same seed)	against the path in every block (same seed)	with the path switched off	against the ordinary new head: where the path occurred earlier / elsewhere	the new heads' lookup in blocks 1-3 (its block)	the eight heads' largest lookup	answer given the address	key-hard look	minutes
seed 1	1.3959	-0.0028	-0.0003	+0.0001	1.4553 (+0.0594)	-0.0042 / +0.0066	0.438 (3)	0.241	0.9992	1.0000	20.3
seed 2	1.3865	-0.0211	-0.0091	-0.0296	1.4382 (+0.0517)	-0.0136 / -0.0009	0.441 (3)	0.222	0.9991	1.0000	20.4
seed 3	1.3874	-0.0154	-0.0070	+0.0060	1.4531 (+0.0657)	-0.0097 / -0.0020	0.444 (3)	0.221	0.9979	1.0000	20.5
seed 4	1.3935	-0.0128	-0.0118	-0.0029	1.4668 (+0.0733)	-0.0173 / -0.0022	0.423 (3)	0.124	0.9979	1.0000	20.4
seed 5	1.3968	-0.0021	-0.0021	-0.0092	1.4568 (+0.0600)	-0.0054 / +0.0039	0.458 (2)	0.233	0.9989	1.0000	20.3
seed 6	1.3907	+0.0001	+0.0075	-0.0008	1.4392 (+0.0485)	+0.0048 / +0.0124	0.446 (2)	0.215	0.9971	1.0000	20.4

What the numbers say:

The lookup forms early. In every model a path head puts 0.423-0.458 of its attention on the right continuation: in block 3 on seeds 1, 2, 3 and 4, in block 2 on seeds 5 and 6. The new heads of blocks 4-6 form none (their largest 0.007-0.052).
Better on average, not on enough seeds. Against the ordinary new head the models read -0.0038 bits a byte on average (-0.0118 to +0.0075): seeds 2, 3 and 4 better by 0.005 or more, seed 6 worse by as much, one seed short of the four of six set as the mark before the runs. Against the path in every block they read -0.0061, and against the models without a new head -0.0090.
Block 3 again. The four models whose lookup formed in block 3 read -0.0070 against the ordinary new head on average; the two that formed it in block 2 read +0.0027. Over the eighteen path models of the three stages (the path in one ordinary head, in every block's new head, in blocks 1-3's), the six that formed the lookup in block 3 read -0.0139 against the models without the path, the ten that formed it in blocks 4-6 +0.0016, the two in block 2 -0.0010: a pattern read after the runs, not a registered test.
Less cost at the start of a row. Against the ordinary new head the cost in the first 64 bytes is +0.0176 (the path in every block: +0.0444) and the gain past 2,048 bytes -0.0055 (there: -0.0015). Switching the path off costs 0.048-0.073 bits a byte, more the further into the row.
The ordinary heads keep part of the job. Their largest lookup is 0.124-0.241 (with the path in every block 0.022-0.141, with the ordinary new head 0.310-0.356).
The lookups are untouched. The key-hard look is 1.0000 on all six models and the answer given the record's address 0.9971-0.9992.

The models are in this repository under bed/weights/<run>/ (the run names end in _pillar1c, with _nhpn16b123 for the path in blocks 1, 2 and 3; the new heads' weights in extra_head.safetensors), with their run records, look and room records and text reads under bed/reports/. The trainer is AbstractEyes/geolip-nano-beatrix-trainer 0.1.95 (--ninth_head=path --path_form=next --path_blocks=1,2,3; ninth_head.py), run by its notebooks colab/nano_pillar1c_lane1.ipynb to colab/nano_pillar1c_lane3.ipynb.

The path in block 3 only (2026-10-27)

In the two stages above, each model's path head did its lookup in one block, and the models whose lookup sat in block 3 read best. This stage asks whether block 3 itself is the reason: the path goes in block 3's new head only, every other block's new head gets the block's own rotation, so every model's path lookup can only form in block 3. The heads are born as in the stages above (each model starts exactly as their models of the same seed). Six seeds at the plain-documents recipe's 24,000 steps; the comparison models are the stages above and the plain-documents models, not trained again.

model	held-out text, bits a byte	against the comparison model (same seed)	against the ordinary new head (same seed)	against the path in blocks 1-3 (same seed)	with the path switched off	the path head's lookup (block 3)	the eight heads' largest lookup (block, head)	answer given the address	key-hard look	minutes
seed 1	1.4020	+0.0033	+0.0058	+0.0061	1.4396 (+0.0376)	0.456	0.231 (3, 7)	0.9993	1.0000	21.0
seed 2	1.3987	-0.0090	+0.0031	+0.0122	1.4110 (+0.0122)	0.017	0.338 (2, 2)	0.9993	1.0000	21.0
seed 3	1.3877	-0.0152	-0.0067	+0.0002	1.4486 (+0.0609)	0.437	0.024 (6, 1)	0.9984	1.0000	20.8
seed 4	1.3919	-0.0144	-0.0134	-0.0016	1.4568 (+0.0649)	0.432	0.105 (5, 7)	0.9979	1.0000	21.0
seed 5	1.4003	+0.0014	+0.0014	+0.0035	1.4158 (+0.0155)	0.011	0.343 (2, 0)	0.9989	1.0000	20.9
seed 6	1.3916	+0.0010	+0.0084	+0.0009	1.4068 (+0.0152)	0.368	0.343 (3, 3)	0.9995	1.0000	20.9

What the numbers say:

Four seeds take the path, two ignore it. On seeds 1, 3, 4 and 6 the path head in block 3 does the lookup (0.368-0.456 of its attention on the right continuation). On seeds 2 and 5 it stays unused (0.011-0.017) and the model builds its own lookup in block 2 instead, in the same head the models without the path use (block 2, head 2 on seed 2; block 2, head 0 on seed 5). On seed 6 the model's own lookup head in block 3 (head 3, 0.343) forms beside the path head and does the same job, so switching the path off costs little there (+0.015, against +0.061 and +0.065 on seeds 3 and 4).
No gain on average. Against the ordinary new head the models read -0.0002 bits a byte on average (-0.0134 to +0.0084): seeds 3 and 4 better by 0.005 or more, two short of the four of six set as the mark before the runs. Against the path in blocks 1-3 they read +0.0036 on average, and against the models without a new head -0.0055.
The seed decides more than the block. Seed by seed, the gain against the ordinary new head follows the one with the path in blocks 1-3 (correlation 0.80) and with the path in every block (0.69): seeds 3 and 4 gain in every form, seed 6 in none. The models of the earlier stages that chose block 3 for their lookup read -0.0084 against the ordinary new head; with the lookup placed in block 3 the six read -0.0002. The earlier pattern came mostly from which seeds chose block 3.
The lookup is a race. Each seed's models without the path grow one lookup head of their own, in a place mostly set by the seed. A path head gets there faster, and how much faster depends on how many path heads there are: against the ordinary new head at 500 steps the training loss is lower by 0.21-0.28 bits a byte with the path in every block, by 0.06-0.25 with the path in blocks 1-3, and with block 3 alone by 0.22 and 0.19 on seeds 3 and 4 but only 0.03-0.04 on the others. Where the path is slow the seed's own lookup head forms first and the path is left unused or shared.
Less cost at the start of a row. Against the ordinary new head the first 64 bytes cost +0.0114 (the path in blocks 1-3: +0.0176) and the bytes past 2,048 read -0.0010 (there: -0.0055). Switching the path off costs 0.012-0.065 bits a byte.
The lookups are untouched. The key-hard look is 1.0000 on all six models and the answer given the record's address 0.9979-0.9995.

The models are in this repository under bed/weights/<run>/ (the run names end in _pillar1d, with _nhpn16b3 for the path in block 3; the new heads' weights in extra_head.safetensors), with their run records, look and room records and text reads under bed/reports/. The trainer is AbstractEyes/geolip-nano-beatrix-trainer 0.1.95 (--ninth_head=path --path_form=next --path_blocks=3; ninth_head.py), run by its notebooks colab/nano_pillar1d_lane1.ipynb to colab/nano_pillar1d_lane3.ipynb.

The path switched off one block at a time (2026-10-28)

With the path in the new heads of blocks 1-3, every model learned its lookup in one of the three path heads; with the path in block 3 alone, the model's own lookup head often formed first. So the three path heads matter while a model trains. This read asks whether they matter in the finished models. Nothing is trained: in each of the six models with the path in blocks 1-3, the path is switched off in one block at a time and in two at a time (that block's new head put on the block's own rotation, the weights unchanged), and the held-out text is read again; the six models with the path in every block are read one block at a time. With no block switched off each read equals the model's stored text read, and with every block switched off its stored switched-off read, exactly.

model	held-out text, bits a byte	path off in block 1	in block 2	in block 3	in blocks 1 and 2	in blocks 1 and 3	in blocks 2 and 3	in all three	the lookup's block (its attention on the right continuation)	that block's share of the whole
seed 1	1.3959	+0.0085	+0.0113	+0.0418	+0.0211	+0.0478	+0.0521	+0.0594	3 (0.438)	0.70
seed 2	1.3865	+0.0075	+0.0068	+0.0386	+0.0152	+0.0446	+0.0446	+0.0517	3 (0.441)	0.75
seed 3	1.3874	+0.0060	+0.0074	+0.0530	+0.0147	+0.0579	+0.0598	+0.0657	3 (0.444)	0.81
seed 4	1.3935	+0.0060	+0.0069	+0.0592	+0.0136	+0.0652	+0.0669	+0.0733	3 (0.423)	0.81
seed 5	1.3968	+0.0076	+0.0418	+0.0094	+0.0488	+0.0184	+0.0515	+0.0600	2 (0.458)	0.70
seed 6	1.3907	+0.0120	+0.0302	+0.0063	+0.0405	+0.0192	+0.0371	+0.0485	2 (0.446)	0.62

What the numbers say (costs in bits a byte against the same model with the path on):

The lookup's block carries most of it, not all. Switching off the path only in the block that holds the lookup costs 0.62-0.81 of switching it off in all three; that cost sits almost entirely where the byte's path occurred earlier in the row.
The other path heads still matter. Their attention on the right continuation is small (0.006-0.152, against 0.423-0.458 in the lookup's head), yet each costs +0.006 to +0.012 alone and the two together +0.014 to +0.021. Byte for byte, that cost is larger where the byte's path had not occurred earlier in the row (+0.014 to +0.025 there, against +0.010 to +0.015 at the repeats): these heads do something besides the lookup. The costs add up: the three single blocks' costs sum to within 0.002 of switching off all three.
The same with the path in every block. The lookup's block carries 0.51-0.78 of the cost on five seeds; on seed 3, the block-6 head with a second lookup (0.36 of its attention) costs more (+0.066) than the block-3 head with the strongest one (0.43; +0.018). Of the rest, the early path heads cost the most (blocks 1 and 2: +0.005 to +0.015; blocks 4-6: +0.001 to +0.005).
What this measures. Switching a trained head's rotation shows how much the model leans on the path in that block, not what the path is worth: against the same models with an ordinary new head in every block, the path in blocks 1-3 gains 0.0038 bits a byte on average.

The records are in this repository under bed/reports/group/ (<run>_group.json: each set's held-out bits and their split at the repeated paths). The trainer is AbstractEyes/geolip-nano-beatrix-trainer 0.1.96 (rotary_scale.block_read), run by its notebook colab/nano_pillar1g.ipynb.

Random phases in place of the path (2026-10-28)

The read above switched the path off by putting a path head on its block's own rotation. That does two things at once: it removes the path's matches, and it gives the head a positional rotation it was never trained with (away from its matches a path head meets its keys at unrelated phases, so it sees no position). This read separates the two. Nothing is trained: each path head is put instead on fresh random phases, its own draw for the query and for the key at every position (the same draw in every read). That removes all the path's phases carry and leaves the head position-free. With no head changed, each read equals the model's stored text read exactly. The table gives each cost on random phases beside the cost of the block's own rotation from the read above, in the six models with the path in blocks 1-3.

model	held-out text, bits a byte	block 1: random / own rotation	block 2	block 3	all three	the lookup's block
seed 1	1.3959	+0.0065 / +0.0085	+0.0094 / +0.0113	+0.0561 / +0.0418	+0.0700 / +0.0594	3
seed 2	1.3865	+0.0057 / +0.0075	+0.0053 / +0.0068	+0.0511 / +0.0386	+0.0609 / +0.0517	3
seed 3	1.3874	+0.0047 / +0.0060	+0.0059 / +0.0074	+0.0702 / +0.0530	+0.0797 / +0.0657	3
seed 4	1.3935	+0.0043 / +0.0060	+0.0051 / +0.0069	+0.0791 / +0.0592	+0.0892 / +0.0733	3
seed 5	1.3968	+0.0064 / +0.0076	+0.0615 / +0.0418	+0.0072 / +0.0094	+0.0767 / +0.0600	2
seed 6	1.3907	+0.0095 / +0.0120	+0.0478 / +0.0302	+0.0050 / +0.0063	+0.0614 / +0.0485	2

What the numbers say (costs in bits a byte against the same model as trained):

The lookup's cost is the path, and the switch under-read it. On random phases the lookup's head costs +0.048 to +0.079, which is 1.32-1.58 times its cost on the block's own rotation (+0.030 to +0.059). With a coherent rotation the lookup head still recovers part of the continuation (the likely route, not measured here: matching by content alone); the difference sits at the repeated paths (-0.0298 to -0.0195 there, -0.0026 to -0.0001 elsewhere). Of the whole path's cost on random phases, the lookup's block carries 0.78-0.89 (the switch said 0.62-0.81).
The other path heads use the path too. On random phases each costs +0.004 to +0.009, which is 0.72-0.85 of its cost on the block's own rotation; the change of rotation adds only +0.001 to +0.003. Byte for byte, that cost is larger where the byte's path had not occurred earlier in the row on five of the six models, and the same on the sixth (+0.0049 to +0.0099 there, against +0.0046 to +0.0068 at the repeats). There is no exact match there, so these heads use the path's phases in some other way. The costs add: all three blocks on random phases cost within 0.0020 of the three single blocks' sum.
The same with the path in every block. The lookup's head costs 1.15-3.44 times its switch on random phases. On seed 3, the read above found the block-3 head with the strongest lookup costing less (+0.0177) than the block-6 head with a second one (+0.0664). On random phases they cost +0.0609 and +0.0778: most of that gap came from the switch.

The records are in this repository under bed/reports/group/ (<run>_group_random.json: each set's held-out bits and their split at the repeated paths). The trainer is AbstractEyes/geolip-nano-beatrix-trainer 0.1.97 (rotary_scale.block_read(..., mode="random"), ninth_head.random_phases), run by its notebook colab/nano_pillar1h.ipynb.

A fresh phase table in place of the path's (2026-10-28)

Random phases take away two things at once: that a path's phases are the same wherever the path stands (so a query meets every earlier copy of its own path exactly), and the particular values the phase table gave each path, which a trained model can learn to use. This read keeps the first and changes the second. Each path head takes its path's phases from a fresh table of the same kind: equal paths still meet at equal phases, so every exact match is kept; only the values change. Nothing is trained. Each read is also split by how deep the byte sits in its unit. With no head changed each read equals the model's stored text read, and the random-phase reads equal the read above, exactly. The six models with the path in blocks 1-3:

model	held-out text, bits a byte	block 1: fresh table / random	block 2	block 3	all three	the lookup's block
seed 1	1.3959	+0.0081 / +0.0065	+0.0114 / +0.0094	+0.0010 / +0.0561	+0.0221 / +0.0700	3
seed 2	1.3865	+0.0067 / +0.0057	+0.0070 / +0.0053	+0.0010 / +0.0511	+0.0162 / +0.0609	3
seed 3	1.3874	+0.0060 / +0.0047	+0.0070 / +0.0059	+0.0007 / +0.0702	+0.0150 / +0.0797	3
seed 4	1.3935	+0.0048 / +0.0043	+0.0054 / +0.0051	+0.0009 / +0.0791	+0.0119 / +0.0892	3
seed 5	1.3968	+0.0073 / +0.0064	+0.0009 / +0.0615	+0.0094 / +0.0072	+0.0189 / +0.0767	2
seed 6	1.3907	+0.0119 / +0.0095	+0.0009 / +0.0478	+0.0062 / +0.0050	+0.0203 / +0.0614	2

What the numbers say (costs in bits a byte against the same model as trained):

The lookup works by exact matching alone. On the fresh table the lookup's head costs +0.0007 to +0.0010 (0.011-0.019 of its cost on random phases, +0.048 to +0.079), and at most +0.0014 at any depth. For the lookup, only that equal paths meet matters; the values do not.
The other path heads read the values. On the fresh table each costs +0.0048 to +0.0119: 1.06-1.32 times its cost on random phases. They have learned a code from the table's values, and a fresh table costs them more than noise does (the likely reading: wrong but consistent values mislead them more than random ones). Their cost is spread over every depth (+0.0044 to +0.0113 by depth, the two heads' mean), not only at short paths, and it is larger where the byte's path had not occurred earlier in the row (+0.0060 to +0.0120 there, against +0.0046 to +0.0085 at the repeats).
The whole path, two layers. All three blocks on the fresh table cost +0.0119 to +0.0221, against +0.0609 to +0.0892 on random phases: of what the trained models lean on in the path, the exact matching is most and the learned code the rest.
The same with the path in every block. Each model's lookup costs 0.006-0.024 of its random-phase cost on the fresh table, and so does seed 3's second lookup in block 6 (0.013); most of the other heads cost more on the fresh table than on random phases.

The records are in this repository under bed/reports/group/ (<run>_group_relabel_depth.json and <run>_group_random_depth.json: each set's held-out bits, their split at the repeated paths, and by depth). The trainer is AbstractEyes/geolip-nano-beatrix-trainer 0.1.98 (rotary_scale.block_read(..., mode="relabel", depth=True), ninth_head.relabel_table), run by its notebook colab/nano_pillar1j.ipynb.

One part of the phase table at a time (2026-10-28)

A byte's phase is a sum of one table term per byte of its path: the current byte's term, the term of the byte before it, and so on back to the start of its unit. The read above changed the whole table. This read changes one part of it at a time: only the current byte's term, only the deeper terms, or only the previous byte's term. Every exact match is kept in each case. Nothing is trained. With every term changed the reads equal the read above, exactly. The six models with the path in blocks 1-3 (costs in bits a byte against the model as trained):

model	block 1: all / current / deeper / previous	block 2	block 3	the lookup's block
seed 1	+0.0081 / +0.0080 / +0.0076 / +0.0075	+0.0114 / +0.0108 / +0.0106 / +0.0099	+0.0010 / +0.0008 / +0.0007 / +0.0007	3
seed 2	+0.0067 / +0.0067 / +0.0058 / +0.0063	+0.0070 / +0.0065 / +0.0063 / +0.0061	+0.0010 / +0.0007 / +0.0007 / +0.0005	3
seed 3	+0.0060 / +0.0060 / +0.0044 / +0.0045	+0.0070 / +0.0069 / +0.0062 / +0.0059	+0.0007 / +0.0005 / +0.0006 / +0.0005	3
seed 4	+0.0048 / +0.0045 / +0.0045 / +0.0045	+0.0054 / +0.0051 / +0.0049 / +0.0050	+0.0009 / +0.0006 / +0.0008 / +0.0007	3
seed 5	+0.0073 / +0.0068 / +0.0067 / +0.0062	+0.0009 / +0.0006 / +0.0008 / +0.0006	+0.0094 / +0.0085 / +0.0080 / +0.0075	2
seed 6	+0.0119 / +0.0113 / +0.0093 / +0.0091	+0.0009 / +0.0009 / +0.0009 / +0.0007	+0.0062 / +0.0060 / +0.0058 / +0.0057	2

What the numbers say:

The lookup is untouched by any part: at most +0.0010 whichever part changes.
The other path heads lose almost all of their code whichever part changes. The current byte's term alone costs 0.90-0.99 of the whole change, the deeper terms alone 0.74-0.94, the previous byte's term alone 0.74-0.94; the first two together come to 1.73-1.93 times the whole. The code is not held in one term: it reads each path's phase as a whole, and a change to any term breaks it.
Where a byte starts its unit, the query's phase has only the current byte's term, so changing the deeper terms moves only the keys' phases: there it costs 0.49-0.66 of the whole change; deeper in the unit it costs 0.93-1.02. The likely reading: these heads have learned what the angle between two particular different paths means, and moving either path's phase breaks it.
Matching on the path minus its last byte (the trie's siblings: the same parent) would leave the deeper change harmless. It costs nearly the whole, so that is not what carries the code.
The same with the path in every block: for the other heads whose code costs more than 0.003, the current byte's term alone costs 0.88-1.08 of their code, the deeper terms alone 0.81-0.97.

The records are in this repository under bed/reports/group/ (<run>_group_relabel_k<slots>_depth.json, slots 0-15, 0, 1-15 and 1). The trainer is AbstractEyes/geolip-nano-beatrix-trainer 0.1.99 (rotary_scale.block_read(..., mode="relabel", slots=...), ninth_head.relabel_table(k, slots)), run by its notebook colab/nano_pillar1k.ipynb.

Caveats

Small synthetic beds: a 4-layer width-128 core, full attention at context 256, 64 value tokens, at most 512 compartments.
Two seeds per cell; several results differ by seed in how the core solved the task (which record its attention points at).
The member weights only make sense with their frozen cores (both are here; the file names pair them).

File names

<phase>_<form>[_<position mode>]_<task>_s<seed>_st<steps>_d128L4.pt for cores (phases A, B, D, F; screens S1-S3). Member stacks name the gate, the site and the core they belong to (C_* and E_* end in _members.pt; S1_members_*, S2_members_*, S3_members_*). smoke_* are the bench's 300-step smoke tests. S3G_* are the S3 follow-ups: the address heads (S3G_head_*, S3G_HEADP*, S3G_HEADF*) and the layout runs (S3G_rope_p_struct_<layout>_*). S3L_* are screen S3L: the cores trained with their address heads (S3L_JF_* one sigmoid per compartment, S3L_JP_* one sigmoid per digit; each head in the matching *_jhead.pt), their member stacks (S3L_members_m256p_<core>_st4000.pt and _st16000.pt, the member steps), and the trainer parity runs (S3L_R3_*). Every run's JSON record in reports/ names its weights file, its configuration and its measured results.

Load a state dict with torch.load(path, map_location="cpu", weights_only=True).

Downloads last month: -; Downloads are not tracked for this model. How to track

Inference Providers NEW

This model isn't deployed by any Inference Provider. 🙋 Ask for provider support