Document completed positional screen 817daa9e

README.md

@@ -0,0 +1,147 @@
+---
+license: mit
+language: en
+library_name: pytorch
+pipeline_tag: sentence-similarity
+tags:
+- captionbert
+- rotary
+- sentence-similarity
+- consensus-distillation
+datasets:
+- AbstractPhil/conceptual-captions-12m-webdataset-berts
+- AbstractPhil/captionbert-8192-v2-consensus
+---
+
+# captionbert-8192-rotary
+
+A controlled positional screen of CaptionBERT v2: the learned-position replication (A0), plain token RoPE (A1), sentence-address rotary (A2), and its anchor ablation (A3). Every arm is trained from scratch against the same precomputed five-teacher consensus. Inference uses the tokenizer and one student; teacher models are not loaded.
+
+This is a short screen, separate from the original v2 full training run. Both seeds and all arms, including weaker results, are retained. No single winner is selected automatically.
+
+## Loading
+
+Install the tagged package in an isolated Python 3.11–3.13 environment:
+
+```bash
+pip install git+https://github.com/AbstractEyes/captionbert-trainer.git@v0.2.0
+```
+
+```python
+from captionbert_rotary.inference import SentenceEncoder
+encoder = SentenceEncoder.from_pretrained("AbstractPhil/captionbert-8192-rotary",
+    run="experiments/817daa9e66a8119dc6b18a9b63fad32490d1c891e5f26750e91125d9e5e14e6c/A2-seed1",
+    revision="4e75920dfe6ddd820db321608a853c2097533f4e")
+embeddings = encoder.encode(["A cat watches the street.", "A person walks nearby."])
+# embeddings: (2, 768), L2-normalized
+```
+
+Choose an explicit arm and seed. A2-seed1 above is a loading example, not a claim that it is best. Model code, requirements, tests and notebooks live in [captionbert-trainer](https://github.com/AbstractEyes/captionbert-trainer/tree/v0.2.0).
+
+The [technical companion](https://github.com/AbstractEyes/captionbert-trainer/blob/v0.2.0/docs/rotary_measurements.md) describes the measurement controls, report fields and limits.
+
+## Benchmark
+
+Spearman correlation on the same eight test tasks as the v2 harness; mean is the unweighted task average. The benchmark uses its original 64-token ceiling.
+
+| arm / seed | STS-B | SICK-R | STS12 | STS13 | STS14 | STS15 | STS16 | BIOSSES | mean |
+|---|---:|---:|---:|---:|---:|---:|---:|---:|---:|
+| A0 / 1 | 0.4531 | 0.6199 | 0.4476 | 0.4354 | 0.4537 | 0.6350 | 0.4667 | 0.5083 | 0.5025 |
+| A0 / 2 | 0.4557 | 0.6224 | 0.4292 | 0.4443 | 0.4470 | 0.6445 | 0.4750 | 0.4291 | 0.4934 |
+| A1 / 1 | 0.5057 | 0.6330 | 0.4756 | 0.4316 | 0.4764 | 0.6601 | 0.5166 | 0.4693 | 0.5211 |
+| A1 / 2 | 0.4965 | 0.6314 | 0.4769 | 0.4673 | 0.4736 | 0.6689 | 0.5148 | 0.3934 | 0.5153 |
+| A2 / 1 | 0.5248 | 0.6339 | 0.4817 | 0.4665 | 0.4946 | 0.6665 | 0.5206 | 0.4607 | 0.5312 |
+| A2 / 2 | 0.4864 | 0.6328 | 0.4706 | 0.4405 | 0.4649 | 0.6756 | 0.4949 | 0.4872 | 0.5191 |
+| A3 / 1 | 0.5252 | 0.6383 | 0.4741 | 0.4755 | 0.4931 | 0.6685 | 0.5251 | 0.4983 | 0.5373 |
+| A3 / 2 | 0.4845 | 0.6352 | 0.4528 | 0.4538 | 0.4751 | 0.6762 | 0.5010 | 0.4168 | 0.5119 |
+
+The published full-run v2 mean was 0.6077 after 26.9M rows and 52,548 steps. That is historical context, not the matched control for this screen; the matched control is A0 here.
+
+## Geometry
+
+Effective rank is the participation ratio of centered variance. Self-cosine is the mean off-diagonal cosine, an isotropy readout. Cosine to consensus measures mimicry, separate from task capability.
+
+The following caption-domain measurements use the fixed held-out chunk-61 sample, separately from the benchmark sentences. The >128-token slice is sampled separately from that chunk.
+
+| arm / seed | parameters | steps | held-out cosine | effective rank | self-cosine | seconds/step |
+|---|---:|---:|---:|---:|---:|---:|
+| A0 / 1 | 58,308,864 | 1952 | 0.7226 | 68.19 | 0.0191 | 0.400 |
+| A0 / 2 | 58,308,864 | 1952 | 0.7478 | 71.79 | 0.0181 | 0.400 |
+| A1 / 1 | 54,114,560 | 1952 | 0.7459 | 71.34 | 0.0194 | 0.464 |
+| A1 / 2 | 54,114,560 | 1952 | 0.7615 | 74.13 | 0.0171 | 0.464 |
+| A2 / 1 | 54,114,560 | 1952 | 0.7427 | 71.87 | 0.0204 | 0.464 |
+| A2 / 2 | 54,114,560 | 1952 | 0.7635 | 73.77 | 0.0158 | 0.465 |
+| A3 / 1 | 54,114,560 | 1952 | 0.7429 | 69.80 | 0.0205 | 0.465 |
+| A3 / 2 | 54,114,560 | 1952 | 0.7622 | 73.08 | 0.0169 | 0.465 |
+
+| arm / seed | >128-token cosine | effective rank | self-cosine | measured rows |
+|---|---:|---:|---:|---:|
+| A0 / 1 | 0.6549 | 53.25 | 0.0312 | 5000 |
+| A0 / 2 | 0.6831 | 50.99 | 0.0610 | 5000 |
+| A1 / 1 | 0.6730 | 54.01 | 0.0358 | 5000 |
+| A1 / 2 | 0.6862 | 53.85 | 0.0672 | 5000 |
+| A2 / 1 | 0.6723 | 55.57 | 0.0345 | 5000 |
+| A2 / 2 | 0.6879 | 53.59 | 0.0551 | 5000 |
+| A3 / 1 | 0.6740 | 55.14 | 0.0344 | 5000 |
+| A3 / 2 | 0.6834 | 52.04 | 0.0611 | 5000 |
+
+## Rotary measurements
+
+The fixed frozen-model protocol measures address kernels, simplex coherence and its explicit constant lift, periodic aliases versus the sentence clamp, identical-content oracle address lookup, controlled address changes, q/k-band removals, and per-layer/head energy and attention-mass changes.
+
+The encoder rotates tokens to their own coordinates. It has no learned request head or AMOE member gate. Code-only oracle lookup and embedding sensitivity are not learned semantic-retrieval accuracy or task improvement. The raw anchor code is a regular simplex; the encoder does not fix the constant coordinate of the orthogonal lift.
+
+| arm / seed | repeated baseline max error | sentence-ID collapse, mean L2 change | anchor q/k removal, mean L2 change |
+|---|---:|---:|---:|
+| A0 / 1 | 0.00000000 | n/a | n/a |
+| A0 / 2 | 0.00000000 | n/a | n/a |
+| A1 / 1 | 0.00000000 | 0.000000 | 0.000000 |
+| A1 / 2 | 0.00000000 | 0.000000 | 0.000000 |
+| A2 / 1 | 0.00000000 | 0.272284 | 0.116291 |
+| A2 / 2 | 0.00000000 | 0.318407 | 0.135013 |
+| A3 / 1 | 0.00000000 | 0.000000 | 0.000000 |
+| A3 / 2 | 0.00000000 | 0.000000 | 0.000000 |
+
+A0 has no rotary bands (n/a). A1/A3 have no anchor band, so anchor-removal effects are structural negative controls. Full probe inputs, checkpoint/source hashes, dtype/device, all interventions and every layer/head are in each run's `rotary_diagnostics.json` or additive `analyses/captionbert-rotary-mechanisms-v1/` report. The frozen repeat measures diagnostic-call precision, not training hardware-repeat uncertainty.
+
+## How it was built
+
+| part | screen recipe |
+|---|---|
+| arms / seeds | A0, A1, A2, A3 / 1, 2 |
+| training chunks | [0, 1, 2, 3] |
+| held-out chunk | 61 |
+| epochs / contrastive batch | 2 / 2048 |
+| optimizer | pure Adam, weight decay zero; peak LR 0.0006 |
+| schedule | 2000-step linear warmup from 1%, then cosine |
+| training token ceiling | 256 |
+| objective / pooling | symmetric InfoNCE (T=0.07) + MSE; masked mean; L2-normalized 768-d output |
+
+Captions and consensus tensors are paired by flat row index, with exact row-count checks. Sentence indices advance after `.`, `!` or `?`; the next token resets its sentence-local slot. A2 has content 8, slot 12 (wavelengths 3–256) and anchor 12 pairs per 64-d head; A3 has content 8 and slot 24. A1 uses all 32 pairs on global token index.
+
+## Known limits
+
+- The short screen spends most of its approximately 2,000 steps in the original 2,000-step warmup. It does not reproduce v2's full-run scale.
+- Training is capped at 256 tokens. The model name and A0's 8,192-position table do not establish 8k-context quality.
+- Sentence IDs clamp at 24. Unclamped rotary codes repeat every 25 addresses; neither is unlimited sentence capacity.
+- Caption-domain consensus agreement and STS capability are separate measurements. These five teachers also constrain the target's shared geometry.
+- Paired seeds expose seed dependence; a settled improvement additionally needs the applicable uncertainty/repeat evidence. A large band-removal response alone does not establish useful structure.
+
+## Output convention
+
+The package's `SentenceEncoder.encode` returns a `(batch, 768)` normalized tensor. This repository contains multiple explicit run artifacts; it is not a root-level Transformers `AutoModel` checkpoint.
+
+## Provenance
+
+Experiment `817daa9e66a8119dc6b18a9b63fad32490d1c891e5f26750e91125d9e5e14e6c`; result files read at Hub revision `4e75920dfe6ddd820db321608a853c2097533f4e`. Model-card organization follows the [v2 card](https://huggingface.co/AbstractPhil/captionbert-8192-v2/blob/ed468fe5aed64a3d03a56a7f7811131d20f81ddb/README.md). Original results, every arm/seed, weights and analysis JSON remain under their immutable experiment paths.
+
+## Citation
+
+```bibtex
+@misc{captionbert_rotary_2026,
+  title = {CaptionBERT Sentence-Address Rotary Screen},
+  author = {AbstractPhil},
+  year = {2026},
+  url = {https://huggingface.co/AbstractPhil/captionbert-8192-rotary}
+}
+```