technical companion (week-in-review docket, verified)

TECHNICAL.md

@@ -0,0 +1,201 @@
+# CaptionBERT-8192 v2 / v2-B — Technical Companion
+
+**Models:** [`AbstractPhil/captionbert-8192-v2`](https://huggingface.co/AbstractPhil/captionbert-8192-v2) · [`AbstractPhil/captionbert-8192-v2-B`](https://huggingface.co/AbstractPhil/captionbert-8192-v2-B)
+**Data:** [`AbstractPhil/captionbert-8192-v2-consensus`](https://huggingface.co/datasets/AbstractPhil/captionbert-8192-v2-consensus) · [`AbstractPhil/conceptual-captions-12m-webdataset-berts`](https://huggingface.co/datasets/AbstractPhil/conceptual-captions-12m-webdataset-berts)
+
+---
+
+## 1. Overview
+
+captionbert-8192-v2 and captionbert-8192-v2-B are sibling 58.3M-parameter text encoders trained from scratch by **consensus distillation**: the training target is the *agreement* of five BERT-family teachers (bert-base, distilbert, roberta-base, albert-base-v2, ModernBERT-base), computed over 27–32 million image captions, and neither model ever saw a similarity label. Each ships with an AMOE adapter library — three task-anchored adapters dispatched by a graded router over the frozen backbone — that lifts the eight-task benchmark mean from .6077/.6031 to .7287/**.7294**, past every teacher. This document records the full recipe, the ten-model benchmark run under one shared harness, the adapter-library findings including a transplant/portability study, and the measured laws and limits the family established.
+
+## 2. Training recipe
+
+### Architecture (identical for both backbones)
+
+| | |
+|---|---|
+| Parameters | 58,308,864 (0.53× bert-base, so the compression claim survives) |
+| Layers / hidden / heads | 12 / 512 / 8 |
+| FFN | 2048 |
+| Positions | 8,192 |
+| Output | 768-d, mean-pooled, L2-normalized |
+
+### The consensus target
+
+Each of the five teachers embeds the CC12M LLaVA-NeXT captions. Each teacher then receives **one global whitened-Procrustes map** into bert-base's coordinate frame — bert-base serves as a fixed reference *member* of the ensemble. The consensus vector for a caption is the normalized centroid of the five aligned embeddings. The student regresses onto these precomputed vectors and contrasts against them; the similarity structure it learns is entirely implicit in teacher agreement.
+
+### Objective and optimization
+
+- **Loss:** InfoNCE (temperature 0.07) + per-sample MSE against the precomputed consensus vector.
+- **Optimizer:** Adam, weight decay 0.
+- **Schedule:** lr 6e-4, cosine decay, batch 2048.
+
+### The two runs
+
+| | v2 | v2-B |
+|---|---|---|
+| Corpus | 26.9M rows (54 of 66 chunks; 56 usable before the ModernBERT repair) | 31,905,616 rows (66/66 chunks, repaired corpus) |
+| Steps | 52,548 | 62,312 |
+| Wall time | 5h24m | ~6.4h |
+| Hardware | single RTX 6000 Pro | single RTX 6000 Pro |
+| Data filtering | none | none |
+
+v2-B reuses **the same alignment maps** as v2. This is load-bearing: refitting the maps on the repaired corpus would place the new consensus targets in a different coordinate frame, and the loss carries no signal about frame identity.
+
+### Alignment-fit discipline (the positive control)
+
+The five alignment maps were fit on a stratified random sample of 24k rows (N/d = 31.3) and reported **out of sample**. The worst teacher map scores 1,833× chance out-of-sample and retains 95% of its in-sample R@1. A retired predecessor audit in this program, fit at N/d = 4.9, retained 3.8% — that single ratio separates a map at the null from a map carrying real structure.
+
+The maps also recovered a lineage nobody supplied: out-of-sample alignment cosine orders distilbert .625 ≫ roberta .372 > albert .331 ≈ ModernBERT .327 — exactly the architectural distance of each teacher from bert-base (DistilBERT is literally distilled from it).
+
+### Reference-member alignment pins the frame
+
+Because the consensus is expressed in a fixed member's coordinates, the gap between the student's frame and the target frame collapsed monotonically toward zero during training (+.0235 → +.0070 → +.0015 → +.0005 → +.0010 final). This line therefore ships with **no rotation matrix**. The vision-side consensus line of the same research program aligns to a running mean — a frame-free target — and must ship a 0.5MB rotation. The same construction, measured on both sides, yields opposite deployment requirements.
+
+## 3. Results
+
+### Training-side gauges
+
+| gauge | v2 | v2-B |
+|---|---|---|
+| Target cosine | .8410 | .8421 |
+| Embedding effective rank | 102.9 | 103.1 |
+| Self-cosine (anisotropy) | +.0031 | +.0030 |
+
+The two backbones are indistinguishable on every training-side gauge.
+
+### The eight-task benchmark: 10 models, one harness
+
+All models are mean-pooled and L2-normalized, with no task tuning, run through one shared harness under identical conditions. This table superseded every earlier comparison in the campaign (earlier tables had mixed a 2-task teacher run with an 8-task run — and mis-identified the best teacher as a result). Scores are Spearman correlations.
+
+| model | params | STS-B | SICK-R | STS12 | STS13 | STS14 | STS15 | STS16 | BIOSSES | mean |
+|---|---|---|---|---|---|---|---|---|---|---|
+| bert-base | 109.5M | .4729 | .5865 | .3087 | .5988 | .4773 | .6029 | .6373 | .5469 | .5289 |
+| ModernBERT-base | 149.0M | .4215 | .5479 | .3527 | .4247 | .3795 | .5349 | .4174 | .5630 | .4552 |
+| roberta-base | 124.6M | .5436 | .6296 | .3211 | .5631 | .4522 | .6134 | .6198 | .5777 | .5401 |
+| albert-base-v2 | 11.7M | .4784 | .5364 | .3101 | .4831 | .3809 | .5542 | .5491 | .4863 | .4723 |
+| distilbert | 66.4M | .5717 | .6424 | .4344 | .6490 | .5410 | .6663 | .6854 | .5162 | .5883 |
+| captionbert-8192-v2 | 58.3M | .5747 | .6526 | .5051 | .5995 | .5452 | .7136 | .6776 | .5933 | **.6077** |
+| captionbert-8192-v2-B | 58.3M | .5752 | .6548 | .5012 | .6037 | .5470 | .7146 | .6782 | .5500 | **.6031** |
+| v2 + arms | 63.2M | .7684 | .7391 | .6682 | .7557 | .6921 | .8055 | .7626 | .6382 | **.7287** |
+| v2-B + arms (native) | 63.2M | .7675 | .7374 | .6706 | .7381 | .6945 | .8109 | .7695 | .6472 | **.7294** |
+| all-MiniLM-L6-v2 | 22.7M | .8203 | .7758 | .7237 | .8058 | .7559 | .8539 | .7899 | .8144 | .7925 |
+
+Reading the table row by row:
+
+- **bert-base (.5289)** — the reference member whose frame every teacher was aligned into; the consensus construction is literally "agreement expressed in bert-base's coordinates," and the student beats it by +.0788 at half the size.
+- **ModernBERT-base (.4552)** — the largest model in the table and the lowest mean: parameter count buys nothing on this benchmark. It was also the teacher whose embeddings went missing from 10 corpus chunks, triggering the repair that produced v2-B.
+- **roberta-base (.5401)** — the previously presumed best teacher, demoted by the 8-task view; its row documents that the earlier 2-task comparison had picked the wrong bar.
+- **albert-base-v2 (.4723)** — the 11.7M-parameter floor of the teacher family, showing the consensus was carried by teacher agreement rather than by uniformly strong parents.
+- **distilbert (.5883)** — the actual best teacher, and therefore the bar the whole "student exceeds its teachers" claim is measured against.
+- **captionbert-8192-v2 (.6077)** — the headline row: exceeds all five teachers at 13% of their combined 461M parameters, with the margin concentrated in STS12, where every teacher collapses to .31–.43 and both students hold ~.50.
+- **captionbert-8192-v2-B (.6031)** — the data-volume null made visible: 19% more data sits in the same row of the table (see §6).
+- **v2 + arms (.7287)** — what ~5M adapter parameters on a frozen backbone buy: +.1210, alongside an effective-rank jump from 36.6 to 57.6.
+- **v2-B + arms (.7294)** — the best artifact in the family, and the closing point of the portability arc: native adapters recover 100% of what transplanted adapters lost (§5).
+- **all-MiniLM-L6-v2 (.7925)** — the supervised reference, trained on 1B+ curated pairs; the adapted pair closes to within .063 of it, converting the recipe's ceiling into a concrete distance.
+
+### Replication at scale
+
+"Student exceeds its teachers" had failed to replicate at v1 scale (26M parameters / 500k rows), where the student sat at the teacher median. At v2 scale it replicates: **.6077 against the best teacher's .5883**, at 13% of the teachers' combined 461M parameters, with no similarity label ever seen. The earlier conclusion failed at that *scale*; the modality itself supports the result.
+
+### Geometry tracks capability
+
+Across all ten models on STS-B sentences, self-cosine (anisotropy) and effective rank track capability: roberta-base sits at the bottom at +.9594 self-cosine / effective rank 19.8, all-MiniLM-L6-v2 at the top at +.0251 / 86.7, and the adapted pair between them at +.0964 / 57.6 and +.0984 / 55.5. No training stack anywhere in the table contains an isotropy objective. Isotropy is the property that moved, unasked.
+
+## 4. The AMOE adapter library
+
+Each backbone carries a library of three **anchors** — adapters trained for equivalence, simplification, and paraphrase — dispatched by a graded router (sinh/cosh gating) over the **frozen** backbone. Anchors toggle bit-exactly (asserted at load, never assumed), so a single artifact serves as both the bare baseline and the adapted model.
+
+### Collective and per-arm numbers
+
+- **v2 + arms: .7287** eight-task mean. **v2-B + its own native arms: .7294**. The two are statistically tied (collective-level seed spread .003–.005); the tiebreak is provenance, and v2-B is the complete-corpus backbone with no asterisk — **the best artifact in the family**.
+- **Eight for eight:** the 3-arm collective beats every one of its own members on every task. The sharpest column is BIOSSES: the equivalence anchor alone reads .5845, *below* the bare backbone's .5933, while the collective reads .6382 — the mixture rescues a task where its strongest member is a liability.
+- **Geometry moves with capability:** the arms lift effective rank 36.6 → 57.6 (v2) and 36.1 → 55.5 (v2-B), and cut self-cosine +.1396 → +.0964 — the first evidence in this line that adaptation *adds usable directions* to the representation rather than only rotating existing ones.
+
+### The capacity control
+
+In greedy roster selection, a completely **random (untrained) arm gains +.0045 by itself** — so every candidate is judged against random-at-same-size rather than against the bare backbone. Under that bar:
+
+- **Paraphrase** cleared random by +.0146 and joined the roster.
+- **Lexical (WordNet)** and **topical (Specter)** candidates scored *below* random and were rejected.
+- Every 4th-arm candidate — including a second random arm — scored below the 3-arm collective, so selection halted at exactly the arm count the conserved-amplitude arithmetic predicted (measured amplitude shares .4987 / .1944 / .1810, sum .874).
+
+### A masked member is a damped member
+
+Masking an anchor leaves the dispatch budget unrenormalized. The simplification anchor reads .7400 trained-and-run alone, and .6104 when read masked inside the collective. Mixture margins must therefore be quoted against solo runs, never against damped masked rows — an earlier +.0213 margin claim was corrected to +.0124 on exactly this.
+
+### Sharper routing is worse
+
+Routing temperature .10 / .05 / .02 gives seed-mean STS-B .7506 / .7492 / .7441 (2 seeds, monotone) while routing separation *rises* — the router committed harder and lost. Graded blending carries the value of the dispatch; hard selection destroys it.
+
+## 5. Anchors are backbone-bound (the transplant study)
+
+The two backbones are indistinguishable on every gauge available: eight STS tasks within ±.0042 (mean delta +.0009 excluding the 100-row BIOSSES), effective rank 36.6 vs 36.1, self-cosine +.1396 vs +.1411. Yet:
+
+- v2's arm collective attached to v2-B **unchanged** falls .7287 → **.6863** — 68.8% of the arm gain retained, a 31% loss with no error and no warning. The collective drops *more* than its worst member (−.0425 vs −.0330): the dispatch compounds the mismatch.
+- **The recovery arc, with preregistered pass bars:** re-aligning only the 1,536 routing keys (800 steps, anchors frozen) recovers 29% (.6987); retraining the three anchors on v2-B (1,500 steps each) recovers 100% (**.7294**, above v2's own .7287). Decomposition: **71% of the loss lives in the anchors (.0307), 29% in the routing keys (.0124)**.
+- **The mechanism:** adapters read the *residual stream*; task gauges read the *pooled output*. The stream carries backbone identity that the pooled output does not — which makes 1.6M adapter parameters a more sensitive instrument for backbone identity than the entire 8-task suite.
+- Three hypotheses died in the retrain, all refuted by the same result: "v2-B is less adaptable" (it reaches .7294), "v2-B is over-specialized to captions" (the self-cosine signal was noise), and "v2-B is worse out of domain" (BIOSSES *inverts* to .6472 on v2-B with native anchors vs .6382 on v2 — the deficit was an artifact of foreign anchors).
+- **Operational cost of doing it right:** one native anchor set per backbone, ~18 minutes of training.
+
+## 6. Laws established by this family
+
+Each law below carries the measurement that earned it.
+
+### Measurement discipline
+
+Established on a retired predecessor audit in this program; enforced throughout this family.
+
+1. **A contrastive objective whose two views share one forward pass is underdetermined.** The cheapest tell: retrieval scores invariant in gallery size.
+2. **No retrieval number is quoted until the query set is shown to vary** — length histogram, pooled-query effective rank, pairwise cosine — checked before the first arm was trained.
+3. **An alignment map is never scored on the rows it was fit to.** At N/d = 4.9 the predecessor retained 3.8% of in-sample R@1 out of sample; at N/d = 31.3 this family retained 95%.
+4. **Effective rank is always reported beside CV.** CV is dimension-dependent; a CV target is implicitly a claim about effective dimension.
+5. **A control must be able to fail before its null is citable.**
+6. **The round-trip law:** a diagnostic of the form f(x, g⁻¹(g(x))) measures the operator, never the input; every diagnostic feature needs data on both sides.
+7. **Mimicry and capability are separate gauges:** student→target retrieval and task scores live on separate lines with separate names.
+8. **Inspect the input distribution before designing the experiment.** Validity picks the split; convenience gets no vote.
+
+### Dispatch laws
+
+Measured on the adapter campaign.
+
+9. **The amplitude budget is conserved.** Arms split a fixed sum — measured shares .4987 / .1944 / .1810, sum .874, as predicted — so arm count is an empirical outcome, never a design choice.
+10. **Sharper routing is worse** (2 seeds, monotone): temperature .10/.05/.02 → STS-B .7506/.7492/.7441. Graded blending carries the value.
+11. **The amplitude decomposition runs before roster selection** — it is a pre-flight instrument.
+12. **The one-task exemption:** cross-task destruction by always-on expert stacks is absent where every arm serves one task; the exemption is scoped strictly to that condition.
+13. **The capacity control is the baseline:** a random arm at the same size (+.0045 by itself) is the correct comparator for any roster claim.
+14. **A masked member is a damped member:** masking leaves the dispatch budget unrenormalized (simplification: .7400 solo vs .6104 masked; a +.0213 margin corrected to +.0124).
+
+### Transfer and reliability laws
+
+Established on the v2-B leg.
+
+15. **Anchors are backbone-bound:** 71% of transplant loss lives in the anchors, 29% in the routing keys; one native anchor set per backbone costs ~18 minutes, and a hardcoded adapter repo in a loader is a silent 31% loss with no error.
+16. **A completed checkpoint is unresumable into a new corpus:** epoch == epochs makes the training loop an empty range that reprints the previous run's metrics as its own.
+17. **Skipping a pipeline stage means fetching its outputs:** a refit alignment map lands in a different frame with no signal in the loss.
+18. **A benchmark must be independent of the artifact it measures:** the harness had briefly resolved the adapters through the model's own remote loader before this was caught.
+19. **A distribution shift is judged on p99 and max, never the mean:** a .906 mean-length ratio fired a false alarm while p99 matched to 1% and the repaired maxima were shorter.
+
+## 7. Known limits and honest caveats
+
+### The consensus-rank ceiling
+
+The consensus **target** itself has effective rank **28.7 of 768** (26.9–29.6 across 56 chunks): five BERT-family teachers agree on only ~29 directions. The **student** builds 102.9 effective directions in-domain (rank grew monotonically through training and never turned down), yet only 33.4–36.6 of them survive on STS-B sentences — the ~74 caption-built directions stay in-domain. For scale: all-MiniLM-L6-v2 uses 86.7–94.3 effective directions on the same sentences at 2.6× fewer parameters. The limit is set by **teacher agreement**, and the student's in-domain rank shows capacity to spare; the named next lever is *heterogeneous* teachers, and it is measurable at consensus-build time, before any training is spent.
+
+One caveat inside this story: the out-of-domain effective rank of 36.6 sits within ~4 points of bert-base's 32.8, so part of that column may be a property of the evaluation data itself. The clean control — bert-base embedded on the CC12M captions — was never run and is the top open item.
+
+### The data-volume null
+
+19% more rows for 19% more steps — the complete repaired corpus versus the 54-chunk subset — moved nothing: 8-task mean .6031 vs .6077, where the entire −.0046 is BIOSSES; excluding it, **+.0009**. Training metrics are equally flat (target cosine .8421 vs .8410, effective rank 103.1 vs 102.9). Under this recipe, more of the same distribution leaves a ~29-direction target where it is. (Corpus provenance, for completeness: ModernBERT embeddings had been missing from 10 of 66 chunks; the repair was gate-verified at cosine 1.00000, minimum 1.00000, against a re-embed, and the "long captions broke it" hypothesis was refuted — p99 caption lengths were identical across repaired and never-missing chunks, and the repaired maxima were *shorter*.)
+
+### Statistical and provenance caveats
+
+- The v2-B backbone is single-seed. The arm and temperature sweeps carry 2 seeds; collective-level seed spread is .003–.005.
+- The .7287 / .7294 comparison is between each line's own shipped checkpoint selection: v2 ships its best checkpoint, v2-B its final. The two v2-B checkpoints differ materially (max weight delta 1.319e-01). It is the right comparison — each number belongs to its line's shipped artifact — and it is stated here so the reader can weigh it.
+- BIOSSES is 100 rows with a Spearman standard error of ~0.10; every BIOSSES delta in this document is sub-sigma except where stated.
+- The supervised reference all-MiniLM-L6-v2 (.7925, trained on 1B+ curated pairs) sits .063 above the adapted pair; that gap is the concrete distance this recipe has left to close.
+
+---
+
+*Trained and evaluated in 2026 by AbstractPhil. All artifacts — both backbones, their adapter libraries, the consensus dataset, and the caption corpus — are at [huggingface.co/AbstractPhil](https://huggingface.co/AbstractPhil).*