TECHNICAL.md

# CaptionBERT-8192 v2 / v2-B — Technical Companion

**Models:** [`AbstractPhil/captionbert-8192-v2`](https://huggingface.co/AbstractPhil/captionbert-8192-v2) · [`AbstractPhil/captionbert-8192-v2-B`](https://huggingface.co/AbstractPhil/captionbert-8192-v2-B)
**Data:** [`AbstractPhil/captionbert-8192-v2-consensus`](https://huggingface.co/datasets/AbstractPhil/captionbert-8192-v2-consensus) · [`AbstractPhil/conceptual-captions-12m-webdataset-berts`](https://huggingface.co/datasets/AbstractPhil/conceptual-captions-12m-webdataset-berts)

---

## 1. Overview

captionbert-8192-v2 and captionbert-8192-v2-B are sibling 58.3M-parameter text encoders trained from scratch by **consensus [distillation](https://arxiv.org/abs/1503.02531)**: the training target is the *agreement* of five BERT-family teachers ([bert-base](https://arxiv.org/abs/1810.04805), [distilbert](https://arxiv.org/abs/1910.01108), [roberta-base](https://arxiv.org/abs/1907.11692), [albert-base-v2](https://arxiv.org/abs/1909.11942), [ModernBERT-base](https://arxiv.org/abs/2412.13663)), computed over 27–32 million image captions, and neither model ever saw a similarity label. Each ships with an AMOE adapter library — three task-anchored adapters dispatched by a graded router over the frozen backbone — that lifts the eight-task benchmark mean from .6077/.6031 to .7287/**.7294**, past every teacher. This document records the full recipe, the ten-model benchmark run under one shared harness, the adapter-library findings including a transplant/portability study, and the measured laws and limits the family established.

## 2. Training recipe

### Architecture (identical for both backbones)

| | |
|---|---|
| Parameters | 58,308,864 (0.53× bert-base, so the compression claim survives) |
| Layers / hidden / heads | 12 / 512 / 8 |
| FFN | 2048 |
| Positions | 8,192 |
| Output | 768-d, mean-pooled, L2-normalized |

### The consensus target

Each of the five teachers embeds the [CC12M](https://arxiv.org/abs/2102.08981) LLaVA-NeXT captions. Each teacher then receives **one global whitened-[Procrustes](https://doi.org/10.1007/BF02291478) map** into bert-base's coordinate frame — bert-base serves as a fixed reference *member* of the ensemble. The consensus vector for a caption is the normalized centroid of the five aligned embeddings. The student regresses onto these precomputed vectors and contrasts against them; the similarity structure it learns is entirely implicit in teacher agreement.

### Objective and optimization

- **Loss:** [InfoNCE](https://arxiv.org/abs/1807.03748) (temperature 0.07) + per-sample MSE against the precomputed consensus vector.
- **Optimizer:** Adam, weight decay 0.
- **Schedule:** lr 6e-4, cosine decay, batch 2048.

### The two runs

| | v2 | v2-B |
|---|---|---|
| Corpus | 26.9M rows (54 of 66 chunks; 56 usable before the ModernBERT repair) | 31,905,616 rows (66/66 chunks, repaired corpus) |
| Steps | 52,548 | 62,312 |
| Wall time | 5h24m | ~6.4h |
| Hardware | single RTX 6000 Pro | single RTX 6000 Pro |
| Data filtering | none | none |

v2-B reuses **the same alignment maps** as v2. This is load-bearing: refitting the maps on the repaired corpus would place the new consensus targets in a different coordinate frame, and the loss carries no signal about frame identity.

### Alignment-fit discipline (the positive control)

The five alignment maps were fit on a stratified random sample of 24k rows (N/d = 31.3) and reported **out of sample**. The worst teacher map scores 1,833× chance out-of-sample and retains 95% of its in-sample R@1. A retired predecessor audit in this program, fit at N/d = 4.9, retained 3.8% — that single ratio separates a map at the null from a map carrying real structure.

The maps also recovered a lineage nobody supplied: out-of-sample alignment cosine orders distilbert .625 ≫ roberta .372 > albert .331 ≈ ModernBERT .327 — exactly the architectural distance of each teacher from bert-base (DistilBERT is literally distilled from it).

### Reference-member alignment pins the frame

Because the consensus is expressed in a fixed member's coordinates, the gap between the student's frame and the target frame collapsed monotonically toward zero during training (+.0235 → +.0070 → +.0015 → +.0005 → +.0010 final). This line therefore ships with **no rotation matrix**. The [vision-side consensus line](https://huggingface.co/blog/AbstractPhil/geometric-memory-ft4) of the same research program aligns to a running mean — a frame-free target — and must ship a 0.5MB rotation. The same construction, measured on both sides, yields opposite deployment requirements.

## 3. Results

### Training-side gauges

| gauge | v2 | v2-B |
|---|---|---|
| Target cosine | .8410 | .8421 |
| Embedding effective rank | 102.9 | 103.1 |
| Self-cosine (anisotropy) | +.0031 | +.0030 |

The two backbones are indistinguishable on every training-side gauge.

### The eight-task benchmark: 10 models, one harness

All models are mean-pooled and L2-normalized, with no task tuning, run through one shared harness under identical conditions. This table superseded every earlier comparison in the campaign (earlier tables had mixed a 2-task teacher run with an 8-task run — and mis-identified the best teacher as a result). Scores are Spearman correlations.

| model | params | STS-B | SICK-R | STS12 | STS13 | STS14 | STS15 | STS16 | BIOSSES | mean |
|---|---|---|---|---|---|---|---|---|---|---|
| bert-base | 109.5M | .4729 | .5865 | .3087 | .5988 | .4773 | .6029 | .6373 | .5469 | .5289 |
| ModernBERT-base | 149.0M | .4215 | .5479 | .3527 | .4247 | .3795 | .5349 | .4174 | .5630 | .4552 |
| roberta-base | 124.6M | .5436 | .6296 | .3211 | .5631 | .4522 | .6134 | .6198 | .5777 | .5401 |
| albert-base-v2 | 11.7M | .4784 | .5364 | .3101 | .4831 | .3809 | .5542 | .5491 | .4863 | .4723 |
| distilbert | 66.4M | .5717 | .6424 | .4344 | .6490 | .5410 | .6663 | .6854 | .5162 | .5883 |
| captionbert-8192-v2 | 58.3M | .5747 | .6526 | .5051 | .5995 | .5452 | .7136 | .6776 | .5933 | **.6077** |
| captionbert-8192-v2-B | 58.3M | .5752 | .6548 | .5012 | .6037 | .5470 | .7146 | .6782 | .5500 | **.6031** |
| v2 + arms | 63.2M | .7684 | .7391 | .6682 | .7557 | .6921 | .8055 | .7626 | .6382 | **.7287** |
| v2-B + arms (native) | 63.2M | .7675 | .7374 | .6706 | .7381 | .6945 | .8109 | .7695 | .6472 | **.7294** |
| [all-MiniLM-L6-v2](https://arxiv.org/abs/2002.10957) | 22.7M | .8203 | .7758 | .7237 | .8058 | .7559 | .8539 | .7899 | .8144 | .7925 |

Reading the table row by row:

- **bert-base (.5289)** — the reference member whose frame every teacher was aligned into; the consensus construction is literally "agreement expressed in bert-base's coordinates," and the student beats it by +.0788 at half the size.
- **ModernBERT-base (.4552)** — the largest model in the table and the lowest mean: parameter count buys nothing on this benchmark. It was also the teacher whose embeddings went missing from 10 corpus chunks, triggering the repair that produced v2-B.
- **roberta-base (.5401)** — the previously presumed best teacher, demoted by the 8-task view; its row documents that the earlier 2-task comparison had picked the wrong bar.
- **albert-base-v2 (.4723)** — the 11.7M-parameter floor of the teacher family, showing the consensus was carried by teacher agreement rather than by uniformly strong parents.
- **distilbert (.5883)** — the actual best teacher, and therefore the bar the whole "student exceeds its teachers" claim is measured against.
- **captionbert-8192-v2 (.6077)** — the headline row: exceeds all five teachers at 13% of their combined 461M parameters, with the margin concentrated in STS12, where every teacher collapses to .31–.43 and both students hold ~.50.
- **captionbert-8192-v2-B (.6031)** — the data-volume null made visible: 19% more data sits in the same row of the table (see §6).
- **v2 + arms (.7287)** — what ~5M adapter parameters on a frozen backbone buy: +.1210, alongside an effective-rank jump from 36.6 to 57.6.
- **v2-B + arms (.7294)** — the best artifact in the family, and the closing point of the portability arc: native adapters recover 100% of what transplanted adapters lost (§5).
- **all-MiniLM-L6-v2 (.7925)** — the supervised reference, trained on 1B+ curated pairs; the adapted pair closes to within .063 of it, converting the recipe's ceiling into a concrete distance.

### Replication at scale

"Student exceeds its teachers" had failed to replicate at v1 scale (26M parameters / 500k rows), where the student sat at the teacher median. At v2 scale it replicates: **.6077 against the best teacher's .5883**, at 13% of the teachers' combined 461M parameters, with no similarity label ever seen. The earlier conclusion failed at that *scale*; the modality itself supports the result.

### Geometry tracks capability

Across all ten models on STS-B sentences, self-cosine (anisotropy) and effective rank track capability: roberta-base sits at the bottom at +.9594 self-cosine / effective rank 19.8, all-MiniLM-L6-v2 at the top at +.0251 / 86.7, and the adapted pair between them at +.0964 / 57.6 and +.0984 / 55.5. No training stack anywhere in the table contains an isotropy objective. Isotropy is the property that moved, unasked.

## 4. The AMOE adapter library

Each backbone carries a library of three **anchors** — adapters trained for equivalence, simplification, and paraphrase — dispatched by a graded router (sinh/cosh gating) over the **frozen** backbone. Anchors toggle bit-exactly (asserted at load, never assumed), so a single artifact serves as both the bare baseline and the adapted model.

### Collective and per-arm numbers

- **v2 + arms: .7287** eight-task mean. **v2-B + its own native arms: .7294**. The two are statistically tied (collective-level seed spread .003–.005); the tiebreak is provenance, and v2-B is the complete-corpus backbone with no asterisk — **the best artifact in the family**.
- **Eight for eight:** the 3-arm collective beats every one of its own members on every task. The sharpest column is BIOSSES: the equivalence anchor alone reads .5845, *below* the bare backbone's .5933, while the collective reads .6382 — the mixture rescues a task where its strongest member is a liability.
- **Geometry moves with capability:** the arms lift effective rank 36.6 → 57.6 (v2) and 36.1 → 55.5 (v2-B), and cut self-cosine +.1396 → +.0964 — the first evidence in this line that adaptation *adds usable directions* to the representation rather than only rotating existing ones.

### The capacity control

In greedy roster selection, a completely **random (untrained) arm gains +.0045 by itself** — so every candidate is judged against random-at-same-size rather than against the bare backbone. Under that bar:

- **Paraphrase** cleared random by +.0146 and joined the roster.
- **Lexical (WordNet)** and **topical (Specter)** candidates scored *below* random and were rejected.
- Every 4th-arm candidate — including a second random arm — scored below the 3-arm collective, so selection halted at exactly the arm count the conserved-amplitude arithmetic predicted (measured amplitude shares .4987 / .1944 / .1810, sum .874).

### A masked member is a damped member

Masking an anchor leaves the dispatch budget unrenormalized. The simplification anchor reads .7400 trained-and-run alone, and .6104 when read masked inside the collective. Mixture margins must therefore be quoted against solo runs, never against damped masked rows — an earlier +.0213 margin claim was corrected to +.0124 on exactly this.

### Sharper routing is worse

Routing temperature .10 / .05 / .02 gives seed-mean STS-B .7506 / .7492 / .7441 (2 seeds, monotone) while routing separation *rises* — the router committed harder and lost. Graded blending carries the value of the dispatch; hard selection destroys it.

## 5. Anchors are backbone-bound (the transplant study)

The two backbones are indistinguishable on every gauge available: eight STS tasks within ±.0042 (mean delta +.0009 excluding the 100-row BIOSSES), effective rank 36.6 vs 36.1, self-cosine +.1396 vs +.1411. Yet:

- v2's arm collective attached to v2-B **unchanged** falls .7287 → **.6863** — 68.8% of the arm gain retained, a 31% loss with no error and no warning. The collective drops *more* than its worst member (−.0425 vs −.0330): the dispatch compounds the mismatch.
- **The recovery arc, with preregistered pass bars:** re-aligning only the 1,536 routing keys (800 steps, anchors frozen) recovers 29% (.6987); retraining the three anchors on v2-B (1,500 steps each) recovers 100% (**.7294**, above v2's own .7287). Decomposition: **71% of the loss lives in the anchors (.0307), 29% in the routing keys (.0124)**.
- **The mechanism:** adapters read the *residual stream*; task gauges read the *pooled output*. The stream carries backbone identity that the pooled output does not — which makes 1.6M adapter parameters a more sensitive instrument for backbone identity than the entire 8-task suite.
- Three hypotheses died in the retrain, all refuted by the same result: "v2-B is less adaptable" (it reaches .7294), "v2-B is over-specialized to captions" (the self-cosine signal was noise), and "v2-B is worse out of domain" (BIOSSES *inverts* to .6472 on v2-B with native anchors vs .6382 on v2 — the deficit was an artifact of foreign anchors).
- **Operational cost of doing it right:** one native anchor set per backbone, ~18 minutes of training.

## 6. Laws established by this family

Each law below carries the measurement that earned it.

### Measurement discipline

Established on a retired predecessor audit in this program; enforced throughout this family.

1. **A contrastive objective whose two views share one forward pass is underdetermined.** The cheapest tell: retrieval scores invariant in gallery size.
2. **No retrieval number is quoted until the query set is shown to vary** — length histogram, pooled-query effective rank, pairwise cosine — checked before the first arm was trained.
3. **An alignment map is never scored on the rows it was fit to.** At N/d = 4.9 the predecessor retained 3.8% of in-sample R@1 out of sample; at N/d = 31.3 this family retained 95%.
4. **Effective rank is always reported beside CV.** CV is dimension-dependent; a CV target is implicitly a claim about effective dimension.
5. **A control must be able to fail before its null is citable.**
6. **The round-trip law:** a diagnostic of the form f(x, g⁻¹(g(x))) measures the operator, never the input; every diagnostic feature needs data on both sides.
7. **Mimicry and capability are separate gauges:** student→target retrieval and task scores live on separate lines with separate names.
8. **Inspect the input distribution before designing the experiment.** Validity picks the split; convenience gets no vote.

### Dispatch laws

Measured on the adapter campaign.

9. **The amplitude budget is conserved.** Arms split a fixed sum — measured shares .4987 / .1944 / .1810, sum .874, as predicted — so arm count is an empirical outcome, never a design choice.
10. **Sharper routing is worse** (2 seeds, monotone): temperature .10/.05/.02 → STS-B .7506/.7492/.7441. Graded blending carries the value.
11. **The amplitude decomposition runs before roster selection** — it is a pre-flight instrument.
12. **The one-task exemption:** cross-task destruction by always-on expert stacks is absent where every arm serves one task; the exemption is scoped strictly to that condition.
13. **The capacity control is the baseline:** a random arm at the same size (+.0045 by itself) is the correct comparator for any roster claim.
14. **A masked member is a damped member:** masking leaves the dispatch budget unrenormalized (simplification: .7400 solo vs .6104 masked; a +.0213 margin corrected to +.0124).

### Transfer and reliability laws

Established on the v2-B leg.

15. **Anchors are backbone-bound:** 71% of transplant loss lives in the anchors, 29% in the routing keys; one native anchor set per backbone costs ~18 minutes, and a hardcoded adapter repo in a loader is a silent 31% loss with no error.
16. **A completed checkpoint is unresumable into a new corpus:** epoch == epochs makes the training loop an empty range that reprints the previous run's metrics as its own.
17. **Skipping a pipeline stage means fetching its outputs:** a refit alignment map lands in a different frame with no signal in the loss.
18. **A benchmark must be independent of the artifact it measures:** the harness had briefly resolved the adapters through the model's own remote loader before this was caught.
19. **A distribution shift is judged on p99 and max, never the mean:** a .906 mean-length ratio fired a false alarm while p99 matched to 1% and the repaired maxima were shorter.

## 7. Known limits and honest caveats

### The consensus-rank ceiling

The consensus **target** itself has effective rank **28.7 of 768** (26.9–29.6 across 56 chunks): five BERT-family teachers agree on only ~29 directions. The **student** builds 102.9 effective directions in-domain (rank grew monotonically through training and never turned down), yet only 33.4–36.6 of them survive on STS-B sentences — the ~74 caption-built directions stay in-domain. For scale: all-MiniLM-L6-v2 uses 86.7–94.3 effective directions on the same sentences at 2.6× fewer parameters. The limit is set by **teacher agreement**, and the student's in-domain rank shows capacity to spare; the named next lever is *heterogeneous* teachers, and it is measurable at consensus-build time, before any training is spent.

One caveat inside this story: the out-of-domain effective rank of 36.6 sits within ~4 points of bert-base's 32.8, so part of that column may be a property of the evaluation data itself. The clean control — bert-base embedded on the CC12M captions — was never run and is the top open item.

### The data-volume null

19% more rows for 19% more steps — the complete repaired corpus versus the 54-chunk subset — moved nothing: 8-task mean .6031 vs .6077, where the entire −.0046 is BIOSSES; excluding it, **+.0009**. Training metrics are equally flat (target cosine .8421 vs .8410, effective rank 103.1 vs 102.9). Under this recipe, more of the same distribution leaves a ~29-direction target where it is. (Corpus provenance, for completeness: ModernBERT embeddings had been missing from 10 of 66 chunks; the repair was gate-verified at cosine 1.00000, minimum 1.00000, against a re-embed, and the "long captions broke it" hypothesis was refuted — p99 caption lengths were identical across repaired and never-missing chunks, and the repaired maxima were *shorter*.)

### Statistical and provenance caveats

- The v2-B backbone is single-seed. The arm and temperature sweeps carry 2 seeds; collective-level seed spread is .003–.005.
- The .7287 / .7294 comparison is between each line's own shipped checkpoint selection: v2 ships its best checkpoint, v2-B its final. The two v2-B checkpoints differ materially (max weight delta 1.319e-01). It is the right comparison — each number belongs to its line's shipped artifact — and it is stated here so the reader can weigh it.
- BIOSSES is 100 rows with a Spearman standard error of ~0.10; every BIOSSES delta in this document is sub-sigma except where stated.
- The supervised reference all-MiniLM-L6-v2 (.7925, trained on 1B+ curated pairs) sits .063 above the adapted pair; that gap is the concrete distance this recipe has left to close.

## References and attribution

**Models**

- [BERT](https://arxiv.org/abs/1810.04805) (Devlin et al.) — teacher and fixed reference member (bert-base); the coordinate frame every alignment map targets.
- [DistilBERT](https://arxiv.org/abs/1910.01108) (Sanh et al.) — teacher; the best single teacher on the eight-task suite.
- [RoBERTa](https://arxiv.org/abs/1907.11692) (Liu et al.) — teacher.
- [ALBERT](https://arxiv.org/abs/1909.11942) (Lan et al.) — teacher.
- [ModernBERT](https://arxiv.org/abs/2412.13663) (Warner et al.) — teacher.
- [MiniLM](https://arxiv.org/abs/2002.10957) (Wang et al.) via [Sentence-BERT](https://arxiv.org/abs/1908.10084) (Reimers & Gurevych) — the supervised reference all-MiniLM-L6-v2.

**Data**

- [CC12M](https://arxiv.org/abs/2102.08981) (Changpinyo et al.), obtained via [pixparse/cc12m-wds](https://huggingface.co/datasets/pixparse/cc12m-wds) — source of the caption corpus.
- [MTEB](https://arxiv.org/abs/2210.07316) (Muennighoff et al.) — the STS-B / SICK-R / STS12–16 / BIOSSES evaluation suites.

**Methods**

- [InfoNCE](https://arxiv.org/abs/1807.03748) (van den Oord et al.) — the contrastive term of the training loss.
- [Orthogonal Procrustes](https://doi.org/10.1007/BF02291478) (Schönemann, 1966) — the whitened alignment maps into the reference frame.
- [Knowledge distillation](https://arxiv.org/abs/1503.02531) (Hinton et al.) — the lineage of the consensus-distillation objective.
- [LoRA](https://arxiv.org/abs/2106.09685) (Hu et al.) — the lineage of the zero-initialized adapter arms.
- [Mixture-of-Experts](https://arxiv.org/abs/1701.06538) (Shazeer et al.) — the dispatch lineage behind the AMOE adapter library.
- [Transformer](https://arxiv.org/abs/1706.03762) (Vaswani et al.) — the backbone architecture.

**Infrastructure**

- [PyTorch](https://pytorch.org) — training and evaluation stack.
- [RunPod](https://runpod.io) and [Google Colab](https://colab.research.google.com) — compute environments.

**Related artifacts:** [captionbert-8192-v2](https://huggingface.co/AbstractPhil/captionbert-8192-v2) · [captionbert-8192-v2-B](https://huggingface.co/AbstractPhil/captionbert-8192-v2-B) · [captionbert-8192-v2-consensus](https://huggingface.co/datasets/AbstractPhil/captionbert-8192-v2-consensus) · [conceptual-captions-12m-webdataset-berts](https://huggingface.co/datasets/AbstractPhil/conceptual-captions-12m-webdataset-berts) · prior installment: [geometric-memory-ft4](https://huggingface.co/blog/AbstractPhil/geometric-memory-ft4)

---

*Trained and evaluated in 2026 by AbstractPhil. All artifacts — both backbones, their adapter libraries, the consensus dataset, and the caption corpus — are at [huggingface.co/AbstractPhil](https://huggingface.co/AbstractPhil).*