AbstractPhil commited on
Commit
bb5440a
Β·
verified Β·
1 Parent(s): 1ad21ba

references+attribution section, inline citations

Browse files
Files changed (1) hide show
  1. TECHNICAL.md +37 -5
TECHNICAL.md CHANGED
@@ -7,7 +7,7 @@
7
 
8
  ## 1. Overview
9
 
10
- captionbert-8192-v2 and captionbert-8192-v2-B are sibling 58.3M-parameter text encoders trained from scratch by **consensus distillation**: the training target is the *agreement* of five BERT-family teachers (bert-base, distilbert, roberta-base, albert-base-v2, ModernBERT-base), computed over 27–32 million image captions, and neither model ever saw a similarity label. Each ships with an AMOE adapter library β€” three task-anchored adapters dispatched by a graded router over the frozen backbone β€” that lifts the eight-task benchmark mean from .6077/.6031 to .7287/**.7294**, past every teacher. This document records the full recipe, the ten-model benchmark run under one shared harness, the adapter-library findings including a transplant/portability study, and the measured laws and limits the family established.
11
 
12
  ## 2. Training recipe
13
 
@@ -23,11 +23,11 @@ captionbert-8192-v2 and captionbert-8192-v2-B are sibling 58.3M-parameter text e
23
 
24
  ### The consensus target
25
 
26
- Each of the five teachers embeds the CC12M LLaVA-NeXT captions. Each teacher then receives **one global whitened-Procrustes map** into bert-base's coordinate frame β€” bert-base serves as a fixed reference *member* of the ensemble. The consensus vector for a caption is the normalized centroid of the five aligned embeddings. The student regresses onto these precomputed vectors and contrasts against them; the similarity structure it learns is entirely implicit in teacher agreement.
27
 
28
  ### Objective and optimization
29
 
30
- - **Loss:** InfoNCE (temperature 0.07) + per-sample MSE against the precomputed consensus vector.
31
  - **Optimizer:** Adam, weight decay 0.
32
  - **Schedule:** lr 6e-4, cosine decay, batch 2048.
33
 
@@ -51,7 +51,7 @@ The maps also recovered a lineage nobody supplied: out-of-sample alignment cosin
51
 
52
  ### Reference-member alignment pins the frame
53
 
54
- Because the consensus is expressed in a fixed member's coordinates, the gap between the student's frame and the target frame collapsed monotonically toward zero during training (+.0235 β†’ +.0070 β†’ +.0015 β†’ +.0005 β†’ +.0010 final). This line therefore ships with **no rotation matrix**. The vision-side consensus line of the same research program aligns to a running mean β€” a frame-free target β€” and must ship a 0.5MB rotation. The same construction, measured on both sides, yields opposite deployment requirements.
55
 
56
  ## 3. Results
57
 
@@ -80,7 +80,7 @@ All models are mean-pooled and L2-normalized, with no task tuning, run through o
80
  | captionbert-8192-v2-B | 58.3M | .5752 | .6548 | .5012 | .6037 | .5470 | .7146 | .6782 | .5500 | **.6031** |
81
  | v2 + arms | 63.2M | .7684 | .7391 | .6682 | .7557 | .6921 | .8055 | .7626 | .6382 | **.7287** |
82
  | v2-B + arms (native) | 63.2M | .7675 | .7374 | .6706 | .7381 | .6945 | .8109 | .7695 | .6472 | **.7294** |
83
- | all-MiniLM-L6-v2 | 22.7M | .8203 | .7758 | .7237 | .8058 | .7559 | .8539 | .7899 | .8144 | .7925 |
84
 
85
  Reading the table row by row:
86
 
@@ -196,6 +196,38 @@ One caveat inside this story: the out-of-domain effective rank of 36.6 sits with
196
  - BIOSSES is 100 rows with a Spearman standard error of ~0.10; every BIOSSES delta in this document is sub-sigma except where stated.
197
  - The supervised reference all-MiniLM-L6-v2 (.7925, trained on 1B+ curated pairs) sits .063 above the adapted pair; that gap is the concrete distance this recipe has left to close.
198
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
199
  ---
200
 
201
  *Trained and evaluated in 2026 by AbstractPhil. All artifacts β€” both backbones, their adapter libraries, the consensus dataset, and the caption corpus β€” are at [huggingface.co/AbstractPhil](https://huggingface.co/AbstractPhil).*
 
7
 
8
  ## 1. Overview
9
 
10
+ captionbert-8192-v2 and captionbert-8192-v2-B are sibling 58.3M-parameter text encoders trained from scratch by **consensus [distillation](https://arxiv.org/abs/1503.02531)**: the training target is the *agreement* of five BERT-family teachers ([bert-base](https://arxiv.org/abs/1810.04805), [distilbert](https://arxiv.org/abs/1910.01108), [roberta-base](https://arxiv.org/abs/1907.11692), [albert-base-v2](https://arxiv.org/abs/1909.11942), [ModernBERT-base](https://arxiv.org/abs/2412.13663)), computed over 27–32 million image captions, and neither model ever saw a similarity label. Each ships with an AMOE adapter library β€” three task-anchored adapters dispatched by a graded router over the frozen backbone β€” that lifts the eight-task benchmark mean from .6077/.6031 to .7287/**.7294**, past every teacher. This document records the full recipe, the ten-model benchmark run under one shared harness, the adapter-library findings including a transplant/portability study, and the measured laws and limits the family established.
11
 
12
  ## 2. Training recipe
13
 
 
23
 
24
  ### The consensus target
25
 
26
+ Each of the five teachers embeds the [CC12M](https://arxiv.org/abs/2102.08981) LLaVA-NeXT captions. Each teacher then receives **one global whitened-[Procrustes](https://doi.org/10.1007/BF02291478) map** into bert-base's coordinate frame β€” bert-base serves as a fixed reference *member* of the ensemble. The consensus vector for a caption is the normalized centroid of the five aligned embeddings. The student regresses onto these precomputed vectors and contrasts against them; the similarity structure it learns is entirely implicit in teacher agreement.
27
 
28
  ### Objective and optimization
29
 
30
+ - **Loss:** [InfoNCE](https://arxiv.org/abs/1807.03748) (temperature 0.07) + per-sample MSE against the precomputed consensus vector.
31
  - **Optimizer:** Adam, weight decay 0.
32
  - **Schedule:** lr 6e-4, cosine decay, batch 2048.
33
 
 
51
 
52
  ### Reference-member alignment pins the frame
53
 
54
+ Because the consensus is expressed in a fixed member's coordinates, the gap between the student's frame and the target frame collapsed monotonically toward zero during training (+.0235 β†’ +.0070 β†’ +.0015 β†’ +.0005 β†’ +.0010 final). This line therefore ships with **no rotation matrix**. The [vision-side consensus line](https://huggingface.co/blog/AbstractPhil/geometric-memory-ft4) of the same research program aligns to a running mean β€” a frame-free target β€” and must ship a 0.5MB rotation. The same construction, measured on both sides, yields opposite deployment requirements.
55
 
56
  ## 3. Results
57
 
 
80
  | captionbert-8192-v2-B | 58.3M | .5752 | .6548 | .5012 | .6037 | .5470 | .7146 | .6782 | .5500 | **.6031** |
81
  | v2 + arms | 63.2M | .7684 | .7391 | .6682 | .7557 | .6921 | .8055 | .7626 | .6382 | **.7287** |
82
  | v2-B + arms (native) | 63.2M | .7675 | .7374 | .6706 | .7381 | .6945 | .8109 | .7695 | .6472 | **.7294** |
83
+ | [all-MiniLM-L6-v2](https://arxiv.org/abs/2002.10957) | 22.7M | .8203 | .7758 | .7237 | .8058 | .7559 | .8539 | .7899 | .8144 | .7925 |
84
 
85
  Reading the table row by row:
86
 
 
196
  - BIOSSES is 100 rows with a Spearman standard error of ~0.10; every BIOSSES delta in this document is sub-sigma except where stated.
197
  - The supervised reference all-MiniLM-L6-v2 (.7925, trained on 1B+ curated pairs) sits .063 above the adapted pair; that gap is the concrete distance this recipe has left to close.
198
 
199
+ ## References and attribution
200
+
201
+ **Models**
202
+
203
+ - [BERT](https://arxiv.org/abs/1810.04805) (Devlin et al.) β€” teacher and fixed reference member (bert-base); the coordinate frame every alignment map targets.
204
+ - [DistilBERT](https://arxiv.org/abs/1910.01108) (Sanh et al.) β€” teacher; the best single teacher on the eight-task suite.
205
+ - [RoBERTa](https://arxiv.org/abs/1907.11692) (Liu et al.) β€” teacher.
206
+ - [ALBERT](https://arxiv.org/abs/1909.11942) (Lan et al.) β€” teacher.
207
+ - [ModernBERT](https://arxiv.org/abs/2412.13663) (Warner et al.) β€” teacher.
208
+ - [MiniLM](https://arxiv.org/abs/2002.10957) (Wang et al.) via [Sentence-BERT](https://arxiv.org/abs/1908.10084) (Reimers & Gurevych) β€” the supervised reference all-MiniLM-L6-v2.
209
+
210
+ **Data**
211
+
212
+ - [CC12M](https://arxiv.org/abs/2102.08981) (Changpinyo et al.), obtained via [pixparse/cc12m-wds](https://huggingface.co/datasets/pixparse/cc12m-wds) β€” source of the caption corpus.
213
+ - [MTEB](https://arxiv.org/abs/2210.07316) (Muennighoff et al.) β€” the STS-B / SICK-R / STS12–16 / BIOSSES evaluation suites.
214
+
215
+ **Methods**
216
+
217
+ - [InfoNCE](https://arxiv.org/abs/1807.03748) (van den Oord et al.) β€” the contrastive term of the training loss.
218
+ - [Orthogonal Procrustes](https://doi.org/10.1007/BF02291478) (SchΓΆnemann, 1966) β€” the whitened alignment maps into the reference frame.
219
+ - [Knowledge distillation](https://arxiv.org/abs/1503.02531) (Hinton et al.) β€” the lineage of the consensus-distillation objective.
220
+ - [LoRA](https://arxiv.org/abs/2106.09685) (Hu et al.) β€” the lineage of the zero-initialized adapter arms.
221
+ - [Mixture-of-Experts](https://arxiv.org/abs/1701.06538) (Shazeer et al.) β€” the dispatch lineage behind the AMOE adapter library.
222
+ - [Transformer](https://arxiv.org/abs/1706.03762) (Vaswani et al.) β€” the backbone architecture.
223
+
224
+ **Infrastructure**
225
+
226
+ - [PyTorch](https://pytorch.org) β€” training and evaluation stack.
227
+ - [RunPod](https://runpod.io) and [Google Colab](https://colab.research.google.com) β€” compute environments.
228
+
229
+ **Related artifacts:** [captionbert-8192-v2](https://huggingface.co/AbstractPhil/captionbert-8192-v2) Β· [captionbert-8192-v2-B](https://huggingface.co/AbstractPhil/captionbert-8192-v2-B) Β· [captionbert-8192-v2-consensus](https://huggingface.co/datasets/AbstractPhil/captionbert-8192-v2-consensus) Β· [conceptual-captions-12m-webdataset-berts](https://huggingface.co/datasets/AbstractPhil/conceptual-captions-12m-webdataset-berts) Β· prior installment: [geometric-memory-ft4](https://huggingface.co/blog/AbstractPhil/geometric-memory-ft4)
230
+
231
  ---
232
 
233
  *Trained and evaluated in 2026 by AbstractPhil. All artifacts β€” both backbones, their adapter libraries, the consensus dataset, and the caption corpus β€” are at [huggingface.co/AbstractPhil](https://huggingface.co/AbstractPhil).*