Feature Extraction
Transformers
TensorBoard
Safetensors
English
captionbert_v2
sentence-similarity
consensus-distillation
geometric-deep-learning
amoe
custom_code
Instructions to use AbstractPhil/captionbert-8192-v2 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use AbstractPhil/captionbert-8192-v2 with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("feature-extraction", model="AbstractPhil/captionbert-8192-v2", trust_remote_code=True)# pip install -U transformers accelerate # Load model directly from transformers import AutoModel model = AutoModel.from_pretrained("AbstractPhil/captionbert-8192-v2", trust_remote_code=True, device_map="auto") - Notebooks
- Google Colab
- Kaggle
references+attribution section, inline citations
Browse files- TECHNICAL.md +37 -5
TECHNICAL.md
CHANGED
|
@@ -7,7 +7,7 @@
|
|
| 7 |
|
| 8 |
## 1. Overview
|
| 9 |
|
| 10 |
-
captionbert-8192-v2 and captionbert-8192-v2-B are sibling 58.3M-parameter text encoders trained from scratch by **consensus distillation**: the training target is the *agreement* of five BERT-family teachers (bert-base, distilbert, roberta-base, albert-base-v2, ModernBERT-base), computed over 27β32 million image captions, and neither model ever saw a similarity label. Each ships with an AMOE adapter library β three task-anchored adapters dispatched by a graded router over the frozen backbone β that lifts the eight-task benchmark mean from .6077/.6031 to .7287/**.7294**, past every teacher. This document records the full recipe, the ten-model benchmark run under one shared harness, the adapter-library findings including a transplant/portability study, and the measured laws and limits the family established.
|
| 11 |
|
| 12 |
## 2. Training recipe
|
| 13 |
|
|
@@ -23,11 +23,11 @@ captionbert-8192-v2 and captionbert-8192-v2-B are sibling 58.3M-parameter text e
|
|
| 23 |
|
| 24 |
### The consensus target
|
| 25 |
|
| 26 |
-
Each of the five teachers embeds the CC12M LLaVA-NeXT captions. Each teacher then receives **one global whitened-Procrustes map** into bert-base's coordinate frame β bert-base serves as a fixed reference *member* of the ensemble. The consensus vector for a caption is the normalized centroid of the five aligned embeddings. The student regresses onto these precomputed vectors and contrasts against them; the similarity structure it learns is entirely implicit in teacher agreement.
|
| 27 |
|
| 28 |
### Objective and optimization
|
| 29 |
|
| 30 |
-
- **Loss:** InfoNCE (temperature 0.07) + per-sample MSE against the precomputed consensus vector.
|
| 31 |
- **Optimizer:** Adam, weight decay 0.
|
| 32 |
- **Schedule:** lr 6e-4, cosine decay, batch 2048.
|
| 33 |
|
|
@@ -51,7 +51,7 @@ The maps also recovered a lineage nobody supplied: out-of-sample alignment cosin
|
|
| 51 |
|
| 52 |
### Reference-member alignment pins the frame
|
| 53 |
|
| 54 |
-
Because the consensus is expressed in a fixed member's coordinates, the gap between the student's frame and the target frame collapsed monotonically toward zero during training (+.0235 β +.0070 β +.0015 β +.0005 β +.0010 final). This line therefore ships with **no rotation matrix**. The vision-side consensus line of the same research program aligns to a running mean β a frame-free target β and must ship a 0.5MB rotation. The same construction, measured on both sides, yields opposite deployment requirements.
|
| 55 |
|
| 56 |
## 3. Results
|
| 57 |
|
|
@@ -80,7 +80,7 @@ All models are mean-pooled and L2-normalized, with no task tuning, run through o
|
|
| 80 |
| captionbert-8192-v2-B | 58.3M | .5752 | .6548 | .5012 | .6037 | .5470 | .7146 | .6782 | .5500 | **.6031** |
|
| 81 |
| v2 + arms | 63.2M | .7684 | .7391 | .6682 | .7557 | .6921 | .8055 | .7626 | .6382 | **.7287** |
|
| 82 |
| v2-B + arms (native) | 63.2M | .7675 | .7374 | .6706 | .7381 | .6945 | .8109 | .7695 | .6472 | **.7294** |
|
| 83 |
-
| all-MiniLM-L6-v2 | 22.7M | .8203 | .7758 | .7237 | .8058 | .7559 | .8539 | .7899 | .8144 | .7925 |
|
| 84 |
|
| 85 |
Reading the table row by row:
|
| 86 |
|
|
@@ -196,6 +196,38 @@ One caveat inside this story: the out-of-domain effective rank of 36.6 sits with
|
|
| 196 |
- BIOSSES is 100 rows with a Spearman standard error of ~0.10; every BIOSSES delta in this document is sub-sigma except where stated.
|
| 197 |
- The supervised reference all-MiniLM-L6-v2 (.7925, trained on 1B+ curated pairs) sits .063 above the adapted pair; that gap is the concrete distance this recipe has left to close.
|
| 198 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 199 |
---
|
| 200 |
|
| 201 |
*Trained and evaluated in 2026 by AbstractPhil. All artifacts β both backbones, their adapter libraries, the consensus dataset, and the caption corpus β are at [huggingface.co/AbstractPhil](https://huggingface.co/AbstractPhil).*
|
|
|
|
| 7 |
|
| 8 |
## 1. Overview
|
| 9 |
|
| 10 |
+
captionbert-8192-v2 and captionbert-8192-v2-B are sibling 58.3M-parameter text encoders trained from scratch by **consensus [distillation](https://arxiv.org/abs/1503.02531)**: the training target is the *agreement* of five BERT-family teachers ([bert-base](https://arxiv.org/abs/1810.04805), [distilbert](https://arxiv.org/abs/1910.01108), [roberta-base](https://arxiv.org/abs/1907.11692), [albert-base-v2](https://arxiv.org/abs/1909.11942), [ModernBERT-base](https://arxiv.org/abs/2412.13663)), computed over 27β32 million image captions, and neither model ever saw a similarity label. Each ships with an AMOE adapter library β three task-anchored adapters dispatched by a graded router over the frozen backbone β that lifts the eight-task benchmark mean from .6077/.6031 to .7287/**.7294**, past every teacher. This document records the full recipe, the ten-model benchmark run under one shared harness, the adapter-library findings including a transplant/portability study, and the measured laws and limits the family established.
|
| 11 |
|
| 12 |
## 2. Training recipe
|
| 13 |
|
|
|
|
| 23 |
|
| 24 |
### The consensus target
|
| 25 |
|
| 26 |
+
Each of the five teachers embeds the [CC12M](https://arxiv.org/abs/2102.08981) LLaVA-NeXT captions. Each teacher then receives **one global whitened-[Procrustes](https://doi.org/10.1007/BF02291478) map** into bert-base's coordinate frame β bert-base serves as a fixed reference *member* of the ensemble. The consensus vector for a caption is the normalized centroid of the five aligned embeddings. The student regresses onto these precomputed vectors and contrasts against them; the similarity structure it learns is entirely implicit in teacher agreement.
|
| 27 |
|
| 28 |
### Objective and optimization
|
| 29 |
|
| 30 |
+
- **Loss:** [InfoNCE](https://arxiv.org/abs/1807.03748) (temperature 0.07) + per-sample MSE against the precomputed consensus vector.
|
| 31 |
- **Optimizer:** Adam, weight decay 0.
|
| 32 |
- **Schedule:** lr 6e-4, cosine decay, batch 2048.
|
| 33 |
|
|
|
|
| 51 |
|
| 52 |
### Reference-member alignment pins the frame
|
| 53 |
|
| 54 |
+
Because the consensus is expressed in a fixed member's coordinates, the gap between the student's frame and the target frame collapsed monotonically toward zero during training (+.0235 β +.0070 β +.0015 β +.0005 β +.0010 final). This line therefore ships with **no rotation matrix**. The [vision-side consensus line](https://huggingface.co/blog/AbstractPhil/geometric-memory-ft4) of the same research program aligns to a running mean β a frame-free target β and must ship a 0.5MB rotation. The same construction, measured on both sides, yields opposite deployment requirements.
|
| 55 |
|
| 56 |
## 3. Results
|
| 57 |
|
|
|
|
| 80 |
| captionbert-8192-v2-B | 58.3M | .5752 | .6548 | .5012 | .6037 | .5470 | .7146 | .6782 | .5500 | **.6031** |
|
| 81 |
| v2 + arms | 63.2M | .7684 | .7391 | .6682 | .7557 | .6921 | .8055 | .7626 | .6382 | **.7287** |
|
| 82 |
| v2-B + arms (native) | 63.2M | .7675 | .7374 | .6706 | .7381 | .6945 | .8109 | .7695 | .6472 | **.7294** |
|
| 83 |
+
| [all-MiniLM-L6-v2](https://arxiv.org/abs/2002.10957) | 22.7M | .8203 | .7758 | .7237 | .8058 | .7559 | .8539 | .7899 | .8144 | .7925 |
|
| 84 |
|
| 85 |
Reading the table row by row:
|
| 86 |
|
|
|
|
| 196 |
- BIOSSES is 100 rows with a Spearman standard error of ~0.10; every BIOSSES delta in this document is sub-sigma except where stated.
|
| 197 |
- The supervised reference all-MiniLM-L6-v2 (.7925, trained on 1B+ curated pairs) sits .063 above the adapted pair; that gap is the concrete distance this recipe has left to close.
|
| 198 |
|
| 199 |
+
## References and attribution
|
| 200 |
+
|
| 201 |
+
**Models**
|
| 202 |
+
|
| 203 |
+
- [BERT](https://arxiv.org/abs/1810.04805) (Devlin et al.) β teacher and fixed reference member (bert-base); the coordinate frame every alignment map targets.
|
| 204 |
+
- [DistilBERT](https://arxiv.org/abs/1910.01108) (Sanh et al.) β teacher; the best single teacher on the eight-task suite.
|
| 205 |
+
- [RoBERTa](https://arxiv.org/abs/1907.11692) (Liu et al.) β teacher.
|
| 206 |
+
- [ALBERT](https://arxiv.org/abs/1909.11942) (Lan et al.) β teacher.
|
| 207 |
+
- [ModernBERT](https://arxiv.org/abs/2412.13663) (Warner et al.) β teacher.
|
| 208 |
+
- [MiniLM](https://arxiv.org/abs/2002.10957) (Wang et al.) via [Sentence-BERT](https://arxiv.org/abs/1908.10084) (Reimers & Gurevych) β the supervised reference all-MiniLM-L6-v2.
|
| 209 |
+
|
| 210 |
+
**Data**
|
| 211 |
+
|
| 212 |
+
- [CC12M](https://arxiv.org/abs/2102.08981) (Changpinyo et al.), obtained via [pixparse/cc12m-wds](https://huggingface.co/datasets/pixparse/cc12m-wds) β source of the caption corpus.
|
| 213 |
+
- [MTEB](https://arxiv.org/abs/2210.07316) (Muennighoff et al.) β the STS-B / SICK-R / STS12β16 / BIOSSES evaluation suites.
|
| 214 |
+
|
| 215 |
+
**Methods**
|
| 216 |
+
|
| 217 |
+
- [InfoNCE](https://arxiv.org/abs/1807.03748) (van den Oord et al.) β the contrastive term of the training loss.
|
| 218 |
+
- [Orthogonal Procrustes](https://doi.org/10.1007/BF02291478) (SchΓΆnemann, 1966) β the whitened alignment maps into the reference frame.
|
| 219 |
+
- [Knowledge distillation](https://arxiv.org/abs/1503.02531) (Hinton et al.) β the lineage of the consensus-distillation objective.
|
| 220 |
+
- [LoRA](https://arxiv.org/abs/2106.09685) (Hu et al.) β the lineage of the zero-initialized adapter arms.
|
| 221 |
+
- [Mixture-of-Experts](https://arxiv.org/abs/1701.06538) (Shazeer et al.) β the dispatch lineage behind the AMOE adapter library.
|
| 222 |
+
- [Transformer](https://arxiv.org/abs/1706.03762) (Vaswani et al.) β the backbone architecture.
|
| 223 |
+
|
| 224 |
+
**Infrastructure**
|
| 225 |
+
|
| 226 |
+
- [PyTorch](https://pytorch.org) β training and evaluation stack.
|
| 227 |
+
- [RunPod](https://runpod.io) and [Google Colab](https://colab.research.google.com) β compute environments.
|
| 228 |
+
|
| 229 |
+
**Related artifacts:** [captionbert-8192-v2](https://huggingface.co/AbstractPhil/captionbert-8192-v2) Β· [captionbert-8192-v2-B](https://huggingface.co/AbstractPhil/captionbert-8192-v2-B) Β· [captionbert-8192-v2-consensus](https://huggingface.co/datasets/AbstractPhil/captionbert-8192-v2-consensus) Β· [conceptual-captions-12m-webdataset-berts](https://huggingface.co/datasets/AbstractPhil/conceptual-captions-12m-webdataset-berts) Β· prior installment: [geometric-memory-ft4](https://huggingface.co/blog/AbstractPhil/geometric-memory-ft4)
|
| 230 |
+
|
| 231 |
---
|
| 232 |
|
| 233 |
*Trained and evaluated in 2026 by AbstractPhil. All artifacts β both backbones, their adapter libraries, the consensus dataset, and the caption corpus β are at [huggingface.co/AbstractPhil](https://huggingface.co/AbstractPhil).*
|