captionbert-8192-v2

A 58.3M standalone sentence encoder distilled from the geometric consensus of five BERT-family teachers. No expert models at inference: tokenizer + this model, 768-d L2-normalized output.

12 layers, 512-d, 8 heads, FFN 2048, 8192 position capacity. 0.53x bert-base.

from transformers import AutoModel, AutoTokenizer
model = AutoModel.from_pretrained("AbstractPhil/captionbert-8192-v2", trust_remote_code=True)
tok   = AutoTokenizer.from_pretrained("google-bert/bert-base-uncased")

emb = model.encode(["a cat on a windowsill", "a feline by the window"])   # (2, 768)
(emb[0] @ emb[1]).item()

Results

Measured in one harness; every model mean-pooled and L2-normalized, no task tuning. erank is the participation ratio of the embedding spectrum -- how many of the 768 directions are actually used.

model	params	STS-B rho	SICK-R rho	self_cos	erank
bert-base	109.5M	.4729	.5865	+.580	32.0
ModernBERT-base	149.0M	.4215	.5479	+.948	--
roberta-base	124.6M	.5436	.6296	+.976	--
albert-base-v2	11.7M	.4784	.5364	+.905	--
distilbert	66.4M	.5717	.6424	+.840	--
captionbert-8192-v2	58.3M	.5747	.6526	+.129	33.4
all-MiniLM-L6-v2 (ref)	22.7M	.8203	.7758	+.023	94.3

It edges every teacher it was distilled from, at 13% of their combined parameters, having never seen a similarity label. The bare trunk does not reach all-MiniLM-L6-v2 (1B+ curated pairs, a different comparison class); with the 3-arm AMOE collective below, STS-B closes to .7684 vs .8203 and SICK-R to .7391 vs .7758.

Isotropy is the mechanism. Mean-pooled BERT-family embeddings sit in a narrow cone (self_cos .58-.98); this model reads +.129, and cosine discriminates far better in a space that is not collapsed.

With AMOE arms (amoe/)

The trunk is frozen; each anchor is 1.6M params across 12 sites. Anchors toggle bit-exact -- all disabled reproduces the bare trunk exactly -- so one artifact serves both the unsupervised baseline and the adapted model.

Best measured roster: equiv + simplify + paraphrase under a trained dispatch, across the full 8-task STS suite:

config	STS-B	SICK-R	STS12	STS13	STS14	STS15	STS16	BIOSSES	mean
bare trunk	.5747	.6526	.5051	.5995	.5452	.7136	.6776	.5933	.6077
best single member	.7208	.7269	.6275	.6988	.6423	.7783	.7107	.5845	.6862
3-arm collective	.7684	.7391	.6682	.7557	.6921	.8055	.7626	.6382	.7287

Eight for eight over every single member, +.1210 mean over the trunk, from 1.6M x 3 adapter params on a model that never moves. Note the BIOSSES column: the strongest member alone reads .5845, below the bare trunk -- the mixture turns a liability into a gain.

Solo anchor scores (trained alone, not masked inside a dispatch):

anchor	trained on	relation	STS-B	SICK-R
equiv	all-nli	equivalence	.7254	.7550
simplify	simple-wiki + altlex + sentence-compression	compression	.7400	.7075
paraphrase	quora-duplicates	question paraphrase	see below	see below

How many arms, and which

Arms do not add capacity -- they split a roughly conserved amplitude budget. w_k/z = sinh(u_k) / SUM_j cosh(u_j) sums over all anchors, so measured sums run .850 at A=2 to .757 at A=6 while each arm's share collapses. And sharper routing does not rescue it: tau .10/.05/.02 gave STS-B .7520 / .7514 / .7468 while key separation rose. The value is graded blending, not selection.

So every candidate arm is judged against an untrained arm at the same arm count, not against the smaller base. An empty arm gains +.0045 here -- real generic capacity. Greedy forward selection, bar = the measured seed spread .0054:

candidate (A=3)	STS-B	vs empty arm	verdict
paraphrase (quora)	.7684	+.0146	kept
untrained control	.7538	--	baseline
lexical (WordNet term/gloss)	.7518	-.0020	rejected
topical (specter citations)	.7508	-.0030	rejected

Every A=4 candidate -- including the control -- came in below .7684. Selection stopped at three arms. Zero starvation alarms; per-block routing spread .348, so the 12 dispatches learned different decisions rather than 12 copies of one.

Loading arms

Arms attach through the same AutoModel object. amoe-lora is imported lazily, so the base model still loads on a machine that has never heard of it.

from transformers import AutoModel
model = AutoModel.from_pretrained("AbstractPhil/captionbert-8192-v2",
                                  trust_remote_code=True)

model.attach_amoe()                       # the shipped 3-arm collective
emb = model.encode(["a cat on a windowsill"])

with model.amoe_off():                    # the unsupervised baseline
    base = model.encode(["a cat on a windowsill"])

model.set_amoe(["equiv"])                 # one arm, inside the dispatch
model.detach_amoe()                       # bit-exact restore, asserted

attach_amoe() with no arguments reads the roster and its order from the dispatch checkpoint -- routing keys are per-arm and positional, so order is not cosmetic. Both attach_amoe and detach_amoe assert the toggle law and raise rather than return a silently-wrong model.

pip install git+https://github.com/AbstractEyes/amoe-lora for the arm methods.

Arm library

amoe/arms/ holds one canonical copy of every anchor with ARMS.json (provenance, training spec, solo scores, capacity verdict). Resolve from there; do not retrain what is already listed. The campaign folders (amoe/{sts,sts-combo,moe,moe-v2,collective}) remain as the records behind the published numbers.

Scope worth stating: these anchors are solo-trained and always-on, which the aleph line identifies as blend regime -- the dispatch blends them rather than containing them. That is survivable here only because every arm serves one task. On a multi-task trunk, expect the cross-task interference that record documents. See amoe-lora.

How it was built

1. Five teachers embedded 33M CC12M llava-next captions (mean-pooled, 768-d).
2. One global whitened Procrustes map per teacher into bert-base's frame, fit on a stratified random sample and reported out-of-sample.
3. Consensus = normalized centroid of the aligned teachers, per chunk.
4. Student trained from scratch: InfoNCE(T=0.07) + per-sample MSE against the consensus. Pure Adam, no weight decay. 26.9M rows, 52,548 steps at batch 2048, ~5.4 h on one RTX 6000 Pro.

Known limits -- read before using

• Consensus rank is ~28.7 of 768. Five BERT-family teachers only agree on about 29 directions. The student uses ~103 in domain but falls back to ~33 on out-of-domain text: the structure it builds on captions does not transfer. This is the model's ceiling and it is a property of the consensus, not the student.
• Alignment quality varies by teacher. Out-of-sample cosine to the bert frame: distil .625, roberta .372, albert .331, modern .327. The ordering tracks architectural distance from bert-base.
• 10 of 66 source chunks lacked ModernBERT, so 54 chunks (~27M rows) were used. No 4-expert fallback: that would change the target definition mid-dataset.
• Trained on image captions. Expect caption-like text to be its strongest domain.
• The trunk is single-seed. The AMOE results are 2-seed with a measured spread of .003-.005, against margins of +.012 to +.019.
• amoe/sts-combo stopped at step 500 of a planned 4000; its card and config describe the full run. amoe/moe-v2 retrained its anchors instead of reusing them, so the moe-vs-moe-v2 comparison moved two variables, not one. Both are recorded rather than quietly corrected.

Output convention (differs from v1)

field	shape
last_hidden_state	(B, L, 512)	token states
pooler_output	(B, 768)	the embedding, L2-normalized
embedding	(B, 768)	alias

geolip-captionbert-8192 (v1) returned the pooled embedding as last_hidden_state. If porting v1 code, use pooler_output. v1 also shipped an AlignmentBank; v2 does not -- measured on v1, its expert-consistency features varied 0.2% across samples because a rotation round-trip carries no data.

Citation

@misc{abstractphil2026captionbertv2,
  title  = {captionbert-8192-v2: consensus distillation at CC12M scale},
  author = {AbstractPhil},
  year   = {2026},
  url    = {https://huggingface.co/AbstractPhil/captionbert-8192-v2}
}

MIT.