clip-vitb-mini-distilled

An 8.66M-parameter ViT image encoder (10.0% of a CLIP-B/16 image tower) producing 512-d embeddings compatible with the CLIP-B/16 LAION-2B text tower. The primary checkpoint was distilled on CC12M (10,968,539 images) against the generalized-Procrustes consensus of five CLIP teachers — never against the deployment teacher — and carries a frozen 512×512 rotation that maps its outputs into the deployment frame, where it outperforms the student distilled directly against that teacher on every task gauge, both seeds (full tables below).

Quick start (AutoModel)

import torch
from transformers import AutoModel, AutoImageProcessor

repo = "AbstractPhil/clip-vitb-mini-distilled"
model = AutoModel.from_pretrained(repo, trust_remote_code=True).eval()
proc = AutoImageProcessor.from_pretrained(repo)

px = proc(images=[img], return_tensors="pt")["pixel_values"]
emb = model.get_image_features(px)     # (1, 512), L2-normalized,
                                       # deployment (LAION-B/16) frame

The rotation is applied by default (config.apply_rotation); pass apply_rotation=False to get_image_features for the raw consensus-frame embedding. Weights are safetensors; the modeling code is in this repo (modeling_clip_mini.py, configuration_clip_mini.py).

Evaluation-path note: all published numbers were measured with the tensor-bicubic evaluation pipeline (interpolate to 160px, CLIP norm — the path implemented in loader.py and train/dist_bed.py). On CIFAR-scale (32px) inputs the shipped CLIPImageProcessor path yields 0.5545 vs 0.5708 zero-shot for the same weights, a resize-path difference that matters only for very small source images; verify against ledger numbers with the tensor path.

Repository inventory

path	content
`model.safetensors`, `config.json`, `modeling_clip_mini.py`, `configuration_clip_mini.py`, `preprocessor_config.json`	the champion (consensus composite, seed 1) in AutoModel format, rotation baked in as a buffer
`cc12m/*_t88000.pt`	all 18 CC12M finals — every arm (InfoNCE, feature-MSE, SigLIP-pairwise, affinity-KL, consensus pure-MSE, consensus composite, gradient-operator/spread/verbatim battery variants), both seeds
`cc12m/rotation_s{0,1}.pt`	the frozen deployment rotations (fp32, 512×512, fit metadata included)
`cc12m/mid/*_mid.pt`	rolling mid-training snapshots, 15 of the 18 arm-seeds (last 20k-step multiple) — the trajectory record behind the set-point and descent analyses. The three absent ones (feature-MSE s0, InfoNCE s0, affinity-KL s0) trained before mid-checkpointing was added; their finals are complete.
`coco/*_t8000.pt`	all 24 generation-one finals (COCO budget: the six matrix arms + the six battery arms, both seeds)
`student_infonce_s0.pt`	the original generation-one release, kept at its historical path
`loader.py`	dependency-free loader for the raw checkpoints (rotation-aware)
`train/`	complete training code: bed, objectives, teacher loading/conversion, CC12M data path, consensus-target builder, reproduction guide (`train/TRAINING.md`)
`ledgers/*.jsonl`	per-run ledgers, both generations (COCO rows `steps=8000`, CC12M rows `steps=88000`; `eval_only` rows carry CIFAR-100)
`ledgers/frame_check_pod_s{0,1}.json`	rotation forensics, all arms, both seeds
`ledgers/rotated_eval_*.json`	full rotated evaluations (champion + pure-MSE consensus)
`article_cc12m_distillation.md`	the campaign report

Architecture

field	value
type	ViT, CLS-token readout, linear projection head
hidden size / depth / heads	240 / 12 / 4
patch / input	16 / 160×160
projection	512-d (deployment teacher's projection space)
parameters	8,664,752 total (8,541,360 trunk + 123,392 head)
rotation buffer	512×512 orthogonal, frozen; fitted post-hoc on 2,500 COCO-val pairs (fp64 orthogonal Procrustes), maps consensus frame → LAION-B/16 frame
preprocessing	shorter edge → 182 (bicubic), center-crop 160, CLIP mean/std

Results

Headline — champion (rotated) vs the directly-distilled student, CC12M, seed 0 / seed 1:

gauge	consensus composite + rotation	InfoNCE vs deployment teacher	teacher	random floor
zero-shot CIFAR-10	.5412 / .5708	.5208 / .4979	.946	.0996
zero-shot CIFAR-100	.2210 / .2487	.1958 / .2092	.759	.0068
COCO R@1 (i→t)	.1322 / .1360	.1252 / .1178	.417	.000
COCO R@5	.3140 / .3300	.2968 / .2910	.669	.001

Champion capability: 44.7% / 48.1% of the teacher's zero-shot mean (CIFAR-10+100) at 10% of its image-tower parameters. Scaling arc across data generations: 21.2% (118k images) → 42.0/41.5% (11M, direct InfoNCE) → 44.7/48.1% (11M, consensus + rotation).

Full CC12M matrix (nine objectives × two seeds, 88k steps ≈ 2.05 epochs; agree = cosine to deployment teacher unless (own) = the arm's own target):

objective	zs C10	zs C100	R@1	R@5	agree	eff. rank
InfoNCE (deployment teacher)	.5208/.4979	.1958/.2092	.1252/.1178	.2968/.2910	.543/.542	188/188
feature MSE (mimicry)	.4429/.4826	.1568/.1656	.0760/.0716	.2074/.1996	.735/.738	154/153
SigLIP-pairwise (768-d, own tower)	.4534/.4573	.1625/.1702	.0904/.0914	.2278/.2278	.742/.743 (own)	155/154
consensus composite (NCE+MSE)	.3271/.3928	.1213/.1254	.0344/.0374	.1180/.1178	.741/.742 (own)	172/173
+ gradient operator	.3398/.3315	.1181/.1335	.0374/.0352	.1160/.1154	.742/.742 (own)	173/172
+ spread force (1e-3)	.2819/.3370	.1144/.1172	.0386/.0328	.1230/.1118	.743/.741 (own)	173/174
verbatim ancestor recipe	.3188/.3616	.1269/.1377	.0554/.0506	.1582/.1512	.531/.530	185/185
consensus pure-MSE	.2738/.2681	.0575/.0567	.0114/.0100	.0410/.0458	.814/.810 (own)	141/142
affinity (similarity-KL)	.1076/.1035	.0117/.0105	.0000/.0008	.0012/.0022	−.011/−.017	103/89

Rotation forensics (agree→deployment pre → post; zs C10 pre → post; both seeds):

arm	agree	zs
consensus composite	.304→.732 / .302→.731	.327→.541 / .393→.571
consensus pure-MSE	.297→.765 / .293→.763	.274→.434 / .268→.394
verbatim recipe	.258→.644 / .258→.642	.319→.487 / .362→.532
+ gradient operator	.302→.731 / .304→.730	.340→.517 / .332→.568
+ spread force	.304→.731 / .304→.728	.282→.485 / .337→.489
affinity	−.011→.304 / −.017→.301	.108→.138 / .104→.132
InfoNCE (control)	.543→.633 / .542→.631	.521→.473 / .498→.503

Note the pure-MSE consensus row: rotated agreement .765/.763 exceeds direct mimicry's in-frame .735/.738 — the consensus target carries the deployment teacher's geometry better than direct distillation preserves it — but without a contrastive term it cannot retrieve (rotated R@1 .046/.051). The composite's two terms are both load-bearing.

Generation one (COCO, 118k images, 8k steps), for the scale comparison: mimicry .2693/.2657 zs vs InfoNCE .2668/.2308 (mimicry won coarse zero-shot at small scale; the verdict reversed at 11M); InfoNCE retrieval .0580/.0544 vs .0248/.0262; consensus raw .1543/.1609 — its loss to single-teacher distillation was preregistered and refuted 2/2 at this scale before reversing at CC12M with the rotation. Full generation-one ledgers included.

Training

Everything needed to reproduce any checkpoint is in train/ (train/TRAINING.md is the step-by-step guide). Summary:

item	value
data (gen 2)	CC12M, 10,968,539 images via pixparse/cc12m-wds; read by tar-offset index, no extraction
targets	precomputed teacher features, bulk-cc12m-features (fp16, unnormalized; L2-normalized per batch on GPU)
champion objective	symmetric InfoNCE (t=0.07) + per-sample MSE, against the 5-teacher GPA consensus mean
consensus target	generalized Procrustes mean of clip_b16_laion2b, clip_b32_openai, clip_b32_laion2b, clip_b32_datacomp, clip_b16_openai (5 iterations, fp64 alignment SVD on a fixed 10k subsample; converged mean alignment ≈ 0.91)
optimizer	Adam 3e-4, weight decay 0, fp32, TF32 off (exception: the verbatim replication arm uses its source recipe — AdamW 0.01, warmup+cosine, clip 1.0)
schedule	88,000 steps, batch 256 (≈ 2.05 epochs); gen-1: 8,000 steps on COCO (≈ 17.3 epochs)
seeds	student and data order CRC-derived from arm name + seed index; two seeds per arm
rotation	fitted after training: fp64 orthogonal Procrustes, student→teacher, on the fixed 2,500-pair COCO-val split; frozen thereafter
hardware	one 24 GB GPU per run (peak 4.83 GB); ~8 h per CC12M run on an A40, CPU-decode bound

Evaluation protocol

CIFAR-10/100 zero-shot via the LAION-B/16 text tower ("a photo of a {class}"), images tensor-upscaled (bicubic) to the student's 160px input; teacher judged at its native 224px in the same harness; floor (random init) and ceiling (teacher) bracket every table. COCO retrieval: image→text R@1/R@5 over the 5,000 val images against first-caption text embeddings, paired protocol. Agreement: mean cosine to the arm's target embeddings on COCO val. Effective rank and a spread statistic (CV of Cayley-Menger 4-simplex volumes on a fixed 16-d projection) are read-only health instruments. Every evaluation is deterministic: --eval-only re-runs reproduce ledger rows to four decimal places, and this release was gated by fresh tokenless downloads reproducing the headline numbers exactly (raw-loader path and AutoModel path, bit-parity verified).

Limitations

Research model. 48% of teacher zero-shot at best; the preregistered 60% viability bar is unmet (the data-scaling curve is unbent — see the article). CIFAR absolute numbers are harness-scoped (32px upscale; see the evaluation-path note). Two seeds per objective; one batch size (256); one student size; one consensus family; one deployment frame. The rotation fit consumes 2,500 labeled pairs of the deployment frame.