Geometric Memory FT4 — Distill Against a Consensus, Ship a Rotation

A student distilled against a consensus of five teachers — never against the deployment teacher — outperformed the directly-distilled student on every task gauge, both seeds, at identical training cost; the only deployment-specific artifact is a half-megabyte frozen rotation. This report documents that result and the campaign around it: what a memory bank actually stores when measured against random controls, the frame trap that falsifies naive evaluation of consensus and relational objectives, and two generations of distillation objectives across 118 thousand and 11 million images.

Notation: paired values a / b are seed 0 / seed 1 throughout. All numbers come from the published ledgers; evaluations are deterministic (re-runs reproduce to four decimal places, cross-machine). Training code for every released checkpoint is included in the model repository (train/).

---

Part I — Scope and discipline

This series studies geometry as memory: whether structured geometric objects — anchor frames, codebooks, consensus shapes — carry usable, transferable information for neural encoders, and how to measure that without self-deception. Installment four comprises three interlocking campaigns:

1. Campaign 1 (Part III): an alignment-bank architecture transplanted from the language side of this series to vision, tested against capacity controls.
2. Campaign 2 (Part IV): an objective battery — a composite loss, its isolated forces, and a verbatim replication of its published ancestor — at COCO scale, extended to CC12M in Part VII.
3. Campaign 3 (Parts V–IX): a nine-objective distillation matrix at two data generations, ending in the consensus result.

Standing rules: preregistered numeric bars with refutation conditions written before launch; two seeds per claim; geometry metrics as read-only instruments unless explicitly promoted to forces; and any recipe adapted from a published prototype runs alongside a verbatim replication arm of that prototype, exceptions flagged rather than smoothed away.

Part II — Substrates

Two public feature banks carry everything.

bulk-coco-features: pooled image features for the 123,287 COCO-2017 images from 34 teacher towers. It supplies the expert frames for Campaign 1, the targets for generation-one distillation, and the reference truth that every later extraction must reproduce before spending compute. Its instructive defect: it did not document its own preprocessing, and recovering those constants — plain GELU vs QuickGELU (a mismatch that reads as cosine ≈ 0.975 against stored features), exact resize/crop geometry, storage precision — required a falsification campaign against the stored rows.

bulk-cc12m-features: built for this installment. CC12M (Changpinyo et al., via pixparse/cc12m-wds) is 1.18 TB of image tars. One streaming pass (download → decode → embed → discard; peak disk under 40 GB) produced CLIP-B/16-LAION-2B (laion/CLIP-ViT-B-16-laion2B-s34B-b88K) features for 10,968,539 images with captions — 12.7 GB fp16, zero decode errors and zero download failures across 2,176 shards. Five more towers followed (SigLIP-384, published in the same dataset; four CLIP variants for the consensus work). Each extraction passed two gates before spending: the fast decode path against the tower's reference processor, and reproduction of the tower's stored COCO features on reference rows — both enforced at cosine ≥ 0.9999 (the conversion of the two open_clip-only checkpoints was additionally verified against stored bank rows at 0.999996+). The recovered preprocessing constants are documented per tower on the dataset card. Precomputed teachers remove teacher inference from the training loop — roughly a third of ordinary distillation compute, prepaid once.

Part III — Campaign 1: the memory bank against its controls

The bank experiment transplants an alignment-bank architecture to vision: expert frames fitted from the 34-tower COCO bank (fp64 whitened Procrustes), a vision alignment bank scoring 538-dimensional signatures under an 8-term geometry loss, attached to a consensus-distilled encoder. Three stages — encoder (A), bank expansions (B), co-trained pairs (C) — with preregistered bars.

An instrument finding preceded the main result: the same expert frames read three ways depending on gauge — unaligned raw cosine ≈ 0.4, whitened cosine 0.70–0.77, rotation-aligned 0.932–0.943, the last matching the recorded consensus-alignment value and serving as the correct continuity gauge. The gate harness also rejected a whitener overfitted on 1k rows (fit 0.80 / held-out 0.42).

The capacity result, 80-class linear probe mAP, both seeds:

Bank expansion lifts the probe by +1.4–1.6 points — and a fully random bank matches the real fitted frames (.4159/.4114 vs .4149/.4107), while geometry training subtracts (−.0038/−.0045) and co-training subtracts further (−.0076/−.0052). The enrichment is generic random-feature capacity; the fitted geometric content contributes approximately zero probe utility on this substrate. Stage C did replicate the ancestor prototype's characteristic trunk shape (agreement rises under co-training, 2/2) while the probe worsened — the shape transfers cross-modally; the utility does not.

Scope, stated exactly: this refutes bank-training and signature-content on this encoder-probe substrate. It does not test, and therefore does not touch, the ancestor prototype's original language-side results or non-classification gauges. It is, however, the third bed in this program where a capacity control dissolved an apparent geometric win — which is why the control is mandatory.

The artifacts shipped regardless — geolip-vit-captionbank-coco: encoders, expansions, co-trained pairs, frames, complete ledgers, with the refutation stated on the card. The governing rule: verdicts gate claims, not artifacts.

Part IV — Campaign 2: the objective battery (COCO scale)

The bank's Stage-A encoder trains with a composite objective: symmetric InfoNCE plus per-sample MSE against a five-teacher consensus target. The composite descends from a published prototype recipe, which ran verbatim alongside (per-element MSE at 1.0, a spread regularizer at 0.1, AdamW with warmup/cosine and clipping). Ten variant encoders — five recipes, two seeds each — on one evaluation board:

Findings, each 2/2 unless noted:

• The verbatim underperforms the adaptation on every gauge (agreement .41 vs .67), with a measured mechanism: per-element MSE is ~1000× smaller than the contrastive term at initialization — an anchor in name only — leaving the stack weakly anchored while the prototype's optimizer machinery damps what remains.
• The probe channel is flat (~.39–.40) across all six recipes: these stacks decide frame and alignment, not feature richness — independently corroborating Part III's capacity verdict.
• A per-term descent decomposition (every term logged on every arm's batches, plus a no-align ablation) resolved why the weakly-anchored stack "learns fastest": its contrastive term descends ~8% faster because nothing opposes it, while the anchored composite front-loads agreement to 93% of its final value by step 500. The apparent speed is unanchored drift; anchor weight is a retention-versus-plasticity dial.
• The BCE force is a consistent whisper (+.008, below claim threshold); the spread force at its 1e-3 ceiling is inert — its third consecutive substrate.

Part V — Campaign 3, generation one (118k images)

The distillation matrix compares objectives from the literature — mimicry (CLIP-KD), contrastive (InfoNCE), affinity (TinyCLIP), decoupled sigmoid (SigLIP) — plus this series' composite and consensus forms, on one bed: an 8.66M-parameter ViT-Ti student (10.0% of the teacher tower), identical recipe and harness (details in Part XII), floor and ceiling bracketed in-harness.

At COCO budget (8k steps ≈ 17.3 epochs), six trainable arms produced:

• A gauge split. Mimicry won coarse zero-shot both seeds (CIFAR-10 .2693/.2657 vs InfoNCE .2668/.2308); InfoNCE won retrieval 2.2× (R@1 .0580/.0544 vs .0248/.0262) and fine-grained zero-shot (CIFAR-100 .0939/.0847 vs .0619/.0623).
• A preregistered consensus refutation. The multi-teacher question was asked at this scale first: does a student distilled from the five-teacher consensus beat the best single-teacher student? Bar: win on zero-shot 2/2; refuted if it loses 2/2. It lost 2/2 — raw zero-shot .1543/.1609 against mimicry's .2693/.2657. Hold this refutation; it reverses at scale (Part IX).
• An inert composite. A sixth arm carried the series' certified composite forces (0.3·Procrustes residual + 1e-3 spread force) over InfoNCE; the preregistered geometry-channel bar was refuted — every gauge within seed noise of plain InfoNCE (effective rank 184.7/184.5 vs 184.4/184.6), and InfoNCE lands the target spread band natively.
• Health passed everywhere. All fourteen runs cleared preregistered spectrum bars (minimum effective rank 106.8 against a random floor of 32; no rank collapse) — the instrument that licensed the next paragraph's diagnosis.
• Two strange floors. The affinity arm sat at floor on every absolute gauge (zs .1091/.1107, retrieval ≈ 0) with healthy spectrum (rank 175) and teacher agreement at chance (−0.0208/−0.0015) — orthogonal, not absent. The consensus arm scored near floor raw while agreeing with its own target at 0.806.

Part VI — The frame law (generation one)

Objectives that never compare student vectors to deployment-teacher vectors directly — similarity-row matching, or absolute targets in a different space — leave the student's global orientation unconstrained, and absolute gauges then report a false floor. The correction is one orthogonal Procrustes rotation: fitted in fp64 on 2,500 held-out pairs in seconds, then frozen.

At generation one the rotation was a full rescue for the affinity arm: agreement −0.0208 → 0.501 (s0) and −0.0015 → 0.4969 (s1) — post-rotation agreements matching across seeds to 0.004 — and zero-shot .109 → .240 (s0), .104 → .218 (s1), parity with the trained-in-frame arms. The control (InfoNCE, already in-frame) moved nothing: .4051 → .4093.

The same forensic on the generation-one consensus arm produced the first sighting of Part IX's result: rotated, its deployment-teacher agreement reached .7132/.7116 — equal to direct mimicry's in-frame .714/.718. The consensus target had taught the deployment teacher's geometry; it had only lost the gauge.

Part VII — Generation two (11M images): the raw matrix

Nine objectives × two seeds at 88k steps (≈ 2.05 epochs of CC12M) — same student, same recipe, eleven times the data seen (and ~93× the unique images). Values s0/s1; agree is cosine to the deployment teacher unless marked (own) = the arm's own target; bold marks the raw column winner; † marks the row Part IX crowns after rotation.

Raw verdicts, each 2/2:

1. The generation-one gauge split did not survive scale. InfoNCE swept every raw task gauge. Objective comparisons made at small scale measure the scale, not the objectives.
2. Batch-coupled beat decoupled (InfoNCE over SigLIP-pairwise) on zero-shot and retrieval at batch 256; generation one's seed split on this comparison also resolved as noise. One batch size; disclosed.
3. The viability curve (preregistered: 60% of teacher zero-shot at ≤10% params; refuted below 40%): 21.2% at COCO budget → 42.0/41.5% (InfoNCE at CC12M) → 44.7/48.1% (Part IX's champion). Floor cleared both seeds; bar unmet; curve unbent.
4. The verbatim-vs-adapted ordering held unchanged at scale (agreement .531/.530 vs .741/.742) — the replication-arm comparison itself is now a two-substrate result.

Part VIII — The frame law at scale

The rotation forensic across the full matrix (agree→deployment-teacher and CIFAR-10 zero-shot, pre → post; both seeds):

Three families:

• Anchored-absolute: in-frame by construction; the control's zero-shot barely moves under rotation (its agreement gains ≈ .09, consistent with a mild residual misalignment the fit removes).
• Consensus-absolute: geometry fully intact. Rotated, the pure-MSE consensus student agrees with the deployment teacher at .765/.763 — above the .735/.738 of the student trained directly against that teacher.
• Relational: frame-free and, at scale, absolutely degraded — recovery fell from generation-one parity to ≈ 0.30 while effective rank sagged from 175 to 103/89. At small scale, gauge freedom is a measurement problem; at large scale it coincides with a genuine optimization pathology.

Evaluating consensus or relational distillation without a frame fit produces a false scoreboard; Parts V and VII's raw tables demonstrate the size of the error.

Part IX — The crown: a refutation reversed by scale

Generation one preregistered the multi-teacher question and refuted it at the letter (Part V). At generation two, the same comparison — consensus-distilled student vs the best student trained directly on the deployment teacher — reverses on every task gauge once the frozen rotation is applied:

Identical training cost; the deployment-specific deliverable is a 512×512 orthogonal matrix (0.5 MB) fitted post-hoc on 2,500 pairs. The recipe: distill against a consensus of teachers; ship a rotation into the frame you serve. The consensus mean suppresses each teacher's idiosyncratic component while preserving the shared semantic geometry; the rotation supplies the only deployment-specific information the student needs. That this exact claim was formally refuted one data generation earlier is the strongest statement of verdict 1 in Part VII: scale reversed a preregistered result, in public, with both versions ledgered.

Part X — The anchor dial

Both terms of the champion are load-bearing, measured as a monotone dial. Recipes ordered by absolute-anchor strength; agreement vs retrieval (deployment frame; rotated where the arm is frame-free):

More anchor buys agreement and sells retrieval; the composite occupies the productive point. The pure-MSE row shows the dial's far end even after rotation: fidelity without separability (R@1 ≈ .05). The dial replicates at COCO scale — pure-MSE consensus reached agreement .805 against the composite's .672/.670 there, the same trade at one eleventh the data — and Part IV's descent decomposition supplies the mechanism: the anchor front-loads and holds the frame; the contrastive term spends the remaining freedom on separation.

Part XI — Two structural observations

Set-points. Across two beds and two scales, each loss stack converged to a teacher-agreement value characteristic of the stack:

• COCO bed: verbatim .411/.412 · full stack .496/.496 (cross-seed ≤ 7×10⁻⁴ for these weak-anchor recipes) · adapted composite .672/.670 · +BCE .680/.677 (cross-seed ≤ 3.3×10⁻³ across the list).
• CC12M bed: verbatim .5314/.5300 · composite family — base .741/.742, +gradient .742/.742, +spread .743/.741 (in-stack forces do not move the set-point) · pure-MSE .814/.810 · mimicry .735/.738.
• Set-points survive rotation (post-rotation cross-seed ≤ .004; ≤ .002 for the composite and InfoNCE arms).
• They are budget-modulated: the same verbatim stack sits at .41 on 118k images and .53 on 11M.

Agreement at convergence is, to good approximation, a readout of loss design — measurable on a single seed.

The variance collapser. One battery force moved no mean anywhere: a gradient-shaping operator (radial damping plus nearest-anchor separation in the backward pass; forward is identity). Its measured effect is on the variance channel: zero-shot pinned at .2539/.2534 on the COCO bed where its base swung .232–.264, and .3398/.3315 at CC12M (spread .0083) where the base swung .3271–.3928 (spread .0657) — an ~8× spread reduction on these two-seed pairs, means statistically indistinguishable at two seeds (.336 vs .360 at CC12M), agreement pin-tight (.742/.742). Suggestive rather than settled — two seeds per pair — but replicated across two beds and two scales, with no mechanism yet. Its rotated zero-shot (.517/.568) is the crown's closest runner-up.

Part XII — Training the models

Complete training code ships in the model repository under train/ (the bed, all objectives, teacher loading and conversion, the CC12M tar-offset data path, the consensus-target builder, and a reproduction guide, train/TRAINING.md). The invariant recipe: batch 256, Adam 3e-4 with zero weight decay, fp32 with TF32 off, CRC-derived seeds; 8,000 steps on COCO or 88,000 on CC12M; student outputs L2-normalized before every loss; targets fp16-resident, normalized per batch on GPU (row-wise normalization is batch-independent, so this is numerically identical to pre-normalizing at half the memory). The single exception is the verbatim replication arm, which reproduces its source optimizer exactly (AdamW wd 0.01, 1000-step warmup, cosine decay, grad-clip 1.0), flagged in its startup banner.

Per-arm objectives, as implemented in train/losses.py:

The intended purpose of each representation shape. The student's 512-d output is the deployment teacher's projection space: the dimension is chosen so the student is a drop-in image tower for the teacher's text encoder, which is what makes zero-shot and retrieval evaluations (and deployment) direct. The 768-d variant is the same statement for the SigLIP teacher's space. The GPA consensus mean is a target constructed to carry only geometry that five teachers share — five L2-normalized towers iteratively orthogonally aligned and averaged (alignment ≈ 0.91 at convergence) — so a student distilling it absorbs common semantic structure rather than any single teacher's residual noise; by construction this target has no privileged frame, which is why the rotation exists: a frozen orthogonal map carrying the consensus frame onto the deployment frame, holding all deployment-specific information in 512×512 numbers. The bank's 538-d signatures (Part III) encode per-sample geometric relations to expert frames — the hypothesis under test there, and the content the capacity controls showed to be probe-inert. Two shapes exist purely as instruments and never receive gradients: effective rank (spectrum health; the gauge that separated "frame problem" from "collapse"), and the spread CV on a fixed 16-d projection (the statistic whose forced version was inert three times as a training term while remaining useful as a readout).

Part XIII — Released artifacts

• clip-vitb-mini-distilled — champion pair with frozen rotations, the single-teacher alternate, the generation-one model for provenance, a rotation-aware standalone loader, complete ledgers and rotation forensics, and the full training code (train/). Fresh tokenless downloads reproduce the headline numbers to four decimals.
• bulk-cc12m-features — CLIP-B/16 + SigLIP-384 features for 10.97M images with captions; fp16 shards and viewer-ready parquet configs.
• bulk-coco-features — the 34-tower COCO substrate.
• geolip-vit-captionbank-coco — the memory-bank line: Stage A/B/C artifacts, expert frames, ten battery variants, the verbatim replication pair, descent-decomposition ledgers, the capacity refutation on the card.
• loss-manifest — the wider objective genealogy (168 rated entries) this installment extends.

Part XIV — Limits, then the result

The 60% viability bar is unmet (48%); the scaling curve is unbent and points at data. CIFAR evaluations upscale 32px images into a 160px student; same-harness floor/ceiling brackets keep comparisons meaningful, but absolute values are harness-scoped. Two seeds per claim, even where cross-seed agreement runs to four decimals. Batch-size sensitivity is unexplored (256 only). The capacity refutation is scoped to its encoder-probe substrate and does not touch the ancestor prototype's language-side results or non-classification gauges. The crown is demonstrated for one student size, one consensus family, one deployment frame. The frame correction consumes 2,500 labeled pairs of the deployment frame.

Within those limits, the campaign's transferable results stand: at scale, contrastive distillation dominates single-teacher objectives on raw gauges; consensus and relational objectives must be evaluated through a fitted rotation; anchor weight sets a measurable agreement-retrieval trade with stack-characteristic set-points; and a student distilled against a consensus of teachers, plus a half-megabyte rotation, outperformed direct distillation against the deployment teacher on every gauge measured — a preregistered refutation at 118k images, reversed at 11 million. The geometric object that transferred was never the memory bank; it was the consensus shape, plus a frame.

Ledgers, forensics, training code, and checkpoints: linked above; every number is re-runnable from the public artifacts.