geolip-vit-large-x3

A 78.3M-parameter from-scratch Vision Transformer (1024-d, 6 layers, 16 heads) trained by 3-expert consensus distillation on COCO-2017, producing L2-normalized 128-d embeddings on a shared consensus hypersphere, with a 256-anchor constellation + 8-compartment patchwork soup head giving 80-class multi-label COCO logits directly from the embedding.

The wide sibling of geolip-vit-base-x3 (384-d) — same recipe, same 6-layer depth, ~4× width.

Recipe (identical to base-x3)

Teacher: an ~800K-parameter "base tier soup" over three frozen experts — CLIP ViT-L/14 (OpenAI) + DINOv2 ViT-B/14 + SigLIP ViT-B/16-384 — features GPA-aligned at 768-d, PCA-projected to 128-d, per-expert whitened-Procrustes calibrated (teacher COCO mAP 0.837; consensus CV 0.2731 at 128-d).
Student: Xavier-init ViT, no pretrained weights; trained on raw COCO-2017 train (118,287 images); teacher targets cached from bulk-coco-features.
Loss: 1.0·InfoNCE(τ=0.07) + 0.5·MSE + 0.3·BCE (through the frozen soup head) + 0.5·whitened-Procrustes alignment + 0.001·pentachoron-CV (Cayley–Menger, target 0.2731).
55 epochs (136,550 steps), Adam 3e-4, batch 48, warmup+cosine, bf16 autocast.

Results (from the training run's TensorBoard, in this repo under `runs/`)

metric (COCO val 5k)	final (E55)	best
multi-label mAP	0.488	0.500
F1	0.492	0.498
R@1 (image → own consensus target)	0.420	0.431
cos to consensus target	0.650	0.663
active anchors	94 / 256	—
val CV	~0.12	—

Base-x3 for comparison: mAP 0.429, R@1 0.381, cos 0.600 — the large model is roughly +0.07 mAP / +0.05 R@1 / +0.06 cos for 4× width at equal depth.

Re-verification (2026-07-27, fresh local benchmark): faithful load reproduces anchor coverage 95/256 and micro-F1 0.491 on COCO val; a linear probe on the raw 128-d embedding (80-class multi-label, BCE linear probe) scores mAP 0.292; embedding-space pentachoron CV(16) measures 0.279 — matching the teacher consensus CV (0.2731/0.2793), i.e. the student faithfully inherited its teacher-space geometry.

Usage

import torch
from PIL import Image
from transformers import AutoModel
from torchvision import transforms

model = AutoModel.from_pretrained(
    "AbstractPhil/geolip-vit-large-x3", trust_remote_code=True).eval()

tf = transforms.Compose([
    transforms.Resize((224, 224)),   # squash resize — the training geometry
    transforms.ToTensor(),
    transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225]),
])
px = tf(Image.open("cat.jpg").convert("RGB")).unsqueeze(0)
with torch.no_grad():
    out = model(px)

out.embedding        # (1, 128) unit vector in the 3-expert consensus space
out.logits           # (1, 80)  multi-label COCO logits
out.triangulation    # (1, 256) 1 - cos to the constellation anchors
out.nearest          # (1,)     nearest anchor id

Note the embedding lives in the model's own consensus space (not CLIP text space) — there is no text tower; zero-shot classification via text prompts is not applicable. The soup head is the supervised route.

Revision history

v2 (2026-07-27) — modeling_geolip_vit.py fixed: the original file's module names did not match the released checkpoint (encoder.layers.* vs layers.*) and included an untrained geometric-injection path, so AutoModel silently returned a mostly-random model. The revision loads with zero missing / zero unexpected keys and bit-reproduces the faithful manual load (max embedding diff 4.5e-7); the same fix was applied to base-x3. Output is now a proper transformers.ModelOutput; diagnostics are opt-in via compute_diagnostics=True.
v1 (2026-03-15) — initial weights + TensorBoard logs.