File size: 4,657 Bytes
b63d4e6 4e64076 b63d4e6 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 | ---
license: mit
base_model: facebook/vjepa2-vitl-fpc16-256-ssv2
tags:
- core-ai
- video-classification
- video-understanding
- vjepa2
- apple-silicon
- feature-extraction
library_name: coreai
pipeline_tag: video-classification
---
# V-JEPA 2 ViT-L (SSv2) β Core AI, **two-output** (logits + router embedding)
A Core AI export of [`facebook/vjepa2-vitl-fpc16-256-ssv2`](https://huggingface.co/facebook/vjepa2-vitl-fpc16-256-ssv2)
(375M, MIT) that returns **both** the SSv2 action logits **and the pooled `[1,1024]` router
embedding**.
```
pixel_values_videos [1,16,3,256,256] fp16
ββ logits [1,174] SSv2 action classes
ββ embedding [1,1024] pooled attentive-pooler output
```
## Why two outputs
The stock export returns `logits` only. That is the wrong contract for anything using V-JEPA 2 as a
**content/router feature extractor** rather than an action classifier: the SSv2 checkpoint ships no
label names, its vocabulary is hand-object interactions, and it is semantically wrong for most
non-action domains. The `[1024]` pooled embedding is the reusable signal β feed it to a small
calibrated head for your own task.
Built for a video-optimization planner that fits a logistic probe over this embedding to pre-classify
content before an encode search.
## Measured
Apple **M5 Max** / 128 GB, fp16, GPU:
| | |
|---|---|
| Bundle | **708 MB** (675 MiB) |
| Load | **0.16 s** |
| Warm forward (16-frame clip) | **44 ms** |
| Engine logits vs torch oracle | cos **0.999964**, top-1 match |
| Engine embedding vs torch reference | cos **0.999940**, relL2 1.1e-02 (fp16 noise) |
| Recomposition `classifier(pooler(backbone(x)))` vs `.logits` | cos **1.0000000**, max\|Ξ\| **0** β bit-exact |
β οΈ Latency is silicon-specific. Published M4 Max figures for the single-output bundle are 150β180 ms;
that is **different hardware**, not a difference between these bundles.
**Embedding separation** (6 signage clips, a semantic check no numeric gate performs): pairwise
cosine spans **0.146 β 0.987**, and the highest-similarity pair is the same content at two different
encodes (`_1080p` vs `_master`, cos 0.987). Content-identical inputs collapse together; unrelated
content separates.
## Preprocessing β on the host
The model does **not** normalize internally.
1. 16 frames, uniformly sampled across the clip
2. aspect-fill to 256Γ256, centre-crop (not a squash)
3. RGB, 0..1
4. ImageNet mean `[0.485, 0.456, 0.406]` / std `[0.229, 0.224, 0.225]`
5. `[1,16,3,256,256]`, fp16
## Use
```python
import asyncio, numpy as np, coreai.runtime as rt
async def main():
opts = rt.SpecializationOptions.from_preferred_compute_unit_kind(rt.ComputeUnitKind.gpu())
fn = (await rt.AIModel.load("vjepa2_ssv2_embed_fp16.aimodel", opts)).load_function("main")
x = ... # [1,16,3,256,256] fp16, preprocessed as above
out = await fn(inputs={"pixel_values_videos": rt.NDArray(np.ascontiguousarray(x))})
logits = out["logits"].numpy() # [1,174]
embedding = out["embedding"].numpy() # [1,1024] β the signal
asyncio.run(main())
```
## Reproduce
`export_fp16_embed.py` is included. It needs `coreai_models`, torch, and transformers β₯4.57, plus
oracles from the reference run (`oracle_*.npy` are included here, so the gate can be re-run without
recomputing them).
Two things in that script are load-bearing and easy to lose in a re-derivation:
- **`skip_predictor=True`.** `VJEPA2ForVideoClassification.forward` passes it; `VJEPA2Model` defaults
it to `False`, which runs the entire JEPA *predictor* stack. `last_hidden_state` is
`sequence_output` either way, so omitting it changes **no number** β every parity gate still
passes, and the bundle silently carries a transformer stack nothing reads. **Parity gates certify
values, never graph extent.**
- **The 3D-RoPE `squeeze(-1)` overlay.** The reference squeezes a non-1 dim (a torch no-op); the
converter maps it to `ShrinkDims`, which requires size 1, and fails. Patched out, math unchanged.
The submodule resolution is discover-then-prove: it resolves `backbone`/`pooler`/`classifier` by
candidate name, then asserts the recomposition reproduces the model's own `.logits` before exporting.
If a future transformers renames them, it fails loudly instead of shipping a wrong "embedding".
## Files
| | |
|---|---|
| `vjepa2_ssv2_embed_fp16.aimodel/` | the bundle (`metadata.json`, `main.mlirb`, `main.hash`) |
| `export_fp16_embed.py` | the exact export that produced it |
| `oracle_input.npy` / `oracle_logits.npy` / `oracle_embedding.npy` | gate fixtures |
## License
MIT, inherited from Meta's V-JEPA 2.
|