| --- |
| license: mit |
| base_model: facebook/vjepa2-vitl-fpc16-256-ssv2 |
| tags: |
| - core-ai |
| - video-classification |
| - video-understanding |
| - vjepa2 |
| - apple-silicon |
| - feature-extraction |
| library_name: coreai |
| pipeline_tag: video-classification |
| --- |
| |
| # V-JEPA 2 ViT-L (SSv2) β Core AI, **two-output** (logits + router embedding) |
|
|
| A Core AI export of [`facebook/vjepa2-vitl-fpc16-256-ssv2`](https://huggingface.co/facebook/vjepa2-vitl-fpc16-256-ssv2) |
| (375M, MIT) that returns **both** the SSv2 action logits **and the pooled `[1,1024]` router |
| embedding**. |
|
|
| ``` |
| pixel_values_videos [1,16,3,256,256] fp16 |
| ββ logits [1,174] SSv2 action classes |
| ββ embedding [1,1024] pooled attentive-pooler output |
| ``` |
|
|
| ## Why two outputs |
|
|
| The stock export returns `logits` only. That is the wrong contract for anything using V-JEPA 2 as a |
| **content/router feature extractor** rather than an action classifier: the SSv2 checkpoint ships no |
| label names, its vocabulary is hand-object interactions, and it is semantically wrong for most |
| non-action domains. The `[1024]` pooled embedding is the reusable signal β feed it to a small |
| calibrated head for your own task. |
|
|
| Built for a video-optimization planner that fits a logistic probe over this embedding to pre-classify |
| content before an encode search. |
|
|
| ## Measured |
|
|
| Apple **M5 Max** / 128 GB, fp16, GPU: |
|
|
| | | | |
| |---|---| |
| | Bundle | **708 MB** (675 MiB) | |
| | Load | **0.16 s** | |
| | Warm forward (16-frame clip) | **44 ms** | |
| | Engine logits vs torch oracle | cos **0.999964**, top-1 match | |
| | Engine embedding vs torch reference | cos **0.999940**, relL2 1.1e-02 (fp16 noise) | |
| | Recomposition `classifier(pooler(backbone(x)))` vs `.logits` | cos **1.0000000**, max\|Ξ\| **0** β bit-exact | |
|
|
| β οΈ Latency is silicon-specific. Published M4 Max figures for the single-output bundle are 150β180 ms; |
| that is **different hardware**, not a difference between these bundles. |
|
|
| **Embedding separation** (6 signage clips, a semantic check no numeric gate performs): pairwise |
| cosine spans **0.146 β 0.987**, and the highest-similarity pair is the same content at two different |
| encodes (`_1080p` vs `_master`, cos 0.987). Content-identical inputs collapse together; unrelated |
| content separates. |
|
|
| ## Preprocessing β on the host |
|
|
| The model does **not** normalize internally. |
|
|
| 1. 16 frames, uniformly sampled across the clip |
| 2. aspect-fill to 256Γ256, centre-crop (not a squash) |
| 3. RGB, 0..1 |
| 4. ImageNet mean `[0.485, 0.456, 0.406]` / std `[0.229, 0.224, 0.225]` |
| 5. `[1,16,3,256,256]`, fp16 |
|
|
| ## Use |
|
|
| ```python |
| import asyncio, numpy as np, coreai.runtime as rt |
| |
| async def main(): |
| opts = rt.SpecializationOptions.from_preferred_compute_unit_kind(rt.ComputeUnitKind.gpu()) |
| fn = (await rt.AIModel.load("vjepa2_ssv2_embed_fp16.aimodel", opts)).load_function("main") |
| x = ... # [1,16,3,256,256] fp16, preprocessed as above |
| out = await fn(inputs={"pixel_values_videos": rt.NDArray(np.ascontiguousarray(x))}) |
| logits = out["logits"].numpy() # [1,174] |
| embedding = out["embedding"].numpy() # [1,1024] β the signal |
| |
| asyncio.run(main()) |
| ``` |
|
|
| ## Reproduce |
|
|
| `export_fp16_embed.py` is included. It needs `coreai_models`, torch, and transformers β₯4.57, plus |
| oracles from the reference run (`oracle_*.npy` are included here, so the gate can be re-run without |
| recomputing them). |
|
|
| Two things in that script are load-bearing and easy to lose in a re-derivation: |
|
|
| - **`skip_predictor=True`.** `VJEPA2ForVideoClassification.forward` passes it; `VJEPA2Model` defaults |
| it to `False`, which runs the entire JEPA *predictor* stack. `last_hidden_state` is |
| `sequence_output` either way, so omitting it changes **no number** β every parity gate still |
| passes, and the bundle silently carries a transformer stack nothing reads. **Parity gates certify |
| values, never graph extent.** |
| - **The 3D-RoPE `squeeze(-1)` overlay.** The reference squeezes a non-1 dim (a torch no-op); the |
| converter maps it to `ShrinkDims`, which requires size 1, and fails. Patched out, math unchanged. |
|
|
| The submodule resolution is discover-then-prove: it resolves `backbone`/`pooler`/`classifier` by |
| candidate name, then asserts the recomposition reproduces the model's own `.logits` before exporting. |
| If a future transformers renames them, it fails loudly instead of shipping a wrong "embedding". |
|
|
| ## Files |
|
|
| | | | |
| |---|---| |
| | `vjepa2_ssv2_embed_fp16.aimodel/` | the bundle (`metadata.json`, `main.mlirb`, `main.hash`) | |
| | `export_fp16_embed.py` | the exact export that produced it | |
| | `oracle_input.npy` / `oracle_logits.npy` / `oracle_embedding.npy` | gate fixtures | |
|
|
| ## License |
|
|
| MIT, inherited from Meta's V-JEPA 2. |
|
|
