assets: optimized tracking demo GIF (8.9MB, same label quality)

.gitattributes

@@ -34,5 +34,3 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
 *.gif filter=lfs diff=lfs merge=lfs -text
-assets/architecture.png filter=lfs diff=lfs merge=lfs -text
-assets/dense_attention_comparison.png filter=lfs diff=lfs merge=lfs -text

README.md

@@ -13,24 +13,25 @@ tags:
 
 # Motif Vision Encoder
 
-Motif Vision Encoder is a unified image + video self-supervised vision encoder on a ViT
-backbone. A single 3D-convolutional tokenizer ingests both modalities — an image is a
-1-frame clip (`T=1`), a video is `T>1` — so the same weights produce dense patch-level
-features and a language-aligned global (CLS) representation.
+Motif Vision Encoder is a **unified image + video** self-supervised vision encoder on a ViT
+backbone. A single **3D-convolutional tokenizer** ingests both modalities — an image is a
+1-frame clip (`T=1`), a video is `T>1` — so the same weights produce **dense patch-level
+features** and a **language-aligned global (CLS) representation**.
 
 Trained on **~1/3 the data of DINOv3** (0.5B vs 1.7B samples), it still reaches competitive
 performance across image and video benchmarks — and leads on DAVIS video tracking.
 
 <p align="center">
-  <img src="assets/haaland_attn_blk20.gif" width="480" alt="Point tracking on a video clip: Motif vs V-JEPA 2.1"/>
+  <img src="assets/haaland_full_attn_blk20.gif" width="480" alt="Point tracking on a video clip: Motif vs V-JEPA 2.1"/>
 </p>
-<p align="center"><em>Point tracking on a video clip (top: Motif, bottom: V-JEPA 2.1) — a query point propagated across frames by patch-feature cosine similarity. Motif tracks the subject more reliably than V-JEPA 2.1.</em></p>
+<p align="center"><em>Point tracking on a video clip (top: Motif, bottom: V-JEPA 2.1) — a query point propagated across frames by patch-feature cosine similarity. Motif stays locked on the subject noticeably better than V-JEPA 2.1.</em></p>
 
 - **Architecture**: ViT-7B (embed 4096 / depth 40 / heads 32), patch 16, 3D axial RoPE
   (`base=100`), SwiGLU FFN, LayerScale, per-head QK-norm, gated attention, 4 register tokens.
 - **Tokenizer**: `Conv3d(kernel=stride=(tubelet, patch, patch))` — image `(B,3,H,W)` → `T=1`,
   video `(B,T,3,H,W)`. Token layout `[CLS] + [register × 4] + [patch × N]`.
-
+- **This repo**: inference-only. Weights + self-contained `modeling_motif_vision_encoder.py` (loaded
+  via `trust_remote_code`). Training/eval code lives in the GitHub repo.
 
 ## Usage
 
@@ -46,7 +47,7 @@ from transformers.image_utils import load_image
 url = "http://images.cocodataset.org/val2017/000000039769.jpg"
 image = load_image(url)
 
-repo = "Motif-Technologies/Motif-Vision-Encoder"
+repo = "Motif-Technologies/Motif-Vision-Encoder-7B"
 processor = AutoImageProcessor.from_pretrained(repo)
 model = AutoModel.from_pretrained(repo, trust_remote_code=True, dtype=torch.bfloat16).to("cuda").eval()
 
@@ -78,10 +79,6 @@ with torch.inference_mode():
 
 ## Model details
 
-<p align="center">
-  <img src="assets/architecture.png" width="820" alt="Motif Vision Encoder architecture: image and video inputs, patch embedding, 40-block transformer stack, and transformer block internals"/>
-</p>
-
 | | |
 |---|---|
 | Backbone | ViT-7B, patch 16, embed 4096, depth 40, heads 32, SwiGLU |
@@ -98,42 +95,49 @@ Outputs (`BaseModelOutputWithPooling`): `last_hidden_state` `(B, 1+4+N, 4096)`,
 ## Evaluation
 
 Compared against the strongest publicly reported self-supervised / vision backbones. Higher is
-better for every column. Best comparable value per column in bold, second best <u>underlined</u>.
+better for every column **except KITTI depth** (lower is better). Best **comparable** value per
+column in **bold**, second best <u>underlined</u>; the footnoted figure (‡) is measured under a
+different protocol and is excluded from the ranking.
 
 DAVIS S/M/L follow the DINOv3 protocol (J&F-mean at video short side 420/480, 840/960,
 1260/1440 px). V-JEPA 2.1 is not part of the DINOv3 Table 5 tracking benchmark, so only its
 single-resolution (S) figure is available.
 
-| Model | Training<br>data | DAVIS S<br>J&F ↑ | DAVIS M<br>J&F ↑ | DAVIS L<br>J&F ↑ | ImageNet-1K<br>lin. probe ↑ | ADE20K<br>mIoU ↑ | K400 ↑ |
-|---|---|---|---|---|---|---|---|
-| **Motif Vision Encoder** | 0.5B | **73.8** | **80.4** | **83.4** | 87.4 | <u>52.0</u> | 87.4 |
-| DINOv3 | 1.7B | <u>71.1</u> | <u>79.7</u> | <u>83.3</u> | 88.4 | **55.9** | <u>87.8</u> |
-| Web-DINO | 2B | 57.2 | 65.8 | 69.5 | 85.9 | 42.7 | 86.8 |
-| PEcore | 5.4B | 48.2 | 53.1 | 49.8 | **89.3** | 38.9 | **87.9** |
-| SigLIP2 | 10B | 56.1 | 62.3 | 62.9 | <u>89.1</u> | 45.4 | 86.9 |
-| OpenCLIP | 2B | – | – | – | – | – | – |
-| V-JEPA 2.1 | 0.022B | 69.0 | – | – | 85.5 | 47.9 | 87.7 |
+| Model | Training<br>data | DAVIS S<br>J&F ↑ | DAVIS M<br>J&F ↑ | DAVIS L<br>J&F ↑ | ImageNet-1K<br>lin. probe ↑ | ADE20K<br>mIoU ↑ | K400 ↑ | KITTI<br>depth MSE ↓ |
+|---|---|---|---|---|---|---|---|---|
+| **Motif Vision Encoder** | 0.5B | **74.0** | **80.5** | **83.5** | 87.2 | <u>52.0</u> | 87.4 | *in progress* |
+| DINOv3 | 1.7B | <u>71.1</u> | <u>79.7</u> | <u>83.3</u> | 88.4 | **55.9** | <u>87.8</u> | **2.3** |
+| PEcore | 5.4B | 48.2 | 53.1 | 49.8 | **89.3** | 38.9 | **87.9** | 4.1 |
+| SigLIP2 | 10B | 56.1 | 62.3 | 62.9 | <u>89.1</u> | 45.4 | 86.9 | – |
+| OpenCLIP | 2B | – | – | – | – | – | – | – |
+| V-JEPA 2.1 | 0.022B | 69.0 | – | – | 85.5 | 47.9 | 87.7 | <u>2.5</u> |
+| VideoMAEv2 | 1.35M | – | – | – | – | – | 90.0<sup>‡</sup> | – |
 
-Protocol: DINOv3-style linear/attentive probes for image tasks; V-JEPA 2-style protocol for
-video. Baseline DAVIS / ADE20K / K400 figures are taken from the DINOv3 technical report's
-unified evaluation (Tab. 3, 5, 6) and ImageNet from Tab. 7; OpenCLIP is not in that report and
-its benchmarks are not reported under a comparable protocol.
-
-Motif is state of the art on DAVIS video tracking at every resolution (74.0 / 80.5 / 83.5 J&F)
-and stays competitive on the other image and video benchmarks, using roughly 1/3 of DINOv3's
-training data (~0.5B samples).
+<sub>‡ VideoMAEv2 ViT-g: K400 top-1 after **full fine-tuning**, not the frozen attentive-probe protocol used for the other rows.</sub>
 
-<p align="center">
-  <img src="assets/davis_mask_propagation.gif" width="820" alt="Mask propagation: ground truth vs DINOv3 vs Motif"/>
-</p>
-
-<p align="center">
-  <img src="assets/dense_attention_comparison.png" width="820" alt="Dense attention and feature-similarity comparison across Motif, DINOv3, V-JEPA 2.1, and SigLIP2"/>
-</p>
-<p align="center"><em>Dense features on a single image (768px). Columns: query point, CLS attention, query-point attention, patch-feature cosine similarity. Motif and DINOv3 keep attention and similarity tightly localized on the queried object, while V-JEPA 2.1 and SigLIP2 are noticeably noisier.</em></p>
+Motif sets a new **state of the art on DAVIS video segmentation**, leading at every resolution
+(74.0 / 80.5 / 83.5 J&F), and stays **competitive across the remaining image and video
+benchmarks** — despite training on only **~0.5B samples**, roughly **1/3 of DINOv3's data**.
+KITTI depth is still being evaluated and will be added as the run completes.
 
+Protocol: DINOv3-style linear/attentive probes for image tasks; V-JEPA 2-style protocol for
+video. Baseline DAVIS / ADE20K / K400 / KITTI figures are taken from the DINOv3 technical
+report's unified evaluation (Tab. 3, 5, 6) and ImageNet from Tab. 7; OpenCLIP and VideoMAEv2
+are not in that report; VideoMAEv2's K400 figure is cited from its own paper (see ‡), and
+OpenCLIP's remaining benchmarks are not reported under a comparable protocol.
 
 ## License
 
 Released under the **MIT License** (see `LICENSE`). The model was trained on data governed by the
 respective dataset licenses; downstream users are responsible for compliance with those terms.
+
+## Citation
+
+```bibtex
+@misc{motif_vision_encoder,
+  title  = {Motif Vision Encoder: a unified image/video self-supervised vision encoder},
+  author = {Motif Technologies},
+  year   = {2026},
+  howpublished = {\url{https://huggingface.co/Motif-Technologies/Motif-Vision-Encoder-7B}}
+}
+```