Image Feature Extraction
Transformers
Safetensors
motif_vision
feature-extraction
motif
vision-transformer
self-supervised
video
custom_code
Instructions to use Motif-Technologies/Motif-Vision-Encoder with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use Motif-Technologies/Motif-Vision-Encoder with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("image-feature-extraction", model="Motif-Technologies/Motif-Vision-Encoder", trust_remote_code=True)# Load model directly from transformers import AutoModel model = AutoModel.from_pretrained("Motif-Technologies/Motif-Vision-Encoder", trust_remote_code=True, device_map="auto") - Notebooks
- Google Colab
- Kaggle
assets: refine architecture figure layout and caption legibility
#5
by gkalstn0 - opened
.gitattributes
CHANGED
|
@@ -35,4 +35,3 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
|
|
| 35 |
*tfevents* filter=lfs diff=lfs merge=lfs -text
|
| 36 |
*.gif filter=lfs diff=lfs merge=lfs -text
|
| 37 |
assets/architecture.png filter=lfs diff=lfs merge=lfs -text
|
| 38 |
-
assets/dense_attention_comparison.png filter=lfs diff=lfs merge=lfs -text
|
|
|
|
| 35 |
*tfevents* filter=lfs diff=lfs merge=lfs -text
|
| 36 |
*.gif filter=lfs diff=lfs merge=lfs -text
|
| 37 |
assets/architecture.png filter=lfs diff=lfs merge=lfs -text
|
|
|
README.md
CHANGED
|
@@ -22,7 +22,7 @@ Trained on **~1/3 the data of DINOv3** (0.5B vs 1.7B samples), it still reaches
|
|
| 22 |
performance across image and video benchmarks β and leads on DAVIS video tracking.
|
| 23 |
|
| 24 |
<p align="center">
|
| 25 |
-
<img src="assets/
|
| 26 |
</p>
|
| 27 |
<p align="center"><em>Point tracking on a video clip (top: Motif, bottom: V-JEPA 2.1) β a query point propagated across frames by patch-feature cosine similarity. Motif tracks the subject more reliably than V-JEPA 2.1.</em></p>
|
| 28 |
|
|
@@ -106,7 +106,7 @@ single-resolution (S) figure is available.
|
|
| 106 |
|
| 107 |
| Model | Training<br>data | DAVIS S<br>J&F β | DAVIS M<br>J&F β | DAVIS L<br>J&F β | ImageNet-1K<br>lin. probe β | ADE20K<br>mIoU β | K400 β |
|
| 108 |
|---|---|---|---|---|---|---|---|
|
| 109 |
-
| **Motif Vision Encoder** | 0.5B | **
|
| 110 |
| DINOv3 | 1.7B | <u>71.1</u> | <u>79.7</u> | <u>83.3</u> | 88.4 | **55.9** | <u>87.8</u> |
|
| 111 |
| Web-DINO | 2B | 57.2 | 65.8 | 69.5 | 85.9 | 42.7 | 86.8 |
|
| 112 |
| PEcore | 5.4B | 48.2 | 53.1 | 49.8 | **89.3** | 38.9 | **87.9** |
|
|
@@ -123,15 +123,6 @@ Motif is state of the art on DAVIS video tracking at every resolution (74.0 / 80
|
|
| 123 |
and stays competitive on the other image and video benchmarks, using roughly 1/3 of DINOv3's
|
| 124 |
training data (~0.5B samples).
|
| 125 |
|
| 126 |
-
<p align="center">
|
| 127 |
-
<img src="assets/davis_mask_propagation.gif" width="820" alt="Mask propagation: ground truth vs DINOv3 vs Motif"/>
|
| 128 |
-
</p>
|
| 129 |
-
|
| 130 |
-
<p align="center">
|
| 131 |
-
<img src="assets/dense_attention_comparison.png" width="820" alt="Dense attention and feature-similarity comparison across Motif, DINOv3, V-JEPA 2.1, and SigLIP2"/>
|
| 132 |
-
</p>
|
| 133 |
-
<p align="center"><em>Dense features on a single image (768px). Columns: query point, CLS attention, query-point attention, patch-feature cosine similarity. Motif and DINOv3 keep attention and similarity tightly localized on the queried object, while V-JEPA 2.1 and SigLIP2 are noticeably noisier.</em></p>
|
| 134 |
-
|
| 135 |
|
| 136 |
## License
|
| 137 |
|
|
|
|
| 22 |
performance across image and video benchmarks β and leads on DAVIS video tracking.
|
| 23 |
|
| 24 |
<p align="center">
|
| 25 |
+
<img src="assets/haaland_full_attn_blk20.gif" width="480" alt="Point tracking on a video clip: Motif vs V-JEPA 2.1"/>
|
| 26 |
</p>
|
| 27 |
<p align="center"><em>Point tracking on a video clip (top: Motif, bottom: V-JEPA 2.1) β a query point propagated across frames by patch-feature cosine similarity. Motif tracks the subject more reliably than V-JEPA 2.1.</em></p>
|
| 28 |
|
|
|
|
| 106 |
|
| 107 |
| Model | Training<br>data | DAVIS S<br>J&F β | DAVIS M<br>J&F β | DAVIS L<br>J&F β | ImageNet-1K<br>lin. probe β | ADE20K<br>mIoU β | K400 β |
|
| 108 |
|---|---|---|---|---|---|---|---|
|
| 109 |
+
| **Motif Vision Encoder** | 0.5B | **74.0** | **80.5** | **83.5** | 87.4 | <u>52.0</u> | 87.4 |
|
| 110 |
| DINOv3 | 1.7B | <u>71.1</u> | <u>79.7</u> | <u>83.3</u> | 88.4 | **55.9** | <u>87.8</u> |
|
| 111 |
| Web-DINO | 2B | 57.2 | 65.8 | 69.5 | 85.9 | 42.7 | 86.8 |
|
| 112 |
| PEcore | 5.4B | 48.2 | 53.1 | 49.8 | **89.3** | 38.9 | **87.9** |
|
|
|
|
| 123 |
and stays competitive on the other image and video benchmarks, using roughly 1/3 of DINOv3's
|
| 124 |
training data (~0.5B samples).
|
| 125 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 126 |
|
| 127 |
## License
|
| 128 |
|
assets/dense_attention_comparison.png
DELETED
Git LFS Details
|
assets/haaland_attn_blk20.gif
DELETED
Git LFS Details
|
assets/{davis_mask_propagation.gif β haaland_full_attn_blk20.gif}
RENAMED
|
File without changes
|