Image Feature Extraction
Transformers
Safetensors
motif_vision
feature-extraction
motif
vision-transformer
self-supervised
video
custom_code
Instructions to use Motif-Technologies/Motif-Vision-Encoder with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use Motif-Technologies/Motif-Vision-Encoder with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("image-feature-extraction", model="Motif-Technologies/Motif-Vision-Encoder", trust_remote_code=True)# pip install -U transformers accelerate # Load model directly from transformers import AutoModel model = AutoModel.from_pretrained("Motif-Technologies/Motif-Vision-Encoder", trust_remote_code=True, device_map="auto") - Notebooks
- Google Colab
- Kaggle
docs: brand name MOTIF -> Motif (title, intro, citation, docstrings)
Browse files- README.md +3 -3
- modeling_motif_vision_encoder.py +6 -6
README.md
CHANGED
|
@@ -11,9 +11,9 @@ tags:
|
|
| 11 |
- custom_code
|
| 12 |
---
|
| 13 |
|
| 14 |
-
#
|
| 15 |
|
| 16 |
-
|
| 17 |
backbone. A single **3D-convolutional tokenizer** ingests both modalities — an image is a
|
| 18 |
1-frame clip (`T=1`), a video is `T>1` — so the same weights produce **dense patch-level
|
| 19 |
features** and a **language-aligned global (CLS) representation**.
|
|
@@ -116,7 +116,7 @@ respective dataset licenses; downstream users are responsible for compliance wit
|
|
| 116 |
|
| 117 |
```bibtex
|
| 118 |
@misc{motif_vision_encoder,
|
| 119 |
-
title = {
|
| 120 |
author = {Motif Technologies},
|
| 121 |
year = {2026},
|
| 122 |
howpublished = {\url{https://huggingface.co/Motif-Technologies/Motif-Vision-Encoder-7B}}
|
|
|
|
| 11 |
- custom_code
|
| 12 |
---
|
| 13 |
|
| 14 |
+
# Motif Vision Encoder
|
| 15 |
|
| 16 |
+
Motif Vision Encoder is a **unified image + video** self-supervised vision encoder on a ViT
|
| 17 |
backbone. A single **3D-convolutional tokenizer** ingests both modalities — an image is a
|
| 18 |
1-frame clip (`T=1`), a video is `T>1` — so the same weights produce **dense patch-level
|
| 19 |
features** and a **language-aligned global (CLS) representation**.
|
|
|
|
| 116 |
|
| 117 |
```bibtex
|
| 118 |
@misc{motif_vision_encoder,
|
| 119 |
+
title = {Motif Vision Encoder: a unified image/video self-supervised vision encoder},
|
| 120 |
author = {Motif Technologies},
|
| 121 |
year = {2026},
|
| 122 |
howpublished = {\url{https://huggingface.co/Motif-Technologies/Motif-Vision-Encoder-7B}}
|
modeling_motif_vision_encoder.py
CHANGED
|
@@ -1,7 +1,7 @@
|
|
| 1 |
# Copyright (c) Motif Technologies.
|
| 2 |
-
# Self-contained inference model for
|
| 3 |
# Auto-assembled from the training repo's inference path; NO training code.
|
| 4 |
-
"""
|
| 5 |
|
| 6 |
Usage:
|
| 7 |
from transformers import AutoModel
|
|
@@ -194,7 +194,7 @@ class PatchEmbed(nn.Module):
|
|
| 194 |
class RopePositionEmbedding3D(nn.Module):
|
| 195 |
"""Full 3D axial RoPE with independent T/H/W frequency bands.
|
| 196 |
|
| 197 |
-
Unlike the original
|
| 198 |
across temporal frames (making temporal positions indistinguishable), this
|
| 199 |
implementation partitions the head dimension into three axis groups:
|
| 200 |
|
|
@@ -354,7 +354,7 @@ class RopePositionEmbedding3D(nn.Module):
|
|
| 354 |
token_angles_w = angles_w[w_idx] # [T*H*W, D_W//2]
|
| 355 |
angles_half = torch.cat([token_angles_t, token_angles_h, token_angles_w], dim=-1) # [T*H*W, D_head//2]
|
| 356 |
|
| 357 |
-
# tile(2) on full concat — matches
|
| 358 |
# This ensures rotate_half pairs (dim i ↔ dim i+D//2) have identical angles,
|
| 359 |
# making the rotation orthogonal (preserves dot products in attention).
|
| 360 |
angles = angles_half.tile(2) # [T*H*W, D_head]
|
|
@@ -1347,7 +1347,7 @@ class MotifVisionTransformer(nn.Module):
|
|
| 1347 |
# HuggingFace transformers wrapper (inference-only)
|
| 1348 |
# ============================================================================
|
| 1349 |
class MotifVisionConfig(PretrainedConfig):
|
| 1350 |
-
"""Config for the
|
| 1351 |
|
| 1352 |
model_type = "motif_vision"
|
| 1353 |
|
|
@@ -1407,7 +1407,7 @@ class MotifVisionConfig(PretrainedConfig):
|
|
| 1407 |
|
| 1408 |
|
| 1409 |
class MotifVisionModel(PreTrainedModel):
|
| 1410 |
-
"""
|
| 1411 |
|
| 1412 |
config_class = MotifVisionConfig
|
| 1413 |
base_model_prefix = "motif"
|
|
|
|
| 1 |
# Copyright (c) Motif Technologies.
|
| 2 |
+
# Self-contained inference model for Motif Vision Encoder (image + video).
|
| 3 |
# Auto-assembled from the training repo's inference path; NO training code.
|
| 4 |
+
"""Motif Vision Encoder — unified image/video ViT backbone (inference-only).
|
| 5 |
|
| 6 |
Usage:
|
| 7 |
from transformers import AutoModel
|
|
|
|
| 194 |
class RopePositionEmbedding3D(nn.Module):
|
| 195 |
"""Full 3D axial RoPE with independent T/H/W frequency bands.
|
| 196 |
|
| 197 |
+
Unlike the original Motif implementation which simply repeats 2D spatial angles
|
| 198 |
across temporal frames (making temporal positions indistinguishable), this
|
| 199 |
implementation partitions the head dimension into three axis groups:
|
| 200 |
|
|
|
|
| 354 |
token_angles_w = angles_w[w_idx] # [T*H*W, D_W//2]
|
| 355 |
angles_half = torch.cat([token_angles_t, token_angles_h, token_angles_w], dim=-1) # [T*H*W, D_head//2]
|
| 356 |
|
| 357 |
+
# tile(2) on full concat — matches Motif 2D RoPE pattern
|
| 358 |
# This ensures rotate_half pairs (dim i ↔ dim i+D//2) have identical angles,
|
| 359 |
# making the rotation orthogonal (preserves dot products in attention).
|
| 360 |
angles = angles_half.tile(2) # [T*H*W, D_head]
|
|
|
|
| 1347 |
# HuggingFace transformers wrapper (inference-only)
|
| 1348 |
# ============================================================================
|
| 1349 |
class MotifVisionConfig(PretrainedConfig):
|
| 1350 |
+
"""Config for the Motif Vision Encoder backbone (image + video)."""
|
| 1351 |
|
| 1352 |
model_type = "motif_vision"
|
| 1353 |
|
|
|
|
| 1407 |
|
| 1408 |
|
| 1409 |
class MotifVisionModel(PreTrainedModel):
|
| 1410 |
+
"""Motif Vision Encoder for HF `AutoModel` (inference). Returns dense + CLS features."""
|
| 1411 |
|
| 1412 |
config_class = MotifVisionConfig
|
| 1413 |
base_model_prefix = "motif"
|