metadata
license: cc-by-nc-4.0
library_name: pytorch
tags:
- material-classification
- multimodal-fusion
- robotics
SegObjectMaterial fusion heads
Trained heads that classify an object's material from frozen vision + language embeddings. Each folder holds one head with its weights, training config, held-out split, evaluation metrics and model card.
| head | dataset | encoders | fusion | mIoU | mAcc |
|---|---|---|---|---|---|
DMS/dinov2_clip_gated |
DMS | DINOv2-CLIP | Gated Multimodal Unit | 0.4543 | 0.6373 |
DMS/dinov2_clip_mlp |
DMS | DINOv2-CLIP | Concat + MLP | 0.4744 | 0.6672 |
DMS/dinov3_siglip2_gated |
DMS | DINOv3-SigLIP2 | Gated Multimodal Unit | 0.4630 | 0.6386 |
DMS/dinov3_siglip2_gated_reg |
DMS | DINOv3-SigLIP2 | Gated Multimodal Unit | 0.4606 | 0.6403 |
DMS/dinov3_siglip2_lmf |
DMS | DINOv3-SigLIP2 | Low-rank Multimodal Fusion | 0.4407 | 0.6181 |
DMS/dinov3_siglip2_lmf_reg |
DMS | DINOv3-SigLIP2 | Low-rank Multimodal Fusion | 0.4453 | 0.6135 |
DMS/dinov3_siglip2_mlp |
DMS | DINOv3-SigLIP2 | Concat + MLP | 0.4592 | 0.6453 |
FMD/dinov2_clip_gated |
FMD | DINOv2-CLIP | Gated Multimodal Unit | 0.9234 | 0.9599 |
FMD/dinov2_clip_mlp |
FMD | DINOv2-CLIP | Concat + MLP | 0.9612 | 0.9802 |
FMD/dinov3_siglip2_gated |
FMD | DINOv3-SigLIP2 | Gated Multimodal Unit | 0.9707 | 0.9850 |
FMD/dinov3_siglip2_lmf |
FMD | DINOv3-SigLIP2 | Low-rank Multimodal Fusion | 0.9805 | 0.9900 |
FMD/dinov3_siglip2_mlp |
FMD | DINOv3-SigLIP2 | Concat + MLP | 0.9519 | 0.9747 |
Every row is a single training run; see each folder's README for caveats.