README.md

---
license: cc-by-nc-4.0
library_name: pytorch
tags:
- material-classification
- multimodal-fusion
- robotics
---

# SegObjectMaterial fusion heads

Trained heads that classify an object's material from frozen vision + language
embeddings. Each folder holds one head with its weights, training config,
held-out split, evaluation metrics and model card.

| head | dataset | encoders | fusion | mIoU | mAcc |
|---|---|---|---|---|---|
| [`DMS/dinov2_clip_gated`](DMS/dinov2_clip_gated) | DMS | DINOv2-CLIP | Gated Multimodal Unit | 0.4543 | 0.6373 |
| [`DMS/dinov2_clip_mlp`](DMS/dinov2_clip_mlp) | DMS | DINOv2-CLIP | Concat + MLP | 0.4744 | 0.6672 |
| [`DMS/dinov3_siglip2_gated`](DMS/dinov3_siglip2_gated) | DMS | DINOv3-SigLIP2 | Gated Multimodal Unit | 0.4630 | 0.6386 |
| [`DMS/dinov3_siglip2_gated_reg`](DMS/dinov3_siglip2_gated_reg) | DMS | DINOv3-SigLIP2 | Gated Multimodal Unit | 0.4606 | 0.6403 |
| [`DMS/dinov3_siglip2_lmf`](DMS/dinov3_siglip2_lmf) | DMS | DINOv3-SigLIP2 | Low-rank Multimodal Fusion | 0.4407 | 0.6181 |
| [`DMS/dinov3_siglip2_lmf_reg`](DMS/dinov3_siglip2_lmf_reg) | DMS | DINOv3-SigLIP2 | Low-rank Multimodal Fusion | 0.4453 | 0.6135 |
| [`DMS/dinov3_siglip2_mlp`](DMS/dinov3_siglip2_mlp) | DMS | DINOv3-SigLIP2 | Concat + MLP | 0.4592 | 0.6453 |
| [`FMD/dinov2_clip_gated`](FMD/dinov2_clip_gated) | FMD | DINOv2-CLIP | Gated Multimodal Unit | 0.9234 | 0.9599 |
| [`FMD/dinov2_clip_mlp`](FMD/dinov2_clip_mlp) | FMD | DINOv2-CLIP | Concat + MLP | 0.9612 | 0.9802 |
| [`FMD/dinov3_siglip2_gated`](FMD/dinov3_siglip2_gated) | FMD | DINOv3-SigLIP2 | Gated Multimodal Unit | 0.9707 | 0.9850 |
| [`FMD/dinov3_siglip2_lmf`](FMD/dinov3_siglip2_lmf) | FMD | DINOv3-SigLIP2 | Low-rank Multimodal Fusion | 0.9805 | 0.9900 |
| [`FMD/dinov3_siglip2_mlp`](FMD/dinov3_siglip2_mlp) | FMD | DINOv3-SigLIP2 | Concat + MLP | 0.9519 | 0.9747 |

Every row is a single training run; see each folder's README for caveats.