| --- |
| license: cc-by-nc-4.0 |
| library_name: pytorch |
| tags: |
| - material-classification |
| - multimodal-fusion |
| - robotics |
| --- |
| |
| # SegObjectMaterial fusion heads |
|
|
| Trained heads that classify an object's material from frozen vision + language |
| embeddings. Each folder holds one head with its weights, training config, |
| held-out split, evaluation metrics and model card. |
|
|
| | head | dataset | encoders | fusion | mIoU | mAcc | |
| |---|---|---|---|---|---| |
| | [`DMS/dinov2_clip_gated`](DMS/dinov2_clip_gated) | DMS | DINOv2-CLIP | Gated Multimodal Unit | 0.4543 | 0.6373 | |
| | [`DMS/dinov2_clip_mlp`](DMS/dinov2_clip_mlp) | DMS | DINOv2-CLIP | Concat + MLP | 0.4744 | 0.6672 | |
| | [`DMS/dinov3_siglip2_gated`](DMS/dinov3_siglip2_gated) | DMS | DINOv3-SigLIP2 | Gated Multimodal Unit | 0.4630 | 0.6386 | |
| | [`DMS/dinov3_siglip2_gated_reg`](DMS/dinov3_siglip2_gated_reg) | DMS | DINOv3-SigLIP2 | Gated Multimodal Unit | 0.4606 | 0.6403 | |
| | [`DMS/dinov3_siglip2_lmf`](DMS/dinov3_siglip2_lmf) | DMS | DINOv3-SigLIP2 | Low-rank Multimodal Fusion | 0.4407 | 0.6181 | |
| | [`DMS/dinov3_siglip2_lmf_reg`](DMS/dinov3_siglip2_lmf_reg) | DMS | DINOv3-SigLIP2 | Low-rank Multimodal Fusion | 0.4453 | 0.6135 | |
| | [`DMS/dinov3_siglip2_mlp`](DMS/dinov3_siglip2_mlp) | DMS | DINOv3-SigLIP2 | Concat + MLP | 0.4592 | 0.6453 | |
| | [`FMD/dinov2_clip_gated`](FMD/dinov2_clip_gated) | FMD | DINOv2-CLIP | Gated Multimodal Unit | 0.9234 | 0.9599 | |
| | [`FMD/dinov2_clip_mlp`](FMD/dinov2_clip_mlp) | FMD | DINOv2-CLIP | Concat + MLP | 0.9612 | 0.9802 | |
| | [`FMD/dinov3_siglip2_gated`](FMD/dinov3_siglip2_gated) | FMD | DINOv3-SigLIP2 | Gated Multimodal Unit | 0.9707 | 0.9850 | |
| | [`FMD/dinov3_siglip2_lmf`](FMD/dinov3_siglip2_lmf) | FMD | DINOv3-SigLIP2 | Low-rank Multimodal Fusion | 0.9805 | 0.9900 | |
| | [`FMD/dinov3_siglip2_mlp`](FMD/dinov3_siglip2_mlp) | FMD | DINOv3-SigLIP2 | Concat + MLP | 0.9519 | 0.9747 | |
|
|
| Every row is a single training run; see each folder's README for caveats. |
|
|
