Add model card
Browse files
README.md
CHANGED
|
@@ -1,3 +1,98 @@
|
|
| 1 |
---
|
| 2 |
-
license:
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 3 |
---
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
---
|
| 2 |
+
license: agpl-3.0
|
| 3 |
+
library_name: pytorch
|
| 4 |
+
pipeline_tag: object-detection
|
| 5 |
+
tags:
|
| 6 |
+
- remote-sensing
|
| 7 |
+
- visual-grounding
|
| 8 |
+
- sar
|
| 9 |
+
- optical
|
| 10 |
+
- cross-domain
|
| 11 |
+
- mixture-of-experts
|
| 12 |
+
- contrastive-learning
|
| 13 |
+
- benchmark
|
| 14 |
---
|
| 15 |
+
|
| 16 |
+
# OptiSAR-Net++ — Official Weights
|
| 17 |
+
|
| 18 |
+
Official trained weights (`best.pt`) of **OptiSAR-Net++: A Large-Scale Benchmark and Transformer-Free Framework for Cross-Domain Remote Sensing Visual Grounding**.
|
| 19 |
+
|
| 20 |
+
OptiSAR-Net++ is a **transformer-free** framework for Cross-Domain Remote Sensing Visual Grounding (CD-RSVG): a single unified model localizes targets described by free-form natural language in **both optical and SAR** remote sensing imagery, replacing a heavy Transformer decoder with **contrastive region–text matching**.
|
| 21 |
+
|
| 22 |
+
- 🤗 **Dataset**: [JunDong-dev/OptSAR-RSVG](https://huggingface.co/datasets/JunDong-dev/OptSAR-RSVG)
|
| 23 |
+
- 💻 **Code**: [GitHub — JunDong-dev/OptiSAR-Net-PlusPlus](https://github.com/JunDong-dev/OptiSAR-Net-PlusPlus)
|
| 24 |
+
|
| 25 |
+
## 🏗️ Architecture
|
| 26 |
+
|
| 27 |
+
The checkpoint follows a YOLOE-style single-stage detector extended with:
|
| 28 |
+
|
| 29 |
+
| Component | Location | Role |
|
| 30 |
+
|---|---|---|
|
| 31 |
+
| **PLoRA-MoE** | Backbone | Patch-level low-rank-adaptation Mixture-of-Experts for optical/SAR feature disentanglement |
|
| 32 |
+
| **TGDG-SSA** | Neck (×3) | Language-guided multi-scale fusion of visual features with text embeddings |
|
| 33 |
+
| **OptiSARNetPlusPlusDetect** | Head | Region–text contrastive matching head with region-aware auxiliary supervision |
|
| 34 |
+
| MobileCLIP2-B (frozen) | Text encoder | Encodes referring expressions |
|
| 35 |
+
|
| 36 |
+
Key config of this checkpoint: `nc: 16` classes, `scale: m`, `reg_max: 16`.
|
| 37 |
+
|
| 38 |
+
## 📦 Files
|
| 39 |
+
|
| 40 |
+
| File | Description |
|
| 41 |
+
|---|---|
|
| 42 |
+
| `best.pt` | Trained weights (≈ 129 MB). The model architecture/modules are defined in the [GitHub repository](https://github.com/JunDong-dev/OptiSAR-Net-PlusPlus). |
|
| 43 |
+
|
| 44 |
+
## 📈 Results — OptSAR-RSVG Test Split
|
| 45 |
+
|
| 46 |
+
Evaluated on the [OptSAR-RSVG](https://huggingface.co/datasets/JunDong-dev/OptSAR-RSVG) test split (4,434 images / 8,103 referring expressions). All values in %.
|
| 47 |
+
|
| 48 |
+
| Domain | Samples | Pr@0.5 | Pr@0.6 | Pr@0.7 | Pr@0.8 | Pr@0.9 | meanIoU | cumIoU |
|
| 49 |
+
|---|---:|---:|---:|---:|---:|---:|---:|---:|
|
| 50 |
+
| **All** | 8,103 | **93.61** | 93.19 | 91.94 | 87.75 | 66.43 | **85.94** | **92.11** |
|
| 51 |
+
| Optical | 6,027 | 93.01 | 92.58 | 91.67 | 88.58 | 72.99 | 86.48 | 92.25 |
|
| 52 |
+
| SAR | 2,076 | 95.33 | 94.94 | 92.73 | 85.31 | 47.40 | 84.38 | 83.21 |
|
| 53 |
+
|
| 54 |
+
Benchmark comparisons against TransVG, LQVG, TACMT, CSDNet, Grounding DINO, GLIP, etc. are reported in the [paper](https://arxiv.org/abs/2603.24876) and the [GitHub README](https://github.com/JunDong-dev/OptiSAR-Net-PlusPlus).
|
| 55 |
+
|
| 56 |
+
## 🚀 Usage
|
| 57 |
+
|
| 58 |
+
`best.pt` contains **custom modules** (`PLoRA_MoE`, `TGDG_SSA`, `OptiSARNetPlusPlusDetect`), so it must be loaded with the project code rather than a stock Ultralytics install:
|
| 59 |
+
|
| 60 |
+
```bash
|
| 61 |
+
# 1. Clone the official repository (defines the custom modules & inference entry points)
|
| 62 |
+
git clone https://github.com/JunDong-dev/OptiSAR-Net-PlusPlus.git
|
| 63 |
+
cd OptiSAR-Net-PlusPlus
|
| 64 |
+
pip install -r requirements.txt
|
| 65 |
+
|
| 66 |
+
# 2. Download the checkpoint and the dataset
|
| 67 |
+
hf download JunDong-dev/OptiSAR-Net-PlusPlus best.pt --local-dir .
|
| 68 |
+
hf download JunDong-dev/OptSAR-RSVG --repo-type dataset --local-dir OptSAR-RSVG
|
| 69 |
+
|
| 70 |
+
# 3. Run evaluation / inference with the scripts provided in the repository README
|
| 71 |
+
```
|
| 72 |
+
|
| 73 |
+
Evaluation metrics follow the standard CD-RSVG protocol: Pr@{0.5–0.9}, meanIoU, cumIoU, with per-domain (`optical` / `sar`) reporting.
|
| 74 |
+
|
| 75 |
+
## 🎯 Intended Use
|
| 76 |
+
|
| 77 |
+
- Cross-domain (optical ↔ SAR) referring expression comprehension / visual grounding in remote sensing
|
| 78 |
+
- Research on multi-modal fusion, parameter-efficient MoE adaptation, and contrastive region–text matching
|
| 79 |
+
|
| 80 |
+
## ⚠️ Limitations
|
| 81 |
+
|
| 82 |
+
- Trained and evaluated only on the 16 categories of OptSAR-RSVG; out-of-distribution categories/imagery may degrade
|
| 83 |
+
- SAR performance at strict IoU thresholds (Pr@0.9) is notably lower than optical, reflecting SAR imaging characteristics
|
| 84 |
+
|
| 85 |
+
## ⚖️ License
|
| 86 |
+
|
| 87 |
+
Code and weights are released under **AGPL-3.0**. The companion dataset is subject to its source datasets' licenses (see the dataset card).
|
| 88 |
+
|
| 89 |
+
## 📚 Citation
|
| 90 |
+
|
| 91 |
+
```bibtex
|
| 92 |
+
@article{tang2026optisar,
|
| 93 |
+
title={OptiSAR-Net++: A Large-Scale Benchmark and Transformer-Free Framework for Cross-Domain Remote Sensing Visual Grounding},
|
| 94 |
+
author={Tang, Xiaoyu and Dong, Jun and Cheng, Jintao and Fan, Rui},
|
| 95 |
+
journal={arXiv preprint arXiv:2603.24876},
|
| 96 |
+
year={2026}
|
| 97 |
+
}
|
| 98 |
+
```
|