| Model weights for **RVAS: Referring Video Active Exploration and Segmentation** (ICML 2026). |
|
|
| [Paper](https://openreview.net/forum?id=95wIQ6actx) · [Code](https://github.com/FudanCVL/RVAS) · [Dataset](https://huggingface.co/datasets/FudanCVL/RVAS) |
|
|
| LESA (Language Environment-aware Segmentation Assistant) performs action planning and referring video segmentation. A lightweight controller invokes the multimodal language model as frames arrive. The generated `[SEG]` embedding conditions a shared SAM2 model for segmentation and tracking, with backward mask revision when new evidence becomes available. |
|
|
| ## Usage |
|
|
| Follow the [installation instructions](https://github.com/FudanCVL/RVAS#installation), then download the model from the repository root: |
|
|
| ```python |
| from huggingface_hub import snapshot_download |
| |
| snapshot_download('FudanCVL/LESA', local_dir='weights') |
| ``` |
|
|
| The package contains four safetensors shards, model configuration, and tokenizer files. LoRA parameters are merged, and SAM2 is included. Model classes are provided by the source repository. |
|
|
| Run inference on a directory of ordered RGB frames: |
|
|
| ```bash |
| CUDA_VISIBLE_DEVICES=0 python -m lesa.infer \ |
| --frames data/example/frames \ |
| --expression 'The person wearing a blue shirt.' \ |
| --video-id example --exp-id 1 \ |
| --output outputs/example |
| ``` |
|
|
| The entry point uses the default command-line settings and loads the checkpoint in `weights/`. See the source repository for [RVAS data preparation](https://github.com/FudanCVL/RVAS#dataset-preparation), [training](https://github.com/FudanCVL/RVAS#training), [benchmark inference](https://github.com/FudanCVL/RVAS#rvas-benchmark), and [evaluation](https://github.com/FudanCVL/RVAS#evaluation). |
|
|
| ## Citation |
|
|
| ```bibtex |
| @inproceedings{hu2026rvas, |
| title = {RVAS: Referring Video Active Exploration and Segmentation}, |
| author = {Hu, Hengrui and Gao, Weiwei and Zhang, Zipei and Ding, Henghui}, |
| booktitle = {Proceedings of the International Conference on Machine Learning}, |
| year = {2026} |
| } |
| ``` |
|
|
| ## License |
|
|
| See [LICENSE](LICENSE) and the source repository's [acknowledgements](https://github.com/FudanCVL/RVAS#acknowledgements-and-license). Upstream components and pretrained weights remain subject to their respective licenses and terms. |
|
|
