Model weights for **RVAS: Referring Video Active Exploration and Segmentation** (ICML 2026).

[Paper](https://openreview.net/forum?id=95wIQ6actx) · [Code](https://github.com/FudanCVL/RVAS) · [Dataset](https://huggingface.co/datasets/FudanCVL/RVAS)

LESA (Language Environment-aware Segmentation Assistant) performs action planning and referring video segmentation. A lightweight controller invokes the multimodal language model as frames arrive. The generated `[SEG]` embedding conditions a shared SAM2 model for segmentation and tracking, with backward mask revision when new evidence becomes available.

## Usage

Follow the [installation instructions](https://github.com/FudanCVL/RVAS#installation), then download the model from the repository root:

```python
from huggingface_hub import snapshot_download

snapshot_download('FudanCVL/LESA', local_dir='weights')
```

The package contains four safetensors shards, model configuration, and tokenizer files. LoRA parameters are merged, and SAM2 is included. Model classes are provided by the source repository.

Run inference on a directory of ordered RGB frames:

```bash
CUDA_VISIBLE_DEVICES=0 python -m lesa.infer \
  --frames data/example/frames \
  --expression 'The person wearing a blue shirt.' \
  --video-id example --exp-id 1 \
  --output outputs/example
```

The entry point uses the default command-line settings and loads the checkpoint in `weights/`. See the source repository for [RVAS data preparation](https://github.com/FudanCVL/RVAS#dataset-preparation), [training](https://github.com/FudanCVL/RVAS#training), [benchmark inference](https://github.com/FudanCVL/RVAS#rvas-benchmark), and [evaluation](https://github.com/FudanCVL/RVAS#evaluation).

## Citation

```bibtex
@inproceedings{hu2026rvas,
  title     = {RVAS: Referring Video Active Exploration and Segmentation},
  author    = {Hu, Hengrui and Gao, Weiwei and Zhang, Zipei and Ding, Henghui},
  booktitle = {Proceedings of the International Conference on Machine Learning},
  year      = {2026}
}
```

## License

See [LICENSE](LICENSE) and the source repository's [acknowledgements](https://github.com/FudanCVL/RVAS#acknowledgements-and-license). Upstream components and pretrained weights remain subject to their respective licenses and terms.