| --- |
| license: cc-by-nc-4.0 |
| base_model: Qwen/Qwen3-VL-4B-Instruct |
| pipeline_tag: robotics |
| language: |
| - en |
| tags: |
| - camera-motion |
| - active-perception |
| - flow-matching |
| - qwen3-vl |
| --- |
| |
| # LIME: Learning Intent-aware Camera Motion from Egocentric Video |
|
|
| [Paper](https://arxiv.org/abs/2607.02417) · [Project page](https://boysun045.github.io/LIME-Page/) · [Code](https://github.com/cvg/LIME) |
|
|
| **CoRL 2026.** Given one RGB image and a free-form intent ("Find my bike"), LIME predicts |
| where the camera should move next: a relative target camera pose, and a description of |
| what the new view should reveal. |
|
|
| These are the `lime_v1` weights: a fine-tuned Qwen3-VL-4B-Instruct backbone and a |
| flow-matching pose head of about 20M parameters. Running them needs the code in the |
| [GitHub repository](https://github.com/cvg/LIME). |
|
|
| ## Usage |
|
|
| ```bash |
| git clone https://github.com/cvg/LIME && cd LIME |
| bash scripts/setup_env.sh # creates the conda environment and downloads these weights |
| conda activate lime |
| python scripts/predict.py --image examples/living_room.jpg --intent "Find my bike." --seed 0 |
| ``` |
|
|
| ```python |
| from lime_camera import LIME |
| |
| model = LIME.from_pretrained("weights/lime_v1") |
| prediction = model.predict("room.jpg", "Find my bike.", n_samples=5, seed=0) |
| prediction.observation_gain # what the new view should reveal |
| prediction.T_mean # 4x4 target pose, the mean of the 5 samples |
| ``` |
|
|
| Poses are the end camera in the start camera's frame, in metres, with OpenCV axes (x right, |
| y down, z forward). |
|
|
| ## Files |
|
|
| | File | What it is | |
| |---|---| |
| | `config.json`, `model-*.safetensors`, tokenizer and processor files | the fine-tuned Qwen3-VL-4B backbone | |
| | `flow_head.safetensors`, `flow_head_config.json` | the flow-matching pose head | |
| | `pose_norm_stats.json` | per-axis mean and std the head's poses are standardized by | |
|
|
| ## License |
|
|
| The weights are released under [CC BY-NC 4.0](LICENSE), for non-commercial use. They are a |
| modified version of [Qwen3-VL-4B-Instruct](https://huggingface.co/Qwen/Qwen3-VL-4B-Instruct), |
| which is licensed under Apache-2.0; its license and our notice are in |
| [LICENSE-QWEN](LICENSE-QWEN). |
|
|
| ## Citation |
|
|
| ```bibtex |
| @inproceedings{sun2026lime, |
| title={LIME: Learning Intent-aware Camera Motion from Egocentric Video}, |
| author={Sun, Boyang and Li, Jiajie and Yang, Yung-Hsu and Zhang, Chenyangguang |
| and Engelbracht, Tim and Hong, Sunghwan and Cadena, Cesar |
| and Pollefeys, Marc and Blum, Hermann}, |
| booktitle={Conference on Robot Learning (CoRL)}, |
| year={2026} |
| } |
| ``` |
|
|
