README.md

---
license: apache-2.0
language:
  - en
base_model: Qwen/Qwen3.5-9B
pipeline_tag: image-text-to-text
library_name: transformers
tags:
  - vision-language-model
  - spatial-reasoning
  - perspective-taking
  - reference-frame
  - tool-use
  - grpo
  - qwen3.5
---

<div align="center">

# LeRF-9B

### LeRF: Learning Reference Coordinate Frames for Perspective Taking Reasoning

**Bang Xiao**<sup>1,2</sup> · **Wenqi Jia**<sup>1</sup> · **Ozgur Kara**<sup>1</sup> · **Tiancheng Shen**<sup>3</sup> · **Yibo Yang**<sup>4</sup>


**Bolin Lai**<sup>1,5,^</sup> · **Junho Kim**<sup>1,^</sup> · **James Matthew Rehg**<sup>1,^</sup>

<sup>1</sup>University of Illinois Urbana-Champaign · <sup>2</sup>Zhiyuan College, Shanghai Jiao Tong University


<sup>3</sup>University of California, Merced · <sup>4</sup>Shanghai Jiao Tong University · <sup>5</sup>Amazon AGI


<sup>^</sup>Corresponding authors

[🏠 Homepage](https://lerf-project.github.io/) · [📄 Paper](https://arxiv.org/abs/2609.36219) · [💻 Code](https://github.com/bangx7/LeRF) · [🤗 Collection](https://huggingface.co/collections/SamuelBang/lerf) 

</div>


## Model Summary

Vision-language models (VLMs) struggle with **perspective taking**: when a question asks about
spatial relations from another entity's or an imagined observer's viewpoint, they often fall back
to the camera view. **LeRF** trains a VLM to build and use an explicit **entity-centered reference
frame**. Given an image and a question, the model decides whether a frame is needed. If it is, it
grounds the reference entity and predicts the frame's projected origin and its **front / left / up**
axes. A lightweight renderer draws the frame on the image, and the model reasons over this visual
cue. No external perception models or 3D reconstruction are used.


## How It Works

**Two-turn inference.**

1. **Turn 1 (thinking off).** The model outputs either a single `draw_reference_frame` call —
   `reference_object` (text), plus `origin`, `axis_front`, `axis_left`, `axis_up` as `[x, y]`
   points in `[0, 1000]` normalized image coordinates — or the literal string `NO_TOOL_CALL`
   when the question can be answered from the camera view.
2. **Tool.** The frame is rendered on the image (red = front, green = left, blue = up; a `⊙`/`⊗`
   depth marker replaces a strongly foreshortened axis).
3. **Turn 2 (thinking on).** The model sees the rendered image (not the numeric coordinates),
   reasons over it, and answers in `\boxed{}`.

**Training.**

- **Stage I: SFT.** Learns reference-entity grounding and projected frame prediction from pose
  datasets (ImageNet3D, Omni6DPose-SOPE, BEDLAM). About 20% of the data are no-tool-call examples
  that teach selective tool use.
- **Stage II: RL (GRPO).** Trained on MultihopSpatial and SpatialReasoner-RL with a binary
  final-answer reward. The frame-prediction turn is
  excluded from the policy gradient, so only the reasoning/answer turn receives the advantage.

## Usage

The model is meant to be used together with the `draw_reference_frame` tool and the two-turn
client from the [GitHub repository](https://github.com/bangx7/LeRF). The system prompt and
tool schema live in `tool/system_prompt.txt` and `tool/prompts.py`.

**1. Serve with vLLM** (OpenAI-compatible API):

```bash
vllm serve SamuelBang/LeRF-9B --served-model-name lerf --max-model-len 32768 \
    --limit-mm-per-prompt '{"image": 2}' --trust-remote-code
```

**2. Run the client:**

```bash
git clone https://github.com/bangx7/LeRF.git && cd LeRF/tool
pip install pillow requests

# single question
python inference.py --model lerf --image example.jpg \
    --question "From the man's perspective, which object is on his left?" \
    --options cup bed laptop phone --save_renders renders/

# batch: one JSON per line with image / question / options (and optionally answer, id)
python inference.py --model lerf --input questions.jsonl --output results.jsonl
```

Recommended settings (matching training): temperature 0.6, images downscaled to at most 1M pixels,
a 10,240-token thinking budget within a 12,288-token answer turn.

Tested with Python 3.12, CUDA 13.0, PyTorch 2.11, vLLM 0.24.0 and transformers 5.10.4.

## Evaluation

Accuracy (%) on OmniSpatial perspective taking (OmniSpatial-PT: Ego / Allo / Hypo), 3DSRBench
(Orientation / Multi-Object) and ViewSpatial-Bench (person-perspective Object View Orientation /
Relative Direction). **Bold** marks the best open-source result.

| Method | Ego | Allo | Hypo | 3DSR Ori | 3DSR M-Obj | VS P-Obj | VS P-Rel |
|---|:-:|:-:|:-:|:-:|:-:|:-:|:-:|
| *Proprietary* | | | | | | | |
| GPT-5.6-Luna (medium) | 83.33 | 49.73 | 45.78 | 60.04 | 55.34 | 46.99 | 70.07 |
| GPT-5.6-Terra (medium) | 81.37 | 55.85 | 53.01 | 63.32 | 56.39 | 45.08 | 77.20 |
| Claude Sonnet 5 (medium) | 80.39 | 42.55 | 49.40 | 34.94 | 44.77 | 51.31 | 51.43 |
| Claude Sonnet 5 (high) | 84.31 | 48.14 | 45.78 | 43.15 | 46.81 | 51.51 | 60.10 |
| *Open-source* | | | | | | | |
| Qwen3.5-4B | 74.71 | 42.55 | 44.34 | 42.28 | 43.26 | 51.01 | 57.43 |
| Qwen3.5-9B | **80.20** | 47.13 | 44.58 | 48.17 | 48.66 | 56.23 | 65.51 |
| Qwen3.5-9B + APC | 42.16 | 27.66 | 30.12 | 44.98 | 33.22 | 59.34 | 37.53 |
| SpatialReasoner | 40.39 | 35.11 | 35.66 | 52.05 | **50.64** | 42.37 | 45.61 |
| *Ours* | | | | | | | |
| LeRF-4B | 72.35 | 49.36 | 46.75 | 45.88 | 44.55 | 56.26 | 67.85 |
| **LeRF-9B (this model)** | 74.31 | **54.04** | **55.66** | **53.76** | 50.29 | **61.91** | **74.23** |

See the paper for more baselines, reference-frame estimation accuracy and ablations.


## Acknowledgements

Built on [Qwen3.5](https://huggingface.co/Qwen), [LLaMA-Factory](https://github.com/hiyouga/LLaMA-Factory)
and [verl](https://github.com/verl-project/verl).

## Citation

```bibtex
@misc{xiao2026lerflearningreferencecoordinate,
      title={LeRF: Learning Reference Coordinate Frames for Perspective Taking Reasoning}, 
      author={Bang Xiao and Wenqi Jia and Ozgur Kara and Tiancheng Shen and Yibo Yang and Bolin Lai and Junho Kim and James Matthew Rehg},
      year={2026},
      eprint={2609.36219},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2609.36219}, 
}
```