Instructions to use SamuelBang/LeRF-9B with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use SamuelBang/LeRF-9B with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("image-text-to-text", model="SamuelBang/LeRF-9B")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# pip install -U transformers accelerate
# Load model directly
from transformers import AutoProcessor, AutoModelForMultimodalLM

processor = AutoProcessor.from_pretrained("SamuelBang/LeRF-9B")
model = AutoModelForMultimodalLM.from_pretrained("SamuelBang/LeRF-9B", device_map="auto")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
inputs = processor.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=256)
print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use SamuelBang/LeRF-9B with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "SamuelBang/LeRF-9B"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "SamuelBang/LeRF-9B",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker

docker model run hf.co/SamuelBang/LeRF-9B

SGLang

How to use SamuelBang/LeRF-9B with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "SamuelBang/LeRF-9B" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "SamuelBang/LeRF-9B",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "SamuelBang/LeRF-9B" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "SamuelBang/LeRF-9B",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Docker Model Runner
How to use SamuelBang/LeRF-9B with Docker Model Runner:
```
docker model run hf.co/SamuelBang/LeRF-9B
```

LeRF-9B / README.md

SamuelBang

Update README.md

43fcd6e verified 22 days ago

preview code

Raw History Blame Contribute Delete

6.2 kB

---
license: apache-2.0
language:
  - en
base_model: Qwen/Qwen3.5-9B
pipeline_tag: image-text-to-text
library_name: transformers
tags:
  - vision-language-model
  - spatial-reasoning
  - perspective-taking
  - reference-frame
  - tool-use
  - grpo
  - qwen3.5
---
<div align="center">

# LeRF-9B

### LeRF: Learning Reference Coordinate Frames for Perspective Taking Reasoning

**Bang Xiao**<sup>1,2</sup> · **Wenqi Jia**<sup>1</sup> · **Ozgur Kara**<sup>1</sup> · **Tiancheng Shen**<sup>3</sup> · **Yibo Yang**<sup>4</sup>


**Bolin Lai**<sup>1,5,^</sup> · **Junho Kim**<sup>1,^</sup> · **James Matthew Rehg**<sup>1,^</sup>

<sup>1</sup>University of Illinois Urbana-Champaign · <sup>2</sup>Zhiyuan College, Shanghai Jiao Tong University


<sup>3</sup>University of California, Merced · <sup>4</sup>Shanghai Jiao Tong University · <sup>5</sup>Amazon AGI


<sup>^</sup>Corresponding authors

[🏠 Homepage](https://lerf-project.github.io/) · [📄 Paper](https://arxiv.org/abs/2609.36219) · [💻 Code](https://github.com/bangx7/LeRF) · [🤗 Collection](https://huggingface.co/collections/SamuelBang/lerf) 

</div>


## Model Summary

Vision-language models (VLMs) struggle with **perspective taking**: when a question asks about
spatial relations from another entity's or an imagined observer's viewpoint, they often fall back
to the camera view. **LeRF** trains a VLM to build and use an explicit **entity-centered reference
frame**. Given an image and a question, the model decides whether a frame is needed. If it is, it
grounds the reference entity and predicts the frame's projected origin and its **front / left / up**
axes. A lightweight renderer draws the frame on the image, and the model reasons over this visual
cue. No external perception models or 3D reconstruction are used.


## How It Works

**Two-turn inference.**

1. **Turn 1 (thinking off).** The model outputs either a single `draw_reference_frame` call —
   `reference_object` (text), plus `origin`, `axis_front`, `axis_left`, `axis_up` as `[x, y]`
   points in `[0, 1000]` normalized image coordinates — or the literal string `NO_TOOL_CALL`
   when the question can be answered from the camera view.
2. **Tool.** The frame is rendered on the image (red = front, green = left, blue = up; a `⊙`/`⊗`
   depth marker replaces a strongly foreshortened axis).
3. **Turn 2 (thinking on).** The model sees the rendered image (not the numeric coordinates),
   reasons over it, and answers in `\boxed{}`.

**Training.**

- **Stage I: SFT.** Learns reference-entity grounding and projected frame prediction from pose
  datasets (ImageNet3D, Omni6DPose-SOPE, BEDLAM). About 20% of the data are no-tool-call examples
  that teach selective tool use.
- **Stage II: RL (GRPO).** Trained on MultihopSpatial and SpatialReasoner-RL with a binary
  final-answer reward. The frame-prediction turn is
  excluded from the policy gradient, so only the reasoning/answer turn receives the advantage.

## Usage

The model is meant to be used together with the `draw_reference_frame` tool and the two-turn
client from the [GitHub repository](https://github.com/bangx7/LeRF). The system prompt and
tool schema live in `tool/system_prompt.txt` and `tool/prompts.py`.

**1. Serve with vLLM** (OpenAI-compatible API):

```bash
vllm serve SamuelBang/LeRF-9B --served-model-name lerf --max-model-len 32768 \
    --limit-mm-per-prompt '{"image": 2}' --trust-remote-code
```

**2. Run the client:**

```bash
git clone https://github.com/bangx7/LeRF.git && cd LeRF/tool
pip install pillow requests
# single question
python inference.py --model lerf --image example.jpg \
    --question "From the man's perspective, which object is on his left?" \
    --options cup bed laptop phone --save_renders renders/
# batch: one JSON per line with image / question / options (and optionally answer, id)
python inference.py --model lerf --input questions.jsonl --output results.jsonl
```

Recommended settings (matching training): temperature 0.6, images downscaled to at most 1M pixels,
a 10,240-token thinking budget within a 12,288-token answer turn.

Tested with Python 3.12, CUDA 13.0, PyTorch 2.11, vLLM 0.24.0 and transformers 5.10.4.

## Evaluation

Accuracy (%) on OmniSpatial perspective taking (OmniSpatial-PT: Ego / Allo / Hypo), 3DSRBench
(Orientation / Multi-Object) and ViewSpatial-Bench (person-perspective Object View Orientation /
Relative Direction). **Bold** marks the best open-source result.

| Method | Ego | Allo | Hypo | 3DSR Ori | 3DSR M-Obj | VS P-Obj | VS P-Rel |
|---|:-:|:-:|:-:|:-:|:-:|:-:|:-:|
| *Proprietary* | | | | | | | |
| GPT-5.6-Luna (medium) | 83.33 | 49.73 | 45.78 | 60.04 | 55.34 | 46.99 | 70.07 |
| GPT-5.6-Terra (medium) | 81.37 | 55.85 | 53.01 | 63.32 | 56.39 | 45.08 | 77.20 |
| Claude Sonnet 5 (medium) | 80.39 | 42.55 | 49.40 | 34.94 | 44.77 | 51.31 | 51.43 |
| Claude Sonnet 5 (high) | 84.31 | 48.14 | 45.78 | 43.15 | 46.81 | 51.51 | 60.10 |
| *Open-source* | | | | | | | |
| Qwen3.5-4B | 74.71 | 42.55 | 44.34 | 42.28 | 43.26 | 51.01 | 57.43 |
| Qwen3.5-9B | **80.20** | 47.13 | 44.58 | 48.17 | 48.66 | 56.23 | 65.51 |
| Qwen3.5-9B + APC | 42.16 | 27.66 | 30.12 | 44.98 | 33.22 | 59.34 | 37.53 |
| SpatialReasoner | 40.39 | 35.11 | 35.66 | 52.05 | **50.64** | 42.37 | 45.61 |
| *Ours* | | | | | | | |
| LeRF-4B | 72.35 | 49.36 | 46.75 | 45.88 | 44.55 | 56.26 | 67.85 |
| **LeRF-9B (this model)** | 74.31 | **54.04** | **55.66** | **53.76** | 50.29 | **61.91** | **74.23** |

See the paper for more baselines, reference-frame estimation accuracy and ablations.


## Acknowledgements

Built on [Qwen3.5](https://huggingface.co/Qwen), [LLaMA-Factory](https://github.com/hiyouga/LLaMA-Factory)
and [verl](https://github.com/verl-project/verl).

## Citation

```bibtex
@misc{xiao2026lerflearningreferencecoordinate,
      title={LeRF: Learning Reference Coordinate Frames for Perspective Taking Reasoning}, 
      author={Bang Xiao and Wenqi Jia and Ozgur Kara and Tiancheng Shen and Yibo Yang and Bolin Lai and Junho Kim and James Matthew Rehg},
      year={2026},
      eprint={2609.36219},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2609.36219}, 
}
```

	---
	license: apache-2.0
	language:
	- en
	base_model: Qwen/Qwen3.5-9B
	pipeline_tag: image-text-to-text
	library_name: transformers
	tags:
	- vision-language-model
	- spatial-reasoning
	- perspective-taking
	- reference-frame
	- tool-use
	- grpo
	- qwen3.5
	---

	<div align="center">

	# LeRF-9B

	### LeRF: Learning Reference Coordinate Frames for Perspective Taking Reasoning

	Bang Xiao<sup>1,2</sup> · Wenqi Jia<sup>1</sup> · Ozgur Kara<sup>1</sup> · Tiancheng Shen<sup>3</sup> · Yibo Yang<sup>4</sup>


	Bolin Lai<sup>1,5,^</sup> · Junho Kim<sup>1,^</sup> · James Matthew Rehg<sup>1,^</sup>

	<sup>1</sup>University of Illinois Urbana-Champaign · <sup>2</sup>Zhiyuan College, Shanghai Jiao Tong University


	<sup>3</sup>University of California, Merced · <sup>4</sup>Shanghai Jiao Tong University · <sup>5</sup>Amazon AGI


	<sup>^</sup>Corresponding authors

	[🏠 Homepage](https://lerf-project.github.io/) · [📄 Paper](https://arxiv.org/abs/2609.36219) · [💻 Code](https://github.com/bangx7/LeRF) · [🤗 Collection](https://huggingface.co/collections/SamuelBang/lerf)

	</div>


	## Model Summary

	Vision-language models (VLMs) struggle with perspective taking: when a question asks about
	spatial relations from another entity's or an imagined observer's viewpoint, they often fall back
	to the camera view. LeRF trains a VLM to build and use an explicit **entity-centered reference
	frame**. Given an image and a question, the model decides whether a frame is needed. If it is, it
	grounds the reference entity and predicts the frame's projected origin and its front / left / up
	axes. A lightweight renderer draws the frame on the image, and the model reasons over this visual
	cue. No external perception models or 3D reconstruction are used.


	## How It Works

	Two-turn inference.

	1. Turn 1 (thinking off). The model outputs either a single `draw_reference_frame` call —
	`reference_object` (text), plus `origin`, `axis_front`, `axis_left`, `axis_up` as `[x, y]`
	points in `[0, 1000]` normalized image coordinates — or the literal string `NO_TOOL_CALL`
	when the question can be answered from the camera view.
	2. Tool. The frame is rendered on the image (red = front, green = left, blue = up; a `⊙`/`⊗`
	depth marker replaces a strongly foreshortened axis).
	3. Turn 2 (thinking on). The model sees the rendered image (not the numeric coordinates),
	reasons over it, and answers in `\boxed{}`.

	Training.

	- Stage I: SFT. Learns reference-entity grounding and projected frame prediction from pose
	datasets (ImageNet3D, Omni6DPose-SOPE, BEDLAM). About 20% of the data are no-tool-call examples
	that teach selective tool use.
	- Stage II: RL (GRPO). Trained on MultihopSpatial and SpatialReasoner-RL with a binary
	final-answer reward. The frame-prediction turn is
	excluded from the policy gradient, so only the reasoning/answer turn receives the advantage.

	## Usage

	The model is meant to be used together with the `draw_reference_frame` tool and the two-turn
	client from the [GitHub repository](https://github.com/bangx7/LeRF). The system prompt and
	tool schema live in `tool/system_prompt.txt` and `tool/prompts.py`.

	1. Serve with vLLM (OpenAI-compatible API):

	```bash
	vllm serve SamuelBang/LeRF-9B --served-model-name lerf --max-model-len 32768 \
	--limit-mm-per-prompt '{"image": 2}' --trust-remote-code
	```

	2. Run the client:

	```bash
	git clone https://github.com/bangx7/LeRF.git && cd LeRF/tool
	pip install pillow requests

	# single question
	python inference.py --model lerf --image example.jpg \
	--question "From the man's perspective, which object is on his left?" \
	--options cup bed laptop phone --save_renders renders/

	# batch: one JSON per line with image / question / options (and optionally answer, id)
	python inference.py --model lerf --input questions.jsonl --output results.jsonl
	```

	Recommended settings (matching training): temperature 0.6, images downscaled to at most 1M pixels,
	a 10,240-token thinking budget within a 12,288-token answer turn.

	Tested with Python 3.12, CUDA 13.0, PyTorch 2.11, vLLM 0.24.0 and transformers 5.10.4.

	## Evaluation

	Accuracy (%) on OmniSpatial perspective taking (OmniSpatial-PT: Ego / Allo / Hypo), 3DSRBench
	(Orientation / Multi-Object) and ViewSpatial-Bench (person-perspective Object View Orientation /
	Relative Direction). Bold marks the best open-source result.

	\| Method \| Ego \| Allo \| Hypo \| 3DSR Ori \| 3DSR M-Obj \| VS P-Obj \| VS P-Rel \|
	\|---\|:-:\|:-:\|:-:\|:-:\|:-:\|:-:\|:-:\|
	\| Proprietary \| \| \| \| \| \| \| \|
	\| GPT-5.6-Luna (medium) \| 83.33 \| 49.73 \| 45.78 \| 60.04 \| 55.34 \| 46.99 \| 70.07 \|
	\| GPT-5.6-Terra (medium) \| 81.37 \| 55.85 \| 53.01 \| 63.32 \| 56.39 \| 45.08 \| 77.20 \|
	\| Claude Sonnet 5 (medium) \| 80.39 \| 42.55 \| 49.40 \| 34.94 \| 44.77 \| 51.31 \| 51.43 \|
	\| Claude Sonnet 5 (high) \| 84.31 \| 48.14 \| 45.78 \| 43.15 \| 46.81 \| 51.51 \| 60.10 \|
	\| Open-source \| \| \| \| \| \| \| \|
	\| Qwen3.5-4B \| 74.71 \| 42.55 \| 44.34 \| 42.28 \| 43.26 \| 51.01 \| 57.43 \|
	\| Qwen3.5-9B \| 80.20 \| 47.13 \| 44.58 \| 48.17 \| 48.66 \| 56.23 \| 65.51 \|
	\| Qwen3.5-9B + APC \| 42.16 \| 27.66 \| 30.12 \| 44.98 \| 33.22 \| 59.34 \| 37.53 \|
	\| SpatialReasoner \| 40.39 \| 35.11 \| 35.66 \| 52.05 \| 50.64 \| 42.37 \| 45.61 \|
	\| Ours \| \| \| \| \| \| \| \|
	\| LeRF-4B \| 72.35 \| 49.36 \| 46.75 \| 45.88 \| 44.55 \| 56.26 \| 67.85 \|
	\| LeRF-9B (this model) \| 74.31 \| 54.04 \| 55.66 \| 53.76 \| 50.29 \| 61.91 \| 74.23 \|

	See the paper for more baselines, reference-frame estimation accuracy and ablations.


	## Acknowledgements

	Built on [Qwen3.5](https://huggingface.co/Qwen), [LLaMA-Factory](https://github.com/hiyouga/LLaMA-Factory)
	and [verl](https://github.com/verl-project/verl).

	## Citation

	```bibtex
	@misc{xiao2026lerflearningreferencecoordinate,
	title={LeRF: Learning Reference Coordinate Frames for Perspective Taking Reasoning},
	author={Bang Xiao and Wenqi Jia and Ozgur Kara and Tiancheng Shen and Yibo Yang and Bolin Lai and Junho Kim and James Matthew Rehg},
	year={2026},
	eprint={2609.36219},
	archivePrefix={arXiv},
	primaryClass={cs.CV},
	url={https://arxiv.org/abs/2609.36219},
	}
	```