Image-Text-to-Text
Transformers
Safetensors
qwen3_vl
Generated from Trainer
sft
trl
vision-language
autonomous-driving
reasoning
conversational
Instructions to use mjf-su/PhysicalAI-base-VLA with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use mjf-su/PhysicalAI-base-VLA with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("image-text-to-text", model="mjf-su/PhysicalAI-base-VLA") messages = [ { "role": "user", "content": [ {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"}, {"type": "text", "text": "What animal is on the candy?"} ] }, ] pipe(text=messages)# pip install -U transformers accelerate # Load model directly from transformers import AutoProcessor, AutoModelForMultimodalLM processor = AutoProcessor.from_pretrained("mjf-su/PhysicalAI-base-VLA") model = AutoModelForMultimodalLM.from_pretrained("mjf-su/PhysicalAI-base-VLA", device_map="auto") messages = [ { "role": "user", "content": [ {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"}, {"type": "text", "text": "What animal is on the candy?"} ] }, ] inputs = processor.apply_chat_template( messages, add_generation_prompt=True, tokenize=True, return_dict=True, return_tensors="pt", ).to(model.device) outputs = model.generate(**inputs, max_new_tokens=256) print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:])) - Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- vLLM
How to use mjf-su/PhysicalAI-base-VLA with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "mjf-su/PhysicalAI-base-VLA" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "mjf-su/PhysicalAI-base-VLA", "messages": [ { "role": "user", "content": [ { "type": "text", "text": "Describe this image in one sentence." }, { "type": "image_url", "image_url": { "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg" } } ] } ] }'Use Docker
docker model run hf.co/mjf-su/PhysicalAI-base-VLA
- SGLang
How to use mjf-su/PhysicalAI-base-VLA with SGLang:
Install from pip and serve model
# Install SGLang from pip: pip install sglang # Start the SGLang server: python3 -m sglang.launch_server \ --model-path "mjf-su/PhysicalAI-base-VLA" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "mjf-su/PhysicalAI-base-VLA", "messages": [ { "role": "user", "content": [ { "type": "text", "text": "Describe this image in one sentence." }, { "type": "image_url", "image_url": { "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg" } } ] } ] }'Use Docker images
docker run --gpus all \ --shm-size 32g \ -p 30000:30000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ --env "HF_TOKEN=<secret>" \ --ipc=host \ lmsysorg/sglang:latest \ python3 -m sglang.launch_server \ --model-path "mjf-su/PhysicalAI-base-VLA" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "mjf-su/PhysicalAI-base-VLA", "messages": [ { "role": "user", "content": [ { "type": "text", "text": "Describe this image in one sentence." }, { "type": "image_url", "image_url": { "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg" } } ] } ] }' - Docker Model Runner
How to use mjf-su/PhysicalAI-base-VLA with Docker Model Runner:
docker model run hf.co/mjf-su/PhysicalAI-base-VLA
Fix README.md
Browse files
README.md
CHANGED
|
@@ -11,40 +11,30 @@ tags:
|
|
| 11 |
- reasoning
|
| 12 |
license: mit
|
| 13 |
datasets:
|
| 14 |
-
-
|
| 15 |
---
|
| 16 |
|
| 17 |
-
#
|
| 18 |
|
| 19 |
-
|
| 20 |
-
It has been post-trained using supervised fine-tuning (SFT) with [TRL](https://github.com/huggingface/trl) to produce **structured reasoning, discrete driving decisions, and future trajectories**.
|
| 21 |
|
| 22 |
-
---
|
| 23 |
-
|
| 24 |
-
## 🧠 Architecture
|
| 25 |
-
|
| 26 |
-
This model is a **vision-language driving policy with structured reasoning**.
|
| 27 |
-
It takes as input:
|
| 28 |
-
|
| 29 |
-
- A forward-facing camera image
|
| 30 |
-
- A sequence of past ego-vehicle waypoints (relative frame)
|
| 31 |
|
| 32 |
-
|
| 33 |
|
| 34 |
-
|
| 35 |
-
2. **Discrete driving decisions (`<action>`)**
|
| 36 |
-
3. **Future trajectory waypoints (`<wp>`)**
|
| 37 |
|
| 38 |
-
|
|
|
|
|
|
|
| 39 |
|
| 40 |
-
|
| 41 |
|
| 42 |
-
The model generates outputs in the following structured format:
|
| 43 |
```
|
| 44 |
<think>
|
| 45 |
{
|
| 46 |
-
"scene": "
|
| 47 |
-
"move_justification": "
|
| 48 |
}
|
| 49 |
</think>
|
| 50 |
<action>
|
|
@@ -54,35 +44,83 @@ The model generates outputs in the following structured format:
|
|
| 54 |
<wp>[x.xx,y.yy,t.tttt]</wp>
|
| 55 |
...
|
| 56 |
```
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 57 |
## Quick Start
|
|
|
|
|
|
|
| 58 |
from transformers import AutoProcessor, AutoModelForImageTextToText
|
|
|
|
| 59 |
|
| 60 |
model_id = "mjf-su/PhysicalAI-reason-VLA"
|
| 61 |
-
|
| 62 |
processor = AutoProcessor.from_pretrained(model_id)
|
| 63 |
model = AutoModelForImageTextToText.from_pretrained(model_id, device_map="auto")
|
| 64 |
|
| 65 |
-
|
|
|
|
|
|
|
|
|
|
| 66 |
messages = [
|
| 67 |
-
{
|
| 68 |
-
|
| 69 |
-
{"type": "
|
| 70 |
-
|
| 71 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 72 |
]
|
| 73 |
|
| 74 |
prompt = processor.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
|
| 75 |
inputs = processor(text=[prompt], images=[image], return_tensors="pt").to(model.device)
|
| 76 |
-
|
| 77 |
outputs = model.generate(**inputs, max_new_tokens=512)
|
| 78 |
print(processor.batch_decode(outputs, skip_special_tokens=True)[0])
|
|
|
|
| 79 |
|
| 80 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
| 81 |
@misc{vonwerra2022trl,
|
| 82 |
-
|
| 83 |
-
|
| 84 |
-
|
| 85 |
-
|
| 86 |
-
|
| 87 |
-
|
| 88 |
-
}
|
|
|
|
|
|
|
|
|
|
|
|
| 11 |
- reasoning
|
| 12 |
license: mit
|
| 13 |
datasets:
|
| 14 |
+
- mjf-su/PhysicalAI-reason-US
|
| 15 |
---
|
| 16 |
|
| 17 |
+
# PhysicalAI-reason-VLA
|
| 18 |
|
| 19 |
+
A vision-language driving policy fine-tuned from [mjf-su/PhysicalAI-base-VLA](https://huggingface.co/mjf-su/PhysicalAI-base-VLA) (itself based on [Qwen/Qwen3-VL-4B-Thinking](https://huggingface.co/Qwen/Qwen3-VL-4B-Thinking)) using supervised fine-tuning with [TRL](https://github.com/huggingface/trl).
|
|
|
|
| 20 |
|
| 21 |
+
This model extends the base waypoint-prediction VLA with **structured chain-of-thought reasoning** and **discrete driving decisions**, trained on 10k Gemini-annotated driving scenes for 2 epochs.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 22 |
|
| 23 |
+
---
|
| 24 |
|
| 25 |
+
## Input / Output
|
|
|
|
|
|
|
| 26 |
|
| 27 |
+
**Inputs**
|
| 28 |
+
- A forward-facing camera image
|
| 29 |
+
- Past ego-vehicle waypoints in the vehicle's relative frame
|
| 30 |
|
| 31 |
+
**Output**
|
| 32 |
|
|
|
|
| 33 |
```
|
| 34 |
<think>
|
| 35 |
{
|
| 36 |
+
"scene": "2–3 sentence static scene description",
|
| 37 |
+
"move_justification": "2–3 sentence causal explanation linking scene to decisions",
|
| 38 |
}
|
| 39 |
</think>
|
| 40 |
<action>
|
|
|
|
| 44 |
<wp>[x.xx,y.yy,t.tttt]</wp>
|
| 45 |
...
|
| 46 |
```
|
| 47 |
+
|
| 48 |
+
The model produces three outputs in sequence: a reasoning trace (`<think>`), discrete longitudinal and lateral driving decisions (`<action>`), and future trajectory waypoints (`<wp>`).
|
| 49 |
+
|
| 50 |
+
---
|
| 51 |
+
|
| 52 |
+
## Decision Tokens
|
| 53 |
+
|
| 54 |
+
Each `<action>` block contains exactly one longitudinal and one lateral token.
|
| 55 |
+
|
| 56 |
+
**Longitudinal** — `<stop>` · `<yield>` · `<follow>` · `<gap_search>` · `<pass>` · `<adapt>` · `<cruise>`
|
| 57 |
+
|
| 58 |
+
**Lateral** — `<turn_left>` · `<turn_right>` · `<lc_left>` · `<lc_right>` · `<merge>` · `<nudge_out_left>` · `<nudge_out_right>` · `<nudge_in_left>` · `<nudge_in_right>` · `<pull_over>` · `<abort>` · `<lane_keep>`
|
| 59 |
+
|
| 60 |
+
These are registered as genuine single tokens in the vocabulary (not subword decompositions), enabling efficient probability measurement over the full decision space with a single forward pass.
|
| 61 |
+
|
| 62 |
+
---
|
| 63 |
+
|
| 64 |
+
## Training
|
| 65 |
+
|
| 66 |
+
| | |
|
| 67 |
+
|---|---|
|
| 68 |
+
| **Base model** | [mjf-su/PhysicalAI-base-VLA](https://huggingface.co/mjf-su/PhysicalAI-base-VLA) |
|
| 69 |
+
| **Dataset** | [mjf-su/PhysicalAI-reason-US](https://huggingface.co/datasets/mjf-su/PhysicalAI-reason-US) |
|
| 70 |
+
| **Annotation** | Gemini batch API (chain-of-thought labels on real US driving data) |
|
| 71 |
+
| **Samples** | 10,000 |
|
| 72 |
+
| **Epochs** | 2 |
|
| 73 |
+
| **Method** | Completion-only SFT via TRL |
|
| 74 |
+
|
| 75 |
+
---
|
| 76 |
+
|
| 77 |
## Quick Start
|
| 78 |
+
|
| 79 |
+
```python
|
| 80 |
from transformers import AutoProcessor, AutoModelForImageTextToText
|
| 81 |
+
from PIL import Image
|
| 82 |
|
| 83 |
model_id = "mjf-su/PhysicalAI-reason-VLA"
|
|
|
|
| 84 |
processor = AutoProcessor.from_pretrained(model_id)
|
| 85 |
model = AutoModelForImageTextToText.from_pretrained(model_id, device_map="auto")
|
| 86 |
|
| 87 |
+
image = Image.open("forward_camera.jpg")
|
| 88 |
+
|
| 89 |
+
past_waypoints = "<wp>[0.00,0.00,0.0000]</wp>\n<wp>[0.51,0.00,0.0001]</wp>\n..."
|
| 90 |
+
|
| 91 |
messages = [
|
| 92 |
+
{
|
| 93 |
+
"role": "system",
|
| 94 |
+
"content": [{"type": "text", "text": "You are a helpful AI assistant ..."}]
|
| 95 |
+
},
|
| 96 |
+
{
|
| 97 |
+
"role": "user",
|
| 98 |
+
"content": [
|
| 99 |
+
{"type": "image"},
|
| 100 |
+
{"type": "text", "text": f"[PAST-VEHICLE-MOTION]:\n{past_waypoints}"}
|
| 101 |
+
]
|
| 102 |
+
}
|
| 103 |
]
|
| 104 |
|
| 105 |
prompt = processor.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
|
| 106 |
inputs = processor(text=[prompt], images=[image], return_tensors="pt").to(model.device)
|
|
|
|
| 107 |
outputs = model.generate(**inputs, max_new_tokens=512)
|
| 108 |
print(processor.batch_decode(outputs, skip_special_tokens=True)[0])
|
| 109 |
+
```
|
| 110 |
|
| 111 |
+
---
|
| 112 |
+
|
| 113 |
+
## Citation
|
| 114 |
+
|
| 115 |
+
```bibtex
|
| 116 |
@misc{vonwerra2022trl,
|
| 117 |
+
title = {{TRL: Transformer Reinforcement Learning}},
|
| 118 |
+
author = {Leandro von Werra and Younes Belkada and Lewis Tunstall and Edward Beeching
|
| 119 |
+
and Tristan Thrush and Nathan Lambert and Shengyi Huang and Kashif Rasul
|
| 120 |
+
and Quentin Gallou{\'e}dec},
|
| 121 |
+
year = 2022,
|
| 122 |
+
journal = {GitHub repository},
|
| 123 |
+
publisher = {GitHub},
|
| 124 |
+
howpublished = {\url{https://github.com/huggingface/trl}}
|
| 125 |
+
}
|
| 126 |
+
```
|