mjf-su commited on
Commit
bd580dd
·
verified ·
1 Parent(s): bef064d

Fix README.md

Browse files
Files changed (1) hide show
  1. README.md +76 -38
README.md CHANGED
@@ -11,40 +11,30 @@ tags:
11
  - reasoning
12
  license: mit
13
  datasets:
14
- - tom-jerry-123/Physical-AI-AV-US
15
  ---
16
 
17
- # Model Card for PhysicalAI-reason-VLA (CoT-SFT)
18
 
19
- This model is a fine-tuned version of [Qwen/Qwen3-VL-4B-Thinking](https://huggingface.co/Qwen/Qwen3-VL-4B-Thinking).
20
- It has been post-trained using supervised fine-tuning (SFT) with [TRL](https://github.com/huggingface/trl) to produce **structured reasoning, discrete driving decisions, and future trajectories**.
21
 
22
- ---
23
-
24
- ## 🧠 Architecture
25
-
26
- This model is a **vision-language driving policy with structured reasoning**.
27
- It takes as input:
28
-
29
- - A forward-facing camera image
30
- - A sequence of past ego-vehicle waypoints (relative frame)
31
 
32
- and outputs:
33
 
34
- 1. **Scene understanding + reasoning (`<think>`)**
35
- 2. **Discrete driving decisions (`<action>`)**
36
- 3. **Future trajectory waypoints (`<wp>`)**
37
 
38
- ---
 
 
39
 
40
- ## 🧾 Output Format
41
 
42
- The model generates outputs in the following structured format:
43
  ```
44
  <think>
45
  {
46
- "scene": "...",
47
- "move_justification": "...",
48
  }
49
  </think>
50
  <action>
@@ -54,35 +44,83 @@ The model generates outputs in the following structured format:
54
  <wp>[x.xx,y.yy,t.tttt]</wp>
55
  ...
56
  ```
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
57
  ## Quick Start
 
 
58
  from transformers import AutoProcessor, AutoModelForImageTextToText
 
59
 
60
  model_id = "mjf-su/PhysicalAI-reason-VLA"
61
-
62
  processor = AutoProcessor.from_pretrained(model_id)
63
  model = AutoModelForImageTextToText.from_pretrained(model_id, device_map="auto")
64
 
65
- # Example usage (pseudo)
 
 
 
66
  messages = [
67
- {"role": "system", "content": [{"type": "text", "text": "..."}]},
68
- {"role": "user", "content": [
69
- {"type": "image"},
70
- {"type": "text", "text": "...past waypoints + objective..."}
71
- ]}
 
 
 
 
 
 
72
  ]
73
 
74
  prompt = processor.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
75
  inputs = processor(text=[prompt], images=[image], return_tensors="pt").to(model.device)
76
-
77
  outputs = model.generate(**inputs, max_new_tokens=512)
78
  print(processor.batch_decode(outputs, skip_special_tokens=True)[0])
 
79
 
80
- # Citation
 
 
 
 
81
  @misc{vonwerra2022trl,
82
- title = {{TRL: Transformer Reinforcement Learning}},
83
- author = {Leandro von Werra and Younes Belkada and Lewis Tunstall and Edward Beeching and Tristan Thrush and Nathan Lambert and Shengyi Huang and Kashif Rasul and Quentin Gallou{\'e}dec},
84
- year = 2020,
85
- journal = {GitHub repository},
86
- publisher = {GitHub},
87
- howpublished = {\url{https://github.com/huggingface/trl}}
88
- }
 
 
 
 
11
  - reasoning
12
  license: mit
13
  datasets:
14
+ - mjf-su/PhysicalAI-reason-US
15
  ---
16
 
17
+ # PhysicalAI-reason-VLA
18
 
19
+ A vision-language driving policy fine-tuned from [mjf-su/PhysicalAI-base-VLA](https://huggingface.co/mjf-su/PhysicalAI-base-VLA) (itself based on [Qwen/Qwen3-VL-4B-Thinking](https://huggingface.co/Qwen/Qwen3-VL-4B-Thinking)) using supervised fine-tuning with [TRL](https://github.com/huggingface/trl).
 
20
 
21
+ This model extends the base waypoint-prediction VLA with **structured chain-of-thought reasoning** and **discrete driving decisions**, trained on 10k Gemini-annotated driving scenes for 2 epochs.
 
 
 
 
 
 
 
 
22
 
23
+ ---
24
 
25
+ ## Input / Output
 
 
26
 
27
+ **Inputs**
28
+ - A forward-facing camera image
29
+ - Past ego-vehicle waypoints in the vehicle's relative frame
30
 
31
+ **Output**
32
 
 
33
  ```
34
  <think>
35
  {
36
+ "scene": "2–3 sentence static scene description",
37
+ "move_justification": "2–3 sentence causal explanation linking scene to decisions",
38
  }
39
  </think>
40
  <action>
 
44
  <wp>[x.xx,y.yy,t.tttt]</wp>
45
  ...
46
  ```
47
+
48
+ The model produces three outputs in sequence: a reasoning trace (`<think>`), discrete longitudinal and lateral driving decisions (`<action>`), and future trajectory waypoints (`<wp>`).
49
+
50
+ ---
51
+
52
+ ## Decision Tokens
53
+
54
+ Each `<action>` block contains exactly one longitudinal and one lateral token.
55
+
56
+ **Longitudinal** — `<stop>` · `<yield>` · `<follow>` · `<gap_search>` · `<pass>` · `<adapt>` · `<cruise>`
57
+
58
+ **Lateral** — `<turn_left>` · `<turn_right>` · `<lc_left>` · `<lc_right>` · `<merge>` · `<nudge_out_left>` · `<nudge_out_right>` · `<nudge_in_left>` · `<nudge_in_right>` · `<pull_over>` · `<abort>` · `<lane_keep>`
59
+
60
+ These are registered as genuine single tokens in the vocabulary (not subword decompositions), enabling efficient probability measurement over the full decision space with a single forward pass.
61
+
62
+ ---
63
+
64
+ ## Training
65
+
66
+ | | |
67
+ |---|---|
68
+ | **Base model** | [mjf-su/PhysicalAI-base-VLA](https://huggingface.co/mjf-su/PhysicalAI-base-VLA) |
69
+ | **Dataset** | [mjf-su/PhysicalAI-reason-US](https://huggingface.co/datasets/mjf-su/PhysicalAI-reason-US) |
70
+ | **Annotation** | Gemini batch API (chain-of-thought labels on real US driving data) |
71
+ | **Samples** | 10,000 |
72
+ | **Epochs** | 2 |
73
+ | **Method** | Completion-only SFT via TRL |
74
+
75
+ ---
76
+
77
  ## Quick Start
78
+
79
+ ```python
80
  from transformers import AutoProcessor, AutoModelForImageTextToText
81
+ from PIL import Image
82
 
83
  model_id = "mjf-su/PhysicalAI-reason-VLA"
 
84
  processor = AutoProcessor.from_pretrained(model_id)
85
  model = AutoModelForImageTextToText.from_pretrained(model_id, device_map="auto")
86
 
87
+ image = Image.open("forward_camera.jpg")
88
+
89
+ past_waypoints = "<wp>[0.00,0.00,0.0000]</wp>\n<wp>[0.51,0.00,0.0001]</wp>\n..."
90
+
91
  messages = [
92
+ {
93
+ "role": "system",
94
+ "content": [{"type": "text", "text": "You are a helpful AI assistant ..."}]
95
+ },
96
+ {
97
+ "role": "user",
98
+ "content": [
99
+ {"type": "image"},
100
+ {"type": "text", "text": f"[PAST-VEHICLE-MOTION]:\n{past_waypoints}"}
101
+ ]
102
+ }
103
  ]
104
 
105
  prompt = processor.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
106
  inputs = processor(text=[prompt], images=[image], return_tensors="pt").to(model.device)
 
107
  outputs = model.generate(**inputs, max_new_tokens=512)
108
  print(processor.batch_decode(outputs, skip_special_tokens=True)[0])
109
+ ```
110
 
111
+ ---
112
+
113
+ ## Citation
114
+
115
+ ```bibtex
116
  @misc{vonwerra2022trl,
117
+ title = {{TRL: Transformer Reinforcement Learning}},
118
+ author = {Leandro von Werra and Younes Belkada and Lewis Tunstall and Edward Beeching
119
+ and Tristan Thrush and Nathan Lambert and Shengyi Huang and Kashif Rasul
120
+ and Quentin Gallou{\'e}dec},
121
+ year = 2022,
122
+ journal = {GitHub repository},
123
+ publisher = {GitHub},
124
+ howpublished = {\url{https://github.com/huggingface/trl}}
125
+ }
126
+ ```