mjf-su commited on
Commit
bef064d
·
verified ·
1 Parent(s): 0f62cd3

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +53 -37
README.md CHANGED
@@ -1,66 +1,83 @@
1
  ---
2
  base_model: Qwen/Qwen3-VL-4B-Thinking
3
  library_name: transformers
4
- model_name: test-full-pipeline
5
  tags:
6
  - generated_from_trainer
7
  - sft
8
  - trl
9
- licence: license
 
 
 
10
  datasets:
11
  - tom-jerry-123/Physical-AI-AV-US
12
  ---
13
 
14
- # Model Card for test-full-pipeline
15
 
16
- This model is a fine-tuned version of [Qwen/Qwen3-VL-4B-Thinking](https://huggingface.co/Qwen/Qwen3-VL-4B-Thinking).
17
- It has been trained using [TRL](https://github.com/huggingface/trl).
18
 
19
- ## Architecture
20
 
21
- This model is a **base vision-language architecture** that maps images directly to vehicle waypoints. It takes a forward-facing camera image as input and outputs a sequence of future 2-D waypoints `[x, y]` in the ego-vehicle frame.
22
 
23
- **Egomotion context:** The model observes 2 seconds of past ego-vehicle motion, sampled at a 0.25 s period (9 historical poses including the current timestep).
 
24
 
25
- **Prediction horizon:** The model predicts 6 seconds into the future, also at a 0.25 s period (24 future waypoints).
 
26
 
27
- ## Training data
28
 
29
- This model was trained on **shards 0–19** (the first 20 shards) of the [tom-jerry-123/Physical-AI-AV-US](https://huggingface.co/datasets/tom-jerry-123/Physical-AI-AV-US) dataset on Hugging Face.
 
 
30
 
31
- ## Quick start
32
 
33
- ```python
34
- from transformers import pipeline
35
 
36
- question = "If you had a time machine, but could only go to the past or the future once and never return, which would you choose and why?"
37
- generator = pipeline("text-generation", model="None", device="cuda")
38
- output = generator([{"role": "user", "content": question}], max_new_tokens=128, return_full_text=False)[0]
39
- print(output["generated_text"])
40
  ```
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
41
 
42
- ## Training procedure
43
-
44
- [<img src="https://raw.githubusercontent.com/wandb/assets/main/wandb-github-badge-28.svg" alt="Visualize in Weights & Biases" width="150" height="24"/>](https://wandb.ai/mjf-su-stanford-university/wds-single-shard/runs/ssph0syr)
45
-
46
-
47
- This model was trained with SFT.
48
-
49
- ### Framework versions
50
 
51
- - TRL: 0.26.1
52
- - Transformers: 4.57.1
53
- - Pytorch: 2.9.0
54
- - Datasets: 4.4.1
55
- - Tokenizers: 0.22.1
56
 
57
- ## Citations
 
 
 
 
 
 
 
58
 
 
 
59
 
 
 
60
 
61
- Cite TRL as:
62
-
63
- ```bibtex
64
  @misc{vonwerra2022trl,
65
  title = {{TRL: Transformer Reinforcement Learning}},
66
  author = {Leandro von Werra and Younes Belkada and Lewis Tunstall and Edward Beeching and Tristan Thrush and Nathan Lambert and Shengyi Huang and Kashif Rasul and Quentin Gallou{\'e}dec},
@@ -68,5 +85,4 @@ Cite TRL as:
68
  journal = {GitHub repository},
69
  publisher = {GitHub},
70
  howpublished = {\url{https://github.com/huggingface/trl}}
71
- }
72
- ```
 
1
  ---
2
  base_model: Qwen/Qwen3-VL-4B-Thinking
3
  library_name: transformers
4
+ model_name: PhysicalAI-reason-VLA
5
  tags:
6
  - generated_from_trainer
7
  - sft
8
  - trl
9
+ - vision-language
10
+ - autonomous-driving
11
+ - reasoning
12
+ license: mit
13
  datasets:
14
  - tom-jerry-123/Physical-AI-AV-US
15
  ---
16
 
17
+ # Model Card for PhysicalAI-reason-VLA (CoT-SFT)
18
 
19
+ This model is a fine-tuned version of [Qwen/Qwen3-VL-4B-Thinking](https://huggingface.co/Qwen/Qwen3-VL-4B-Thinking).
20
+ It has been post-trained using supervised fine-tuning (SFT) with [TRL](https://github.com/huggingface/trl) to produce **structured reasoning, discrete driving decisions, and future trajectories**.
21
 
22
+ ---
23
 
24
+ ## 🧠 Architecture
25
 
26
+ This model is a **vision-language driving policy with structured reasoning**.
27
+ It takes as input:
28
 
29
+ - A forward-facing camera image
30
+ - A sequence of past ego-vehicle waypoints (relative frame)
31
 
32
+ and outputs:
33
 
34
+ 1. **Scene understanding + reasoning (`<think>`)**
35
+ 2. **Discrete driving decisions (`<action>`)**
36
+ 3. **Future trajectory waypoints (`<wp>`)**
37
 
38
+ ---
39
 
40
+ ## 🧾 Output Format
 
41
 
42
+ The model generates outputs in the following structured format:
 
 
 
43
  ```
44
+ <think>
45
+ {
46
+ "scene": "...",
47
+ "move_justification": "...",
48
+ }
49
+ </think>
50
+ <action>
51
+ <longitudinal_token><lateral_token>
52
+ </action>
53
+ <wp>[x.xx,y.yy,t.tttt]</wp>
54
+ <wp>[x.xx,y.yy,t.tttt]</wp>
55
+ ...
56
+ ```
57
+ ## Quick Start
58
+ from transformers import AutoProcessor, AutoModelForImageTextToText
59
 
60
+ model_id = "mjf-su/PhysicalAI-reason-VLA"
 
 
 
 
 
 
 
61
 
62
+ processor = AutoProcessor.from_pretrained(model_id)
63
+ model = AutoModelForImageTextToText.from_pretrained(model_id, device_map="auto")
 
 
 
64
 
65
+ # Example usage (pseudo)
66
+ messages = [
67
+ {"role": "system", "content": [{"type": "text", "text": "..."}]},
68
+ {"role": "user", "content": [
69
+ {"type": "image"},
70
+ {"type": "text", "text": "...past waypoints + objective..."}
71
+ ]}
72
+ ]
73
 
74
+ prompt = processor.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
75
+ inputs = processor(text=[prompt], images=[image], return_tensors="pt").to(model.device)
76
 
77
+ outputs = model.generate(**inputs, max_new_tokens=512)
78
+ print(processor.batch_decode(outputs, skip_special_tokens=True)[0])
79
 
80
+ # Citation
 
 
81
  @misc{vonwerra2022trl,
82
  title = {{TRL: Transformer Reinforcement Learning}},
83
  author = {Leandro von Werra and Younes Belkada and Lewis Tunstall and Edward Beeching and Tristan Thrush and Nathan Lambert and Shengyi Huang and Kashif Rasul and Quentin Gallou{\'e}dec},
 
85
  journal = {GitHub repository},
86
  publisher = {GitHub},
87
  howpublished = {\url{https://github.com/huggingface/trl}}
88
+ }