chenyvathf commited on
Commit
1357427
·
verified ·
1 Parent(s): 67b995c

gua v2: continue v1 on 12,000 balanced questions from macos-cua v3 (adds Logic Pro, OBS, Plasticity)

Browse files
README.md CHANGED
@@ -22,7 +22,8 @@ on confident answers and defer on the rest.
22
 
23
  It is [Qwen/Qwen3.5-4B](https://huggingface.co/Qwen/Qwen3.5-4B) fine-tuned (supervised
24
  fine-tuning with LoRA, rank 8 on every linear layer of the language model) on GUI
25
- trajectories extracted from screen-recorded software tutorials. This repository holds the
 
26
  LoRA merged into the base weights (MLX, bf16) and, in `adapter/`, the LoRA itself.
27
 
28
  > Research model. The training data was derived from public YouTube tutorials and is not
@@ -79,41 +80,43 @@ Without it, only text candidates are offered, which lowers recall on icon-only c
79
 
80
  ## Training
81
 
 
 
82
  | | |
83
  |---|---|
84
  | Method | Supervised fine-tuning with LoRA: rank 8, alpha 16, on every linear layer of the language model (`q/k/v/o_proj`, linear-attention `in_proj_*`/`out_proj`, MLP `gate/up/down_proj`); vision tower frozen |
85
  | Objective | Log loss of the right option label (and end-of-turn token) only, never of the prompt |
86
- | Data | The first 8,000 of 20,623 shuffled training questions of our dataset *macos-cua v2* (see below) |
87
- | Optimiser | Adam, learning rate 5e-5, batch 1, gradient clipping 1.0, one pass |
88
- | Hardware | One Apple Silicon Mac (MLX), gradient checkpointing, about 48 h, peak 161 GB |
89
-
90
- The recipe was chosen on a held-out dev split, never on the eval split: on the same 2,000
91
- samples, LoRA on every linear layer matched rank 64 and edged out attention-only LoRA, and
92
- a full-parameter run of the same model on earlier data did not beat LoRA.
93
 
94
- Dev score during training (mean accuracy of the three questions on 200 fixed dev steps):
95
-
96
- | Training samples | 0 (base) | 2,000 | 4,000 | 6,000 | 7,000 | 8,000 |
97
- |---|---|---|---|---|---|---|
98
- | Dev score | 0.322 | 0.429 | 0.421 | 0.441 | 0.445 | 0.447 |
99
 
100
  ## Training data
101
 
102
  Trajectories were extracted automatically from screen-recorded software tutorials on
103
  YouTube: videos were searched and screened (recording quality, platform), segmented and
104
  annotated into step-by-step actions by vision-language models, each click was located on
105
- the frame, and before/after screenshots were cut. The **training split** of *macos-cua v2*:
 
106
 
107
- | Topic | Videos | Steps |
108
- |---|---|---|
109
- | Godot 4 (game engine) | 79 | 6,271 |
110
- | Blender (3D) | 39 | 3,764 |
111
- | macOS video editing (Final Cut Pro, Premiere Pro, CapCut, iMovie) | 7 | 640 |
112
- | macOS system and productivity apps | 6 | 420 |
113
-
114
- Splits are by video (dev 14 videos, eval 30 videos, 20 applications appear only in eval).
115
- Godot typing steps were left out of training: an audit found their labels mostly wrong.
116
- Many Blender and Godot tutorials were recorded on Windows or full screen; steps that
 
 
 
 
117
  would be wrong on a Mac (Windows shortcuts, taskbar, native dialogs) were removed.
118
 
119
  **Label quality.** Labels are model-generated, not human-verified. A stronger model audited
@@ -121,67 +124,94 @@ a random sample per topic:
121
 
122
  | Topic | Audited steps | Element located correctly | Action correct |
123
  |---|---|---|---|
124
- | macOS system and productivity | 200 | 91.5% | 75.5% |
 
125
  | macOS video editing | 395 | 92.4% | 78.2% |
126
- | Blender | 150 | 89.3% | 70.7% |
 
 
127
  | Godot | 270 | 87.0% | 54.8% |
128
 
129
  The dataset itself is not released.
130
 
131
  ## Evaluation
132
 
133
- On the eval split of *macos-cua v2* (3,442 steps from 30 videos never seen in training;
134
- 2,079 steps with an element target), against the untrained base with the same prompt and
135
- candidates:
136
 
137
- | Question | Base | This model | Difference (95% CI, bootstrap paired by trajectory) |
138
  |---|---|---|---|
139
- | `action` accuracy | 0.481 | 0.525 | +0.043 (+0.005, +0.089) |
140
- | `grounding` accuracy | 0.443 | 0.603 | +0.160 (+0.132, +0.186) |
141
- | `next_target` accuracy | 0.155 | 0.261 | +0.106 (+0.085, +0.127) |
142
- | `action` and `next_target` both right | 0.117 | 0.183 | |
143
 
144
- On **applications never seen in training** (905 steps from software that appears only in
145
- the eval split) the gains hold: action 0.439 → 0.550, grounding 0.501 → 0.650, next target
146
- 0.221 → 0.313. A candidate covers the target in 83.4% of the element steps, which bounds
147
- the element questions.
148
 
149
- Excluding the 527 Godot typing steps, whose labels an audit found mostly wrong, action
150
- accuracy is 0.505 → 0.617 (+0.111). Because those steps were left out of training, the
151
- model rarely answers `type` (22 times against 588 typing steps in eval): expect low recall
152
- for typing.
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
153
 
154
- **Selective accuracy.** Acting only on the most confident half of the answers, grounding
155
- accuracy is 0.822 and action accuracy 0.636.
156
 
157
- **Calibration.** `temperatures.json` holds one temperature per question, fitted on the dev
158
- split by log loss and applied as `p^(1/T)` renormalised (`inference/cua_decide.py
159
- --temperatures`). Expected calibration error on eval, which the fit never saw:
160
 
161
  | Question | Temperature | ECE before | ECE after |
162
  |---|---|---|---|
163
- | `action` | 2.05 | 0.259 | 0.047 |
164
- | `grounding` | 1.20 | 0.145 | 0.107 |
165
- | `next_target` | 0.90 | 0.064 | 0.085 |
166
 
167
- The action head is strongly overconfident without scaling; the next-target head is
168
- already close to calibrated, and its fitted temperature slightly lowers log loss on eval
169
- while slightly raising ECE.
170
 
171
  ## Limitations
172
 
173
- - **macOS screenshots only**, English UI and tasks; tutorials over-represent Godot and
174
- Blender.
175
  - **Candidate-bound.** The model picks among candidates; if no candidate covers the target
176
- (16.6% of the element steps in our eval), the element questions have no right answer.
177
- - **Noisy labels**, as audited above; action-type labels are the weakest.
 
 
 
178
  - **Not an agent by itself.** It decides the action kind and target; the text to type, the
179
- keys of a shortcut and drag end points are not predicted.
180
  - **Merged weights round to bf16.** Against base plus adapter, option probabilities differ
181
- by at most 0.04 in total variation on our check; use `adapter/` for exact reproduction.
182
  - Confidence is only meaningful after temperature scaling, and only for the question types
183
  and candidate sources it was fitted with.
184
- - Rarely predicts `type` (see Evaluation).
 
 
 
 
 
 
185
 
186
  ## Licence and provenance
187
 
 
22
 
23
  It is [Qwen/Qwen3.5-4B](https://huggingface.co/Qwen/Qwen3.5-4B) fine-tuned (supervised
24
  fine-tuning with LoRA, rank 8 on every linear layer of the language model) on GUI
25
+ trajectories extracted from screen-recorded tutorials of seven kinds of software: macOS
26
+ system apps, video editing, Blender, Godot, Logic Pro, OBS Studio and Plasticity. This repository holds the
27
  LoRA merged into the base weights (MLX, bf16) and, in `adapter/`, the LoRA itself.
28
 
29
  > Research model. The training data was derived from public YouTube tutorials and is not
 
80
 
81
  ## Training
82
 
83
+ gua v2 continues the v1 adapter on a larger, more varied dataset.
84
+
85
  | | |
86
  |---|---|
87
  | Method | Supervised fine-tuning with LoRA: rank 8, alpha 16, on every linear layer of the language model (`q/k/v/o_proj`, linear-attention `in_proj_*`/`out_proj`, MLP `gate/up/down_proj`); vision tower frozen |
88
  | Objective | Log loss of the right option label (and end-of-turn token) only, never of the prompt |
89
+ | Stage 1 (v1) | 8,000 questions from *macos-cua v2* (macOS system, video editing, Blender, Godot) |
90
+ | Stage 2 (v2) | 12,000 further questions from *macos-cua v3*, shared evenly across its seven topics (each topic at most about 1,940; smaller topics give all they have) |
91
+ | Optimiser | Adam, learning rate 5e-5, batch 1, gradient clipping 1.0 |
92
+ | Hardware | One Apple Silicon Mac (MLX), gradient checkpointing, about 110 h in total, peak 182 GB |
 
 
 
93
 
94
+ The recipe was chosen on held-out dev splits, never on an eval split. The released weights
95
+ are the end of stage 2, which scored higher on the full v3 dev split (mean accuracy 0.453)
96
+ than the checkpoint with the best interim dev check (0.442).
 
 
97
 
98
  ## Training data
99
 
100
  Trajectories were extracted automatically from screen-recorded software tutorials on
101
  YouTube: videos were searched and screened (recording quality, platform), segmented and
102
  annotated into step-by-step actions by vision-language models, each click was located on
103
+ the frame, and before/after screenshots were cut. The **training split** of *macos-cua v3*,
104
+ and the questions stage 2 drew from it:
105
 
106
+ | Topic | Videos | Steps | Questions used in stage 2 |
107
+ |---|---|---|---|
108
+ | Plasticity (CAD) | 162 | 16,656 | 1,937 |
109
+ | Godot 4 (game engine) | 79 | 6,271 | 1,937 |
110
+ | Blender (3D) | 40 | 3,833 | 1,937 |
111
+ | OBS Studio (recording) | 37 | 2,890 | 1,937 |
112
+ | Logic Pro (audio) | 43 | 2,020 | 1,937 |
113
+ | macOS video editing (Final Cut Pro, Premiere Pro, CapCut, iMovie) | 7 | 640 | 1,468 |
114
+ | macOS system and productivity apps | 6 | 420 | 844 |
115
+
116
+ Splits are by video (dev 44 videos, eval 99 videos, 38 applications appear only in eval);
117
+ every video of the earlier v2 release kept its split, so v2's eval is part of v3's eval and
118
+ was never trained on. Godot typing steps were left out of training: an audit found their
119
+ labels mostly wrong. Many tutorials were recorded on Windows or full screen; steps that
120
  would be wrong on a Mac (Windows shortcuts, taskbar, native dialogs) were removed.
121
 
122
  **Label quality.** Labels are model-generated, not human-verified. A stronger model audited
 
124
 
125
  | Topic | Audited steps | Element located correctly | Action correct |
126
  |---|---|---|---|
127
+ | Logic Pro | 150 | 94.0% | 84.0% |
128
+ | OBS Studio | 150 | 92.0% | 81.3% |
129
  | macOS video editing | 395 | 92.4% | 78.2% |
130
+ | macOS system and productivity | 200 | 91.5% | 75.5% |
131
+ | Blender | 180 | 89.4% | 71.7% |
132
+ | Plasticity | 570 | 79.6% | 71.4% |
133
  | Godot | 270 | 87.0% | 54.8% |
134
 
135
  The dataset itself is not released.
136
 
137
  ## Evaluation
138
 
139
+ **macos-cua v3 eval split** (9,972 steps from 99 videos never seen in training, 6,509 with
140
+ an element target), gua v2 against gua v1 with the same prompt and candidates, 95% CI by
141
+ bootstrap paired by trajectory:
142
 
143
+ | Question | gua v1 | gua v2 | Difference |
144
  |---|---|---|---|
145
+ | `action` accuracy | 0.507 | 0.537 | +0.030 (+0.021, +0.040) |
146
+ | `grounding` accuracy | 0.516 | 0.524 | +0.008 (+0.002, +0.015) |
147
+ | `next_target` accuracy | 0.303 | 0.323 | +0.020 (+0.012, +0.029) |
148
+ | `action` and `next_target` both right | 0.182 | 0.216 | |
149
 
150
+ On the 1,767 steps from **applications never seen in training**: action 0.602, grounding
151
+ 0.628, next target 0.357. By topic (gua v2):
 
 
152
 
153
+ | Topic | `action` | `grounding` | `next_target` |
154
+ |---|---|---|---|
155
+ | OBS Studio | 0.806 | 0.803 | 0.453 |
156
+ | macOS system and productivity | 0.720 | 0.643 | 0.465 |
157
+ | Logic Pro | 0.687 | 0.566 | 0.341 |
158
+ | macOS video editing | 0.592 | 0.644 | 0.388 |
159
+ | Blender | 0.539 | 0.503 | 0.253 |
160
+ | Godot | 0.519 | 0.657 | 0.275 |
161
+ | Plasticity | 0.462 | 0.381 | 0.308 |
162
+
163
+ **macos-cua v2 eval split** (3,442 steps, the eval gua v1 was published with), against the
164
+ untrained base:
165
+
166
+ | Question | Base | gua v1 | gua v2 |
167
+ |---|---|---|---|
168
+ | `action` accuracy | 0.481 | 0.525 | 0.542 |
169
+ | `grounding` accuracy | 0.443 | 0.603 | 0.605 |
170
+ | `next_target` accuracy | 0.155 | 0.261 | 0.289 |
171
+
172
+ A candidate covers the target in 70.2% of v3's element steps (83.4% in v2's), which bounds
173
+ the element questions; Plasticity's small, dense CAD icons are often missed. Excluding the
174
+ Godot typing steps, whose labels are mostly wrong, v3 action accuracy is 0.563. Because
175
+ those steps were left out of training, the model rarely answers `type`.
176
 
177
+ **Selective accuracy** (v3 eval). Acting only on the most confident half of the answers:
178
+ grounding 0.711, action 0.665, next target 0.444.
179
 
180
+ **Calibration.** `temperatures.json` holds one temperature per question, fitted on the v3
181
+ dev split by log loss and applied as `p^(1/T)` renormalised (`inference/cua_decide.py
182
+ --temperatures`). Expected calibration error on the v3 eval split, which the fit never saw:
183
 
184
  | Question | Temperature | ECE before | ECE after |
185
  |---|---|---|---|
186
+ | `action` | 2.15 | 0.294 | 0.060 |
187
+ | `grounding` | 1.25 | 0.259 | 0.173 |
188
+ | `next_target` | 1.20 | 0.253 | 0.141 |
189
 
190
+ Without scaling the model is overconfident on every question; after it, element answers
191
+ remain somewhat overconfident.
 
192
 
193
  ## Limitations
194
 
195
+ - **macOS screenshots only**, English UI and tasks.
 
196
  - **Candidate-bound.** The model picks among candidates; if no candidate covers the target
197
+ (about 30% of element steps in v3's eval, most often in CAD), the element questions have
198
+ no right answer. On a live Mac, the accessibility tree is a better candidate source.
199
+ - **Noisy labels**, as audited above; action-type labels are the weakest, and Plasticity's
200
+ element labels the least reliable.
201
+ - **Weak on dense professional UIs**: Plasticity grounding is 0.38.
202
  - **Not an agent by itself.** It decides the action kind and target; the text to type, the
203
+ keys of a shortcut and drag end points are not predicted. It rarely predicts `type`.
204
  - **Merged weights round to bf16.** Against base plus adapter, option probabilities differ
205
+ by at most 0.035 in total variation on our check; use `adapter/` for exact reproduction.
206
  - Confidence is only meaningful after temperature scaling, and only for the question types
207
  and candidate sources it was fitted with.
208
+
209
+ ## Versions
210
+
211
+ | Version | Training | Notes |
212
+ |---|---|---|
213
+ | v2 (this) | v1 continued on 12,000 balanced questions from macos-cua v3 | adds Logic Pro, OBS Studio, Plasticity; better on every question |
214
+ | v1 | 8,000 questions from macos-cua v2 | first release; tagged `v1` in this repository |
215
 
216
  ## Licence and provenance
217
 
adapter/adapters.safetensors CHANGED
@@ -1,3 +1,3 @@
1
  version https://git-lfs.github.com/spec/v1
2
- oid sha256:872135863fbb2c45785d965898d85819efe15052004c76eddf7a2923e9f6f365
3
  size 64991946
 
1
  version https://git-lfs.github.com/spec/v1
2
+ oid sha256:e125d445722f9f931e5473988f3cc44a942d230af1aff1669d68aa271ef79966
3
  size 64991946
merge_check.json CHANGED
@@ -1,5 +1,5 @@
1
  {
2
  "fused_layers": 248,
3
  "check_steps": 5,
4
- "worst_total_variation_vs_adapter": 0.04049
5
  }
 
1
  {
2
  "fused_layers": 248,
3
  "check_steps": 5,
4
+ "worst_total_variation_vs_adapter": 0.03447
5
  }
model-00001-of-00002.safetensors CHANGED
@@ -1,3 +1,3 @@
1
  version https://git-lfs.github.com/spec/v1
2
- oid sha256:83a8837096b98320146ea9705130cb70e9ee445bc74ae04f1338985b54a75687
3
  size 5366980383
 
1
  version https://git-lfs.github.com/spec/v1
2
+ oid sha256:b7c089d993c6939edf367da5cf6a1d250f0414cd9e9dcd38981541d0d7506b92
3
  size 5366980383
model-00002-of-00002.safetensors CHANGED
@@ -1,3 +1,3 @@
1
  version https://git-lfs.github.com/spec/v1
2
- oid sha256:6b6428660ef6d18286f9a92ef85263f178189eb4c099541ebc1b964f73818f18
3
  size 3711641426
 
1
  version https://git-lfs.github.com/spec/v1
2
+ oid sha256:b85549825e9a4785f4728ee780f114e1775f9545817208653f79294151236b84
3
  size 3711641426
temperatures.json CHANGED
@@ -1,68 +1,68 @@
1
  {
2
- "dev": "experiments/decision/dev-v2-lora-v2b.jsonl",
3
- "eval": "experiments/decision/eval-v2-lora-v2b.jsonl",
4
  "questions": {
5
  "grounding": {
6
- "temperature": 1.2,
7
- "dev_items": 771,
8
  "dev_ece": {
9
- "before": 0.1964,
10
- "after": 0.145
11
  },
12
  "dev_log_loss": {
13
- "before": 1.2577,
14
- "after": 1.2158
15
  },
16
- "eval_items": 2079,
17
  "eval_ece": {
18
- "before": 0.1447,
19
- "after": 0.1069
20
  },
21
  "eval_log_loss": {
22
- "before": 1.2496,
23
- "after": 1.2284
24
  }
25
  },
26
  "action": {
27
- "temperature": 2.05,
28
- "dev_items": 1317,
29
  "dev_ece": {
30
- "before": 0.2948,
31
- "after": 0.0523
32
  },
33
  "dev_log_loss": {
34
- "before": 2.1282,
35
- "after": 1.6834
36
  },
37
- "eval_items": 3442,
38
  "eval_ece": {
39
- "before": 0.2593,
40
- "after": 0.0467
41
  },
42
  "eval_log_loss": {
43
- "before": 2.0045,
44
- "after": 1.6253
45
  }
46
  },
47
  "next_target": {
48
- "temperature": 0.9,
49
- "dev_items": 771,
50
  "dev_ece": {
51
- "before": 0.1169,
52
- "after": 0.1149
53
  },
54
  "dev_log_loss": {
55
- "before": 2.9552,
56
- "after": 2.9388
57
  },
58
- "eval_items": 2079,
59
  "eval_ece": {
60
- "before": 0.0635,
61
- "after": 0.0852
62
  },
63
  "eval_log_loss": {
64
- "before": 3.1757,
65
- "after": 3.1547
66
  }
67
  }
68
  }
 
1
  {
2
+ "dev": "experiments/decision/dev-v3-lora-v3.jsonl",
3
+ "eval": "experiments/decision/eval-v3-lora-v3.jsonl",
4
  "questions": {
5
  "grounding": {
6
+ "temperature": 1.25,
7
+ "dev_items": 2555,
8
  "dev_ece": {
9
+ "before": 0.2877,
10
+ "after": 0.2073
11
  },
12
  "dev_log_loss": {
13
+ "before": 1.0868,
14
+ "after": 1.0408
15
  },
16
+ "eval_items": 6508,
17
  "eval_ece": {
18
+ "before": 0.259,
19
+ "after": 0.1725
20
  },
21
  "eval_log_loss": {
22
+ "before": 1.1036,
23
+ "after": 1.0632
24
  }
25
  },
26
  "action": {
27
+ "temperature": 2.15,
28
+ "dev_items": 4006,
29
  "dev_ece": {
30
+ "before": 0.3158,
31
+ "after": 0.0655
32
  },
33
  "dev_log_loss": {
34
+ "before": 1.9669,
35
+ "after": 1.4436
36
  },
37
+ "eval_items": 9972,
38
  "eval_ece": {
39
+ "before": 0.2941,
40
+ "after": 0.0603
41
  },
42
  "eval_log_loss": {
43
+ "before": 1.9069,
44
+ "after": 1.4122
45
  }
46
  },
47
  "next_target": {
48
+ "temperature": 1.2,
49
+ "dev_items": 2555,
50
  "dev_ece": {
51
+ "before": 0.2854,
52
+ "after": 0.1744
53
  },
54
  "dev_log_loss": {
55
+ "before": 2.5119,
56
+ "after": 2.4338
57
  },
58
+ "eval_items": 6508,
59
  "eval_ece": {
60
+ "before": 0.2527,
61
+ "after": 0.1405
62
  },
63
  "eval_log_loss": {
64
+ "before": 2.5645,
65
+ "after": 2.483
66
  }
67
  }
68
  }