gpriday commited on
Commit
19808e2
·
verified ·
1 Parent(s): 09f0d09

Canopy reader: ONNX (INT8, FP32), readers, model card

Browse files
Files changed (5) hide show
  1. README.md +14 -14
  2. bench.json +150 -155
  3. model.fp32.onnx +1 -1
  4. model.int8.onnx +1 -1
  5. reader.json +1 -1
README.md CHANGED
@@ -28,8 +28,8 @@ INT8, CPU, one thread, one screen at a time on an Apple M4 Max; real screens are
28
 
29
  | Model | Download | Per screen | Real gold category | Real agent screens | Headline line | Card F1 | |
30
  | --- | --- | --- | --- | --- | --- | --- | --- |
31
- | [canopy-reader-17m](https://huggingface.co/daintreehq/canopy-reader-17m) | 17 MB | 24 ms | 57/60 | 86/93 | 80.4% | 0.68 | **this model** |
32
- | [canopy-reader-68m](https://huggingface.co/daintreehq/canopy-reader-68m) | 70 MB | 122 ms | 58/60 | 91/93 | 81.0% | 0.72 | |
33
 
34
  *Headline line* and *card F1* are scored on the 153 real screens against careful line labels made blind,
35
  independently of the training labels. Two careful labellers agree with each other at about 84% on the
@@ -134,7 +134,7 @@ with more than three characters). Outputs, in order:
134
  | `key_lines` | (B, M) | per line: does a reader need it (a recall-heavy extract) | logit > 0 |
135
 
136
  The **attention gate** is `g = max(sigmoid(needs_attention), P(approval) + P(question) + P(finished) + P(error))`,
137
- mapped so that 0.3 reads as 0.5: `g < 0.3 ? 0.5·g/0.3 : 0.5 + 0.5·(g−0.3)/(1−0.3)`.
138
  At ≥ 0.5 the agent needs the user. All of this is in `reader.json`, which the readers load.
139
 
140
  The headline is picked to be **the most important line to look at right now**: the question or approval
@@ -144,7 +144,7 @@ result, the reason, or what failed and why, never chrome, spinners, diffs or men
144
 
145
  ## How well it works
146
 
147
- INT8, as shipped. FP32 scores the same within noise (98.3% vs 98.3% category on test).
148
 
149
  **Real screens** (never trained on): 60 hand-checked gold screens from real sessions and 93 researched
150
  screens across 17 agent CLIs.
@@ -154,18 +154,18 @@ screens across 17 agent CLIs.
154
  | Gold screens, category (question and finished counted together, as Canopy shows them) | 57/60 |
155
  | Gold screens, attention gate right | 58/60 |
156
  | Researched agent screens, category | 86/93 |
157
- | Researched agent screens, attention gate right | 90/93 |
158
- | Headline line = the blind reference label's | 80.4% |
159
- | Options exactly right (screens with a menu) | 69.6% |
160
- | Task line right | 94.8% |
161
- | Whole card (headline + summary lines) F1 | 0.675 |
162
- | Tests / changes field right (gold) | 55/60, 50/60 |
163
 
164
  **Held-out synthetic test** (1,142 screens in agent × situation cells never seen in training):
165
- category 98.3%, gate missed 5 screens that needed the user, question line
166
- 100.0%, headline line 94.0%, options exact 94.9%,
167
- first summary line among the labelled ones 73.5%, tests 97.2%,
168
- changes 91.4%, score within one anchor 95.9%.
169
 
170
  ## Training
171
 
 
28
 
29
  | Model | Download | Per screen | Real gold category | Real agent screens | Headline line | Card F1 | |
30
  | --- | --- | --- | --- | --- | --- | --- | --- |
31
+ | [canopy-reader-17m](https://huggingface.co/daintreehq/canopy-reader-17m) | 17 MB | 24 ms | 57/60 | 86/93 | 75.8% | 0.66 | **this model** |
32
+ | [canopy-reader-68m](https://huggingface.co/daintreehq/canopy-reader-68m) | 70 MB | 133 ms | 57/60 | 90/93 | 81.0% | 0.72 | |
33
 
34
  *Headline line* and *card F1* are scored on the 153 real screens against careful line labels made blind,
35
  independently of the training labels. Two careful labellers agree with each other at about 84% on the
 
134
  | `key_lines` | (B, M) | per line: does a reader need it (a recall-heavy extract) | logit > 0 |
135
 
136
  The **attention gate** is `g = max(sigmoid(needs_attention), P(approval) + P(question) + P(finished) + P(error))`,
137
+ mapped so that 0.4 reads as 0.5: `g < 0.4 ? 0.5·g/0.4 : 0.5 + 0.5·(g−0.4)/(1−0.4)`.
138
  At ≥ 0.5 the agent needs the user. All of this is in `reader.json`, which the readers load.
139
 
140
  The headline is picked to be **the most important line to look at right now**: the question or approval
 
144
 
145
  ## How well it works
146
 
147
+ INT8, as shipped. FP32 scores the same within noise (99.2% vs 99.4% category on test).
148
 
149
  **Real screens** (never trained on): 60 hand-checked gold screens from real sessions and 93 researched
150
  screens across 17 agent CLIs.
 
154
  | Gold screens, category (question and finished counted together, as Canopy shows them) | 57/60 |
155
  | Gold screens, attention gate right | 58/60 |
156
  | Researched agent screens, category | 86/93 |
157
+ | Researched agent screens, attention gate right | 89/93 |
158
+ | Headline line = the blind reference label's | 75.8% |
159
+ | Options exactly right (screens with a menu) | 76.1% |
160
+ | Task line right | 96.1% |
161
+ | Whole card (headline + summary lines) F1 | 0.661 |
162
+ | Tests / changes field right (gold) | 56/60, 51/60 |
163
 
164
  **Held-out synthetic test** (1,142 screens in agent × situation cells never seen in training):
165
+ category 99.4%, gate missed 3 screens that needed the user, question line
166
+ 99.8%, headline line 93.0%, options exact 94.9%,
167
+ first summary line among the labelled ones 72.8%, tests 97.3%,
168
+ changes 91.4%, score within one anchor 96.7%.
169
 
170
  ## Training
171
 
bench.json CHANGED
@@ -1,181 +1,176 @@
1
  {
2
  "fp32": {
3
  "mb": 67.7,
4
- "load_s": 0.07,
5
- "ms_median": 22.8,
6
- "ms_p90": 47.4,
7
  "split": {
8
  "n": 1142,
9
- "category_acc": 0.9833625218914186,
10
- "category_macro_f1": 0.9865220618829879,
11
- "gate_acc": 0.989492119089317,
12
- "gate_hid": 5,
13
- "gate_surfaced": 7,
14
  "ask_recall": 1.0,
15
- "false_finished": 1,
16
- "blocked_agree": 0.999124343257443,
17
- "question_line_acc": 1.0,
18
- "status_line_acc": 0.9387040280210157,
19
- "key_recall": 0.9516508722158508,
20
- "key_precision": 0.8834472710971646,
21
- "notable_agree": 0.9754816112084063,
22
- "busy_notable_precision": 0.9767441860465116,
23
- "busy_notable_recall": 0.8689655172413793,
24
  "busy_n": 412,
25
  "busy_notable_n": 145,
26
  "confusions": {
27
- "finished->working": 15,
28
- "error->working": 1,
29
- "working->finished": 1,
30
- "finished->question": 1,
31
- "idle->working": 1
32
  },
33
  "summary_n": 1142,
34
- "summary_empty_agree": 0.9369527145359019,
35
- "summary_top1_in_gold": 0.7368421052631579,
36
- "summary_top1_exact": 0.44279176201373,
37
- "summary_any_hit": 0.8752860411899314,
38
- "summary_precision": 0.6432598667350077,
39
- "summary_recall": 0.6380274529740722,
40
- "summary_f1": 0.6406329760081673,
41
- "summary_lines_per_card": 1.7084063047285465,
42
- "options_exact": 0.948905109489051,
43
- "options_false_on_none": 0.006944444444444444,
44
- "action_line_acc": 0.9640980735551664,
45
- "task_line_acc": 0.9483362521891419,
46
- "tests_acc": 0.9719789842381786,
47
- "changes_acc": 0.9168126094570929,
48
- "risk_acc": 0.9868651488616462,
49
  "instruction_acc": 0.9903677758318739,
50
- "progress_acc": 0.5814360770577933,
51
- "failure_repeats_acc": 0.9089316987740805,
52
- "score_acc": 0.8362521891418564,
53
- "progress_within_10": 0.7206654991243433,
54
- "score_within_1_anchor": 0.9605954465849387
55
  },
56
  "real": {
57
  "gold": {
58
  "n": 60,
59
  "category_coarse": 57,
60
  "category_exact": 56,
61
- "gate": 58,
62
  "hid": 1,
63
- "surfaced": 1,
64
  "misses": [
65
- "2b4274ede3 [idle/codex] got working att=0.06 want idle",
66
- "c8f4b58a8e [working/claude] got error att=0.82 want working|idle",
67
- "03f3c93a3e [error/cursor] got idle att=0.38 want error"
 
68
  ],
69
  "tests": 56,
70
- "changes": 51,
71
- "score": 53,
72
- "score_within_1_anchor": 58,
73
  "task_present_agree": 25
74
  },
75
  "agents": {
76
  "n": 93,
77
- "category_coarse": 85,
78
- "category_exact": 85,
79
  "gate": 90,
80
  "hid": 3,
81
  "surfaced": 0,
82
  "misses": [
83
- "aider#2 [finished/aider] got idle att=0.10 want finished",
84
- "copilot#5 [interrupted/copilot] got working att=0.44 want finished",
85
- "crush#2 [finished/crush] got working att=0.59 want finished",
86
- "crush#3 [question/crush] got working att=0.30 want question",
87
  "goose#3 [question/goose] got approval att=1.00 want question",
88
- "interpreter#1 [working/interpreter] got idle att=0.11 want working",
89
- "mistral#1 [question/mistral] got approval att=1.00 want question",
90
- "opencode#4 [error/opencode] got working att=0.67 want error"
91
  ]
92
  },
93
  "summary_vs_luna": {
94
  "summary_n": 153,
95
- "summary_empty_agree": 0.738562091503268,
96
- "summary_top1_in_gold": 0.6018518518518519,
97
- "summary_top1_exact": 0.4166666666666667,
98
- "summary_any_hit": 0.6296296296296297,
99
- "summary_precision": 0.675,
100
- "summary_recall": 0.5625,
101
- "summary_f1": 0.6136363636363636,
102
- "summary_lines_per_card": 1.0457516339869282
103
  },
104
  "lines_vs_gold": {
105
  "n": 153,
106
- "headline_acc": 0.803921568627451,
107
- "task_acc": 0.9477124183006536,
108
- "options_exact": 0.7608695652173914,
109
  "options_false_on_none": 0.009345794392523364,
110
  "summary_n": 153,
111
- "summary_empty_agree": 0.7124183006535948,
112
- "summary_top1_in_gold": 0.5245901639344263,
113
- "summary_top1_exact": 0.38524590163934425,
114
  "summary_any_hit": 0.5655737704918032,
115
- "summary_precision": 0.6375,
116
- "summary_recall": 0.41975308641975306,
117
- "summary_f1": 0.5062034739454094,
118
- "summary_lines_per_card": 1.0457516339869282,
119
- "card_any_hit": 0.9103448275862069,
120
- "card_precision": 0.7972027972027972,
121
- "card_recall": 0.5876288659793815,
122
- "card_f1": 0.6765578635014837
123
  }
124
  }
125
  },
126
  "int8": {
127
  "mb": 17.3,
128
  "load_s": 0.07,
129
- "ms_median": 23.5,
130
- "ms_p90": 49.4,
131
  "split": {
132
  "n": 1142,
133
- "category_acc": 0.9833625218914186,
134
- "category_macro_f1": 0.9865220618829879,
135
- "gate_acc": 0.9921190893169878,
136
- "gate_hid": 5,
137
- "gate_surfaced": 4,
138
  "ask_recall": 1.0,
139
  "false_finished": 1,
140
- "blocked_agree": 0.999124343257443,
141
- "question_line_acc": 1.0,
142
- "status_line_acc": 0.9404553415061296,
143
- "key_recall": 0.9518923160499789,
144
- "key_precision": 0.8804645190106638,
145
- "notable_agree": 0.9746059544658494,
146
- "busy_notable_precision": 0.9763779527559056,
147
- "busy_notable_recall": 0.8551724137931035,
148
  "busy_n": 412,
149
  "busy_notable_n": 145,
150
  "confusions": {
151
- "finished->working": 15,
152
- "error->working": 1,
153
- "working->finished": 1,
154
- "finished->question": 1,
155
- "idle->working": 1
156
  },
157
  "summary_n": 1142,
158
- "summary_empty_agree": 0.9395796847635727,
159
- "summary_top1_in_gold": 0.7345537757437071,
160
- "summary_top1_exact": 0.43363844393592677,
161
- "summary_any_hit": 0.8935926773455377,
162
- "summary_precision": 0.6400797607178464,
163
- "summary_recall": 0.6527707168276563,
164
- "summary_f1": 0.6463629499119053,
165
- "summary_lines_per_card": 1.7565674255691768,
166
  "options_exact": 0.948905109489051,
167
- "options_false_on_none": 0.006944444444444444,
168
- "action_line_acc": 0.9614711033274956,
169
- "task_line_acc": 0.946584938704028,
170
- "tests_acc": 0.9719789842381786,
171
  "changes_acc": 0.9141856392294221,
172
  "risk_acc": 0.9842381786339754,
173
  "instruction_acc": 0.9903677758318739,
174
- "progress_acc": 0.5761821366024519,
175
  "failure_repeats_acc": 0.9106830122591943,
176
- "score_acc": 0.8362521891418564,
177
- "progress_within_10": 0.7119089316987741,
178
- "score_within_1_anchor": 0.9588441330998249
179
  },
180
  "real": {
181
  "gold": {
@@ -183,66 +178,66 @@
183
  "category_coarse": 57,
184
  "category_exact": 56,
185
  "gate": 58,
186
- "hid": 1,
187
- "surfaced": 1,
188
  "misses": [
189
- "2b4274ede3 [idle/codex] got working att=0.06 want idle",
190
- "c8f4b58a8e [working/claude] got error att=0.88 want working|idle",
191
- "03f3c93a3e [error/cursor] got idle att=0.35 want error"
192
  ],
193
- "tests": 55,
194
- "changes": 50,
195
  "score": 53,
196
- "score_within_1_anchor": 58,
197
  "task_present_agree": 25
198
  },
199
  "agents": {
200
  "n": 93,
201
  "category_coarse": 86,
202
  "category_exact": 86,
203
- "gate": 90,
204
- "hid": 3,
205
  "surfaced": 0,
206
  "misses": [
207
- "aider#2 [finished/aider] got idle att=0.12 want finished",
208
- "copilot#5 [interrupted/copilot] got working att=0.43 want finished",
209
- "crush#2 [finished/crush] got working att=0.56 want finished",
210
- "crush#3 [question/crush] got working att=0.27 want question",
211
- "cursor#2 [finished/cursor] got working att=0.63 want finished",
212
  "goose#3 [question/goose] got approval att=1.00 want question",
213
- "mistral#1 [question/mistral] got approval att=1.00 want question"
 
 
 
214
  ]
215
  },
216
  "summary_vs_luna": {
217
  "summary_n": 153,
218
- "summary_empty_agree": 0.7516339869281046,
219
- "summary_top1_in_gold": 0.6296296296296297,
220
- "summary_top1_exact": 0.42592592592592593,
221
- "summary_any_hit": 0.6666666666666666,
222
- "summary_precision": 0.6547619047619048,
223
- "summary_recall": 0.5729166666666666,
224
- "summary_f1": 0.611111111111111,
225
- "summary_lines_per_card": 1.0980392156862746
226
  },
227
  "lines_vs_gold": {
228
  "n": 153,
229
- "headline_acc": 0.803921568627451,
230
- "task_acc": 0.9477124183006536,
231
- "options_exact": 0.6956521739130435,
232
  "options_false_on_none": 0.009345794392523364,
233
  "summary_n": 153,
234
- "summary_empty_agree": 0.7254901960784313,
235
- "summary_top1_in_gold": 0.5491803278688525,
236
- "summary_top1_exact": 0.39344262295081966,
237
  "summary_any_hit": 0.5819672131147541,
238
- "summary_precision": 0.6190476190476191,
239
- "summary_recall": 0.4279835390946502,
240
- "summary_f1": 0.5060827250608273,
241
- "summary_lines_per_card": 1.0980392156862746,
242
- "card_any_hit": 0.903448275862069,
243
- "card_precision": 0.7849829351535836,
244
- "card_recall": 0.5927835051546392,
245
- "card_f1": 0.6754772393538914
246
  }
247
  }
248
  }
 
1
  {
2
  "fp32": {
3
  "mb": 67.7,
4
+ "load_s": 0.08,
5
+ "ms_median": 23.3,
6
+ "ms_p90": 49.9,
7
  "split": {
8
  "n": 1142,
9
+ "category_acc": 0.9921190893169878,
10
+ "category_macro_f1": 0.9933529790794,
11
+ "gate_acc": 0.9912434325744308,
12
+ "gate_hid": 4,
13
+ "gate_surfaced": 6,
14
  "ask_recall": 1.0,
15
+ "false_finished": 3,
16
+ "blocked_agree": 1.0,
17
+ "question_line_acc": 0.999124343257443,
18
+ "status_line_acc": 0.9299474605954466,
19
+ "key_recall": 0.9470030784088851,
20
+ "key_precision": 0.886634642554394,
21
+ "notable_agree": 0.9807355516637478,
22
+ "busy_notable_precision": 0.9629629629629629,
23
+ "busy_notable_recall": 0.896551724137931,
24
  "busy_n": 412,
25
  "busy_notable_n": 145,
26
  "confusions": {
27
+ "finished->working": 5,
28
+ "working->finished": 2,
29
+ "idle->finished": 1,
30
+ "working->question": 1
 
31
  },
32
  "summary_n": 1142,
33
+ "summary_empty_agree": 0.9238178633975481,
34
+ "summary_top1_in_gold": 0.7219679633867276,
35
+ "summary_top1_exact": 0.43363844393592677,
36
+ "summary_any_hit": 0.8615560640732265,
37
+ "summary_precision": 0.6434599156118144,
38
+ "summary_recall": 0.6202338586680224,
39
+ "summary_f1": 0.6316334455086721,
40
+ "summary_lines_per_card": 1.660245183887916,
41
+ "options_exact": 0.9525547445255474,
42
+ "options_false_on_none": 0.005787037037037037,
43
+ "action_line_acc": 0.9754816112084063,
44
+ "task_line_acc": 0.9562171628721541,
45
+ "tests_acc": 0.9693520140105079,
46
+ "changes_acc": 0.9124343257443083,
47
+ "risk_acc": 0.9859894921190894,
48
  "instruction_acc": 0.9903677758318739,
49
+ "progress_acc": 0.5893169877408057,
50
+ "failure_repeats_acc": 0.9115586690017513,
51
+ "score_acc": 0.8450087565674256,
52
+ "progress_within_10": 0.7241681260945709,
53
+ "score_within_1_anchor": 0.9667250437828371
54
  },
55
  "real": {
56
  "gold": {
57
  "n": 60,
58
  "category_coarse": 57,
59
  "category_exact": 56,
60
+ "gate": 56,
61
  "hid": 1,
62
+ "surfaced": 3,
63
  "misses": [
64
+ "c8f4b58a8e [working/claude] got error att=0.81 want working|idle",
65
+ "0ee96e895b [working/claude] got finished att=0.87 want working",
66
+ "f19b520cb2 [interrupted/codex] got working att=0.51 want working",
67
+ "03f3c93a3e [error/cursor] got idle att=0.40 want error"
68
  ],
69
  "tests": 56,
70
+ "changes": 52,
71
+ "score": 51,
72
+ "score_within_1_anchor": 55,
73
  "task_present_agree": 25
74
  },
75
  "agents": {
76
  "n": 93,
77
+ "category_coarse": 88,
78
+ "category_exact": 88,
79
  "gate": 90,
80
  "hid": 3,
81
  "surfaced": 0,
82
  "misses": [
83
+ "aider#1 [approval/aider] got working att=0.16 want approval",
84
+ "crush#3 [question/crush] got idle att=0.19 want question",
 
 
85
  "goose#3 [question/goose] got approval att=1.00 want question",
86
+ "kiro#3 [interrupted/kiro] got working att=0.41 want finished",
87
+ "mistral#1 [question/mistral] got approval att=1.00 want question"
 
88
  ]
89
  },
90
  "summary_vs_luna": {
91
  "summary_n": 153,
92
+ "summary_empty_agree": 0.7450980392156863,
93
+ "summary_top1_in_gold": 0.6296296296296297,
94
+ "summary_top1_exact": 0.49074074074074076,
95
+ "summary_any_hit": 0.6666666666666666,
96
+ "summary_precision": 0.6369047619047619,
97
+ "summary_recall": 0.5572916666666666,
98
+ "summary_f1": 0.5944444444444444,
99
+ "summary_lines_per_card": 1.0980392156862746
100
  },
101
  "lines_vs_gold": {
102
  "n": 153,
103
+ "headline_acc": 0.7581699346405228,
104
+ "task_acc": 0.954248366013072,
105
+ "options_exact": 0.782608695652174,
106
  "options_false_on_none": 0.009345794392523364,
107
  "summary_n": 153,
108
+ "summary_empty_agree": 0.7189542483660131,
109
+ "summary_top1_in_gold": 0.48360655737704916,
110
+ "summary_top1_exact": 0.3770491803278688,
111
  "summary_any_hit": 0.5655737704918032,
112
+ "summary_precision": 0.5714285714285714,
113
+ "summary_recall": 0.3950617283950617,
114
+ "summary_f1": 0.46715328467153283,
115
+ "summary_lines_per_card": 1.0980392156862746,
116
+ "card_any_hit": 0.8620689655172413,
117
+ "card_precision": 0.7686832740213523,
118
+ "card_recall": 0.5567010309278351,
119
+ "card_f1": 0.6457399103139013
120
  }
121
  }
122
  },
123
  "int8": {
124
  "mb": 17.3,
125
  "load_s": 0.07,
126
+ "ms_median": 24.5,
127
+ "ms_p90": 50.8,
128
  "split": {
129
  "n": 1142,
130
+ "category_acc": 0.9938704028021016,
131
+ "category_macro_f1": 0.9947484950966307,
132
+ "gate_acc": 0.9929947460595446,
133
+ "gate_hid": 3,
134
+ "gate_surfaced": 5,
135
  "ask_recall": 1.0,
136
  "false_finished": 1,
137
+ "blocked_agree": 1.0,
138
+ "question_line_acc": 0.9982486865148862,
139
+ "status_line_acc": 0.9299474605954466,
140
+ "key_recall": 0.9529184523450233,
141
+ "key_precision": 0.8797436611869601,
142
+ "notable_agree": 0.978108581436077,
143
+ "busy_notable_precision": 0.9624060150375939,
144
+ "busy_notable_recall": 0.8827586206896552,
145
  "busy_n": 412,
146
  "busy_notable_n": 145,
147
  "confusions": {
148
+ "finished->working": 5,
149
+ "idle->finished": 1,
150
+ "working->error": 1
 
 
151
  },
152
  "summary_n": 1142,
153
+ "summary_empty_agree": 0.9290718038528897,
154
+ "summary_top1_in_gold": 0.7276887871853547,
155
+ "summary_top1_exact": 0.42105263157894735,
156
+ "summary_any_hit": 0.8764302059496567,
157
+ "summary_precision": 0.6371951219512195,
158
+ "summary_recall": 0.6375190645653279,
159
+ "summary_f1": 0.6373570520965692,
160
+ "summary_lines_per_card": 1.723292469352014,
161
  "options_exact": 0.948905109489051,
162
+ "options_false_on_none": 0.004629629629629629,
163
+ "action_line_acc": 0.9728546409807356,
164
+ "task_line_acc": 0.9535901926444834,
165
+ "tests_acc": 0.9728546409807356,
166
  "changes_acc": 0.9141856392294221,
167
  "risk_acc": 0.9842381786339754,
168
  "instruction_acc": 0.9903677758318739,
169
+ "progress_acc": 0.5831873905429071,
170
  "failure_repeats_acc": 0.9106830122591943,
171
+ "score_acc": 0.841506129597198,
172
+ "progress_within_10": 0.7189141856392294,
173
+ "score_within_1_anchor": 0.9667250437828371
174
  },
175
  "real": {
176
  "gold": {
 
178
  "category_coarse": 57,
179
  "category_exact": 56,
180
  "gate": 58,
181
+ "hid": 0,
182
+ "surfaced": 2,
183
  "misses": [
184
+ "c8f4b58a8e [working/claude] got finished att=0.79 want working|idle",
185
+ "0ee96e895b [working/claude] got finished att=0.77 want working",
186
+ "03f3c93a3e [error/cursor] got idle att=0.54 want error"
187
  ],
188
+ "tests": 56,
189
+ "changes": 51,
190
  "score": 53,
191
+ "score_within_1_anchor": 56,
192
  "task_present_agree": 25
193
  },
194
  "agents": {
195
  "n": 93,
196
  "category_coarse": 86,
197
  "category_exact": 86,
198
+ "gate": 89,
199
+ "hid": 4,
200
  "surfaced": 0,
201
  "misses": [
202
+ "aider#1 [approval/aider] got working att=0.16 want approval",
203
+ "crush#3 [question/crush] got idle att=0.21 want question",
 
 
 
204
  "goose#3 [question/goose] got approval att=1.00 want question",
205
+ "interpreter#0 [approval/interpreter] got working att=0.41 want approval",
206
+ "kiro#3 [interrupted/kiro] got working att=0.51 want finished",
207
+ "mistral#1 [question/mistral] got approval att=1.00 want question",
208
+ "opencode#4 [error/opencode] got working att=0.34 want error"
209
  ]
210
  },
211
  "summary_vs_luna": {
212
  "summary_n": 153,
213
+ "summary_empty_agree": 0.7581699346405228,
214
+ "summary_top1_in_gold": 0.6111111111111112,
215
+ "summary_top1_exact": 0.4351851851851852,
216
+ "summary_any_hit": 0.6759259259259259,
217
+ "summary_precision": 0.6566265060240963,
218
+ "summary_recall": 0.5677083333333334,
219
+ "summary_f1": 0.6089385474860336,
220
+ "summary_lines_per_card": 1.0849673202614378
221
  },
222
  "lines_vs_gold": {
223
  "n": 153,
224
+ "headline_acc": 0.7581699346405228,
225
+ "task_acc": 0.9607843137254902,
226
+ "options_exact": 0.7608695652173914,
227
  "options_false_on_none": 0.009345794392523364,
228
  "summary_n": 153,
229
+ "summary_empty_agree": 0.7320261437908496,
230
+ "summary_top1_in_gold": 0.5163934426229508,
231
+ "summary_top1_exact": 0.36885245901639346,
232
  "summary_any_hit": 0.5819672131147541,
233
+ "summary_precision": 0.5963855421686747,
234
+ "summary_recall": 0.4074074074074074,
235
+ "summary_f1": 0.4841075794621027,
236
+ "summary_lines_per_card": 1.0849673202614378,
237
+ "card_any_hit": 0.8758620689655172,
238
+ "card_precision": 0.7816901408450704,
239
+ "card_recall": 0.5721649484536082,
240
+ "card_f1": 0.6607142857142857
241
  }
242
  }
243
  }
model.fp32.onnx CHANGED
@@ -1,3 +1,3 @@
1
  version https://git-lfs.github.com/spec/v1
2
- oid sha256:2699ccce7e0d0ac16fd8425e8da153ece822bed6c3e670f0bbcae46628f50628
3
  size 67726050
 
1
  version https://git-lfs.github.com/spec/v1
2
+ oid sha256:fe6fb4d76e68ae79bb25a833394912a420d0a0654390bdc5ca7fddc9ebd5d011
3
  size 67726050
model.int8.onnx CHANGED
@@ -1,3 +1,3 @@
1
  version https://git-lfs.github.com/spec/v1
2
- oid sha256:c616d0b5339df3d3617e63ab85f8c9d96ada85cb4b4f38fe42037d978dece7a8
3
  size 17298433
 
1
  version https://git-lfs.github.com/spec/v1
2
+ oid sha256:21807b2c66b4e6e53e28e290538a53de93228eec2e760c127e19d469aa9b2136
3
  size 17298433
reader.json CHANGED
@@ -121,7 +121,7 @@
121
  "summary_first": 0.5,
122
  "summary_more": 0.6,
123
  "option": 0.5,
124
- "gate": 0.3,
125
  "gate_categories": [
126
  "approval",
127
  "question",
 
121
  "summary_first": 0.5,
122
  "summary_more": 0.6,
123
  "option": 0.5,
124
+ "gate": 0.4,
125
  "gate_categories": [
126
  "approval",
127
  "question",