Canopy reader: ONNX (INT8, FP32), readers, model card
Browse files- README.md +14 -14
- bench.json +150 -155
- model.fp32.onnx +1 -1
- model.int8.onnx +1 -1
- reader.json +1 -1
README.md
CHANGED
|
@@ -28,8 +28,8 @@ INT8, CPU, one thread, one screen at a time on an Apple M4 Max; real screens are
|
|
| 28 |
|
| 29 |
| Model | Download | Per screen | Real gold category | Real agent screens | Headline line | Card F1 | |
|
| 30 |
| --- | --- | --- | --- | --- | --- | --- | --- |
|
| 31 |
-
| [canopy-reader-17m](https://huggingface.co/daintreehq/canopy-reader-17m) | 17 MB | 24 ms | 57/60 | 86/93 |
|
| 32 |
-
| [canopy-reader-68m](https://huggingface.co/daintreehq/canopy-reader-68m) | 70 MB |
|
| 33 |
|
| 34 |
*Headline line* and *card F1* are scored on the 153 real screens against careful line labels made blind,
|
| 35 |
independently of the training labels. Two careful labellers agree with each other at about 84% on the
|
|
@@ -134,7 +134,7 @@ with more than three characters). Outputs, in order:
|
|
| 134 |
| `key_lines` | (B, M) | per line: does a reader need it (a recall-heavy extract) | logit > 0 |
|
| 135 |
|
| 136 |
The **attention gate** is `g = max(sigmoid(needs_attention), P(approval) + P(question) + P(finished) + P(error))`,
|
| 137 |
-
mapped so that 0.
|
| 138 |
At ≥ 0.5 the agent needs the user. All of this is in `reader.json`, which the readers load.
|
| 139 |
|
| 140 |
The headline is picked to be **the most important line to look at right now**: the question or approval
|
|
@@ -144,7 +144,7 @@ result, the reason, or what failed and why, never chrome, spinners, diffs or men
|
|
| 144 |
|
| 145 |
## How well it works
|
| 146 |
|
| 147 |
-
INT8, as shipped. FP32 scores the same within noise (
|
| 148 |
|
| 149 |
**Real screens** (never trained on): 60 hand-checked gold screens from real sessions and 93 researched
|
| 150 |
screens across 17 agent CLIs.
|
|
@@ -154,18 +154,18 @@ screens across 17 agent CLIs.
|
|
| 154 |
| Gold screens, category (question and finished counted together, as Canopy shows them) | 57/60 |
|
| 155 |
| Gold screens, attention gate right | 58/60 |
|
| 156 |
| Researched agent screens, category | 86/93 |
|
| 157 |
-
| Researched agent screens, attention gate right |
|
| 158 |
-
| Headline line = the blind reference label's |
|
| 159 |
-
| Options exactly right (screens with a menu) |
|
| 160 |
-
| Task line right |
|
| 161 |
-
| Whole card (headline + summary lines) F1 | 0.
|
| 162 |
-
| Tests / changes field right (gold) |
|
| 163 |
|
| 164 |
**Held-out synthetic test** (1,142 screens in agent × situation cells never seen in training):
|
| 165 |
-
category
|
| 166 |
-
|
| 167 |
-
first summary line among the labelled ones
|
| 168 |
-
changes 91.4%, score within one anchor
|
| 169 |
|
| 170 |
## Training
|
| 171 |
|
|
|
|
| 28 |
|
| 29 |
| Model | Download | Per screen | Real gold category | Real agent screens | Headline line | Card F1 | |
|
| 30 |
| --- | --- | --- | --- | --- | --- | --- | --- |
|
| 31 |
+
| [canopy-reader-17m](https://huggingface.co/daintreehq/canopy-reader-17m) | 17 MB | 24 ms | 57/60 | 86/93 | 75.8% | 0.66 | **this model** |
|
| 32 |
+
| [canopy-reader-68m](https://huggingface.co/daintreehq/canopy-reader-68m) | 70 MB | 133 ms | 57/60 | 90/93 | 81.0% | 0.72 | |
|
| 33 |
|
| 34 |
*Headline line* and *card F1* are scored on the 153 real screens against careful line labels made blind,
|
| 35 |
independently of the training labels. Two careful labellers agree with each other at about 84% on the
|
|
|
|
| 134 |
| `key_lines` | (B, M) | per line: does a reader need it (a recall-heavy extract) | logit > 0 |
|
| 135 |
|
| 136 |
The **attention gate** is `g = max(sigmoid(needs_attention), P(approval) + P(question) + P(finished) + P(error))`,
|
| 137 |
+
mapped so that 0.4 reads as 0.5: `g < 0.4 ? 0.5·g/0.4 : 0.5 + 0.5·(g−0.4)/(1−0.4)`.
|
| 138 |
At ≥ 0.5 the agent needs the user. All of this is in `reader.json`, which the readers load.
|
| 139 |
|
| 140 |
The headline is picked to be **the most important line to look at right now**: the question or approval
|
|
|
|
| 144 |
|
| 145 |
## How well it works
|
| 146 |
|
| 147 |
+
INT8, as shipped. FP32 scores the same within noise (99.2% vs 99.4% category on test).
|
| 148 |
|
| 149 |
**Real screens** (never trained on): 60 hand-checked gold screens from real sessions and 93 researched
|
| 150 |
screens across 17 agent CLIs.
|
|
|
|
| 154 |
| Gold screens, category (question and finished counted together, as Canopy shows them) | 57/60 |
|
| 155 |
| Gold screens, attention gate right | 58/60 |
|
| 156 |
| Researched agent screens, category | 86/93 |
|
| 157 |
+
| Researched agent screens, attention gate right | 89/93 |
|
| 158 |
+
| Headline line = the blind reference label's | 75.8% |
|
| 159 |
+
| Options exactly right (screens with a menu) | 76.1% |
|
| 160 |
+
| Task line right | 96.1% |
|
| 161 |
+
| Whole card (headline + summary lines) F1 | 0.661 |
|
| 162 |
+
| Tests / changes field right (gold) | 56/60, 51/60 |
|
| 163 |
|
| 164 |
**Held-out synthetic test** (1,142 screens in agent × situation cells never seen in training):
|
| 165 |
+
category 99.4%, gate missed 3 screens that needed the user, question line
|
| 166 |
+
99.8%, headline line 93.0%, options exact 94.9%,
|
| 167 |
+
first summary line among the labelled ones 72.8%, tests 97.3%,
|
| 168 |
+
changes 91.4%, score within one anchor 96.7%.
|
| 169 |
|
| 170 |
## Training
|
| 171 |
|
bench.json
CHANGED
|
@@ -1,181 +1,176 @@
|
|
| 1 |
{
|
| 2 |
"fp32": {
|
| 3 |
"mb": 67.7,
|
| 4 |
-
"load_s": 0.
|
| 5 |
-
"ms_median":
|
| 6 |
-
"ms_p90":
|
| 7 |
"split": {
|
| 8 |
"n": 1142,
|
| 9 |
-
"category_acc": 0.
|
| 10 |
-
"category_macro_f1": 0.
|
| 11 |
-
"gate_acc": 0.
|
| 12 |
-
"gate_hid":
|
| 13 |
-
"gate_surfaced":
|
| 14 |
"ask_recall": 1.0,
|
| 15 |
-
"false_finished":
|
| 16 |
-
"blocked_agree":
|
| 17 |
-
"question_line_acc":
|
| 18 |
-
"status_line_acc": 0.
|
| 19 |
-
"key_recall": 0.
|
| 20 |
-
"key_precision": 0.
|
| 21 |
-
"notable_agree": 0.
|
| 22 |
-
"busy_notable_precision": 0.
|
| 23 |
-
"busy_notable_recall": 0.
|
| 24 |
"busy_n": 412,
|
| 25 |
"busy_notable_n": 145,
|
| 26 |
"confusions": {
|
| 27 |
-
"finished->working":
|
| 28 |
-
"
|
| 29 |
-
"
|
| 30 |
-
"
|
| 31 |
-
"idle->working": 1
|
| 32 |
},
|
| 33 |
"summary_n": 1142,
|
| 34 |
-
"summary_empty_agree": 0.
|
| 35 |
-
"summary_top1_in_gold": 0.
|
| 36 |
-
"summary_top1_exact": 0.
|
| 37 |
-
"summary_any_hit": 0.
|
| 38 |
-
"summary_precision": 0.
|
| 39 |
-
"summary_recall": 0.
|
| 40 |
-
"summary_f1": 0.
|
| 41 |
-
"summary_lines_per_card": 1.
|
| 42 |
-
"options_exact": 0.
|
| 43 |
-
"options_false_on_none": 0.
|
| 44 |
-
"action_line_acc": 0.
|
| 45 |
-
"task_line_acc": 0.
|
| 46 |
-
"tests_acc": 0.
|
| 47 |
-
"changes_acc": 0.
|
| 48 |
-
"risk_acc": 0.
|
| 49 |
"instruction_acc": 0.9903677758318739,
|
| 50 |
-
"progress_acc": 0.
|
| 51 |
-
"failure_repeats_acc": 0.
|
| 52 |
-
"score_acc": 0.
|
| 53 |
-
"progress_within_10": 0.
|
| 54 |
-
"score_within_1_anchor": 0.
|
| 55 |
},
|
| 56 |
"real": {
|
| 57 |
"gold": {
|
| 58 |
"n": 60,
|
| 59 |
"category_coarse": 57,
|
| 60 |
"category_exact": 56,
|
| 61 |
-
"gate":
|
| 62 |
"hid": 1,
|
| 63 |
-
"surfaced":
|
| 64 |
"misses": [
|
| 65 |
-
"
|
| 66 |
-
"
|
| 67 |
-
"
|
|
|
|
| 68 |
],
|
| 69 |
"tests": 56,
|
| 70 |
-
"changes":
|
| 71 |
-
"score":
|
| 72 |
-
"score_within_1_anchor":
|
| 73 |
"task_present_agree": 25
|
| 74 |
},
|
| 75 |
"agents": {
|
| 76 |
"n": 93,
|
| 77 |
-
"category_coarse":
|
| 78 |
-
"category_exact":
|
| 79 |
"gate": 90,
|
| 80 |
"hid": 3,
|
| 81 |
"surfaced": 0,
|
| 82 |
"misses": [
|
| 83 |
-
"aider#
|
| 84 |
-
"
|
| 85 |
-
"crush#2 [finished/crush] got working att=0.59 want finished",
|
| 86 |
-
"crush#3 [question/crush] got working att=0.30 want question",
|
| 87 |
"goose#3 [question/goose] got approval att=1.00 want question",
|
| 88 |
-
"
|
| 89 |
-
"mistral#1 [question/mistral] got approval att=1.00 want question"
|
| 90 |
-
"opencode#4 [error/opencode] got working att=0.67 want error"
|
| 91 |
]
|
| 92 |
},
|
| 93 |
"summary_vs_luna": {
|
| 94 |
"summary_n": 153,
|
| 95 |
-
"summary_empty_agree": 0.
|
| 96 |
-
"summary_top1_in_gold": 0.
|
| 97 |
-
"summary_top1_exact": 0.
|
| 98 |
-
"summary_any_hit": 0.
|
| 99 |
-
"summary_precision": 0.
|
| 100 |
-
"summary_recall": 0.
|
| 101 |
-
"summary_f1": 0.
|
| 102 |
-
"summary_lines_per_card": 1.
|
| 103 |
},
|
| 104 |
"lines_vs_gold": {
|
| 105 |
"n": 153,
|
| 106 |
-
"headline_acc": 0.
|
| 107 |
-
"task_acc": 0.
|
| 108 |
-
"options_exact": 0.
|
| 109 |
"options_false_on_none": 0.009345794392523364,
|
| 110 |
"summary_n": 153,
|
| 111 |
-
"summary_empty_agree": 0.
|
| 112 |
-
"summary_top1_in_gold": 0.
|
| 113 |
-
"summary_top1_exact": 0.
|
| 114 |
"summary_any_hit": 0.5655737704918032,
|
| 115 |
-
"summary_precision": 0.
|
| 116 |
-
"summary_recall": 0.
|
| 117 |
-
"summary_f1": 0.
|
| 118 |
-
"summary_lines_per_card": 1.
|
| 119 |
-
"card_any_hit": 0.
|
| 120 |
-
"card_precision": 0.
|
| 121 |
-
"card_recall": 0.
|
| 122 |
-
"card_f1": 0.
|
| 123 |
}
|
| 124 |
}
|
| 125 |
},
|
| 126 |
"int8": {
|
| 127 |
"mb": 17.3,
|
| 128 |
"load_s": 0.07,
|
| 129 |
-
"ms_median":
|
| 130 |
-
"ms_p90":
|
| 131 |
"split": {
|
| 132 |
"n": 1142,
|
| 133 |
-
"category_acc": 0.
|
| 134 |
-
"category_macro_f1": 0.
|
| 135 |
-
"gate_acc": 0.
|
| 136 |
-
"gate_hid":
|
| 137 |
-
"gate_surfaced":
|
| 138 |
"ask_recall": 1.0,
|
| 139 |
"false_finished": 1,
|
| 140 |
-
"blocked_agree":
|
| 141 |
-
"question_line_acc":
|
| 142 |
-
"status_line_acc": 0.
|
| 143 |
-
"key_recall": 0.
|
| 144 |
-
"key_precision": 0.
|
| 145 |
-
"notable_agree": 0.
|
| 146 |
-
"busy_notable_precision": 0.
|
| 147 |
-
"busy_notable_recall": 0.
|
| 148 |
"busy_n": 412,
|
| 149 |
"busy_notable_n": 145,
|
| 150 |
"confusions": {
|
| 151 |
-
"finished->working":
|
| 152 |
-
"
|
| 153 |
-
"working->
|
| 154 |
-
"finished->question": 1,
|
| 155 |
-
"idle->working": 1
|
| 156 |
},
|
| 157 |
"summary_n": 1142,
|
| 158 |
-
"summary_empty_agree": 0.
|
| 159 |
-
"summary_top1_in_gold": 0.
|
| 160 |
-
"summary_top1_exact": 0.
|
| 161 |
-
"summary_any_hit": 0.
|
| 162 |
-
"summary_precision": 0.
|
| 163 |
-
"summary_recall": 0.
|
| 164 |
-
"summary_f1": 0.
|
| 165 |
-
"summary_lines_per_card": 1.
|
| 166 |
"options_exact": 0.948905109489051,
|
| 167 |
-
"options_false_on_none": 0.
|
| 168 |
-
"action_line_acc": 0.
|
| 169 |
-
"task_line_acc": 0.
|
| 170 |
-
"tests_acc": 0.
|
| 171 |
"changes_acc": 0.9141856392294221,
|
| 172 |
"risk_acc": 0.9842381786339754,
|
| 173 |
"instruction_acc": 0.9903677758318739,
|
| 174 |
-
"progress_acc": 0.
|
| 175 |
"failure_repeats_acc": 0.9106830122591943,
|
| 176 |
-
"score_acc": 0.
|
| 177 |
-
"progress_within_10": 0.
|
| 178 |
-
"score_within_1_anchor": 0.
|
| 179 |
},
|
| 180 |
"real": {
|
| 181 |
"gold": {
|
|
@@ -183,66 +178,66 @@
|
|
| 183 |
"category_coarse": 57,
|
| 184 |
"category_exact": 56,
|
| 185 |
"gate": 58,
|
| 186 |
-
"hid":
|
| 187 |
-
"surfaced":
|
| 188 |
"misses": [
|
| 189 |
-
"
|
| 190 |
-
"
|
| 191 |
-
"03f3c93a3e [error/cursor] got idle att=0.
|
| 192 |
],
|
| 193 |
-
"tests":
|
| 194 |
-
"changes":
|
| 195 |
"score": 53,
|
| 196 |
-
"score_within_1_anchor":
|
| 197 |
"task_present_agree": 25
|
| 198 |
},
|
| 199 |
"agents": {
|
| 200 |
"n": 93,
|
| 201 |
"category_coarse": 86,
|
| 202 |
"category_exact": 86,
|
| 203 |
-
"gate":
|
| 204 |
-
"hid":
|
| 205 |
"surfaced": 0,
|
| 206 |
"misses": [
|
| 207 |
-
"aider#
|
| 208 |
-
"
|
| 209 |
-
"crush#2 [finished/crush] got working att=0.56 want finished",
|
| 210 |
-
"crush#3 [question/crush] got working att=0.27 want question",
|
| 211 |
-
"cursor#2 [finished/cursor] got working att=0.63 want finished",
|
| 212 |
"goose#3 [question/goose] got approval att=1.00 want question",
|
| 213 |
-
"
|
|
|
|
|
|
|
|
|
|
| 214 |
]
|
| 215 |
},
|
| 216 |
"summary_vs_luna": {
|
| 217 |
"summary_n": 153,
|
| 218 |
-
"summary_empty_agree": 0.
|
| 219 |
-
"summary_top1_in_gold": 0.
|
| 220 |
-
"summary_top1_exact": 0.
|
| 221 |
-
"summary_any_hit": 0.
|
| 222 |
-
"summary_precision": 0.
|
| 223 |
-
"summary_recall": 0.
|
| 224 |
-
"summary_f1": 0.
|
| 225 |
-
"summary_lines_per_card": 1.
|
| 226 |
},
|
| 227 |
"lines_vs_gold": {
|
| 228 |
"n": 153,
|
| 229 |
-
"headline_acc": 0.
|
| 230 |
-
"task_acc": 0.
|
| 231 |
-
"options_exact": 0.
|
| 232 |
"options_false_on_none": 0.009345794392523364,
|
| 233 |
"summary_n": 153,
|
| 234 |
-
"summary_empty_agree": 0.
|
| 235 |
-
"summary_top1_in_gold": 0.
|
| 236 |
-
"summary_top1_exact": 0.
|
| 237 |
"summary_any_hit": 0.5819672131147541,
|
| 238 |
-
"summary_precision": 0.
|
| 239 |
-
"summary_recall": 0.
|
| 240 |
-
"summary_f1": 0.
|
| 241 |
-
"summary_lines_per_card": 1.
|
| 242 |
-
"card_any_hit": 0.
|
| 243 |
-
"card_precision": 0.
|
| 244 |
-
"card_recall": 0.
|
| 245 |
-
"card_f1": 0.
|
| 246 |
}
|
| 247 |
}
|
| 248 |
}
|
|
|
|
| 1 |
{
|
| 2 |
"fp32": {
|
| 3 |
"mb": 67.7,
|
| 4 |
+
"load_s": 0.08,
|
| 5 |
+
"ms_median": 23.3,
|
| 6 |
+
"ms_p90": 49.9,
|
| 7 |
"split": {
|
| 8 |
"n": 1142,
|
| 9 |
+
"category_acc": 0.9921190893169878,
|
| 10 |
+
"category_macro_f1": 0.9933529790794,
|
| 11 |
+
"gate_acc": 0.9912434325744308,
|
| 12 |
+
"gate_hid": 4,
|
| 13 |
+
"gate_surfaced": 6,
|
| 14 |
"ask_recall": 1.0,
|
| 15 |
+
"false_finished": 3,
|
| 16 |
+
"blocked_agree": 1.0,
|
| 17 |
+
"question_line_acc": 0.999124343257443,
|
| 18 |
+
"status_line_acc": 0.9299474605954466,
|
| 19 |
+
"key_recall": 0.9470030784088851,
|
| 20 |
+
"key_precision": 0.886634642554394,
|
| 21 |
+
"notable_agree": 0.9807355516637478,
|
| 22 |
+
"busy_notable_precision": 0.9629629629629629,
|
| 23 |
+
"busy_notable_recall": 0.896551724137931,
|
| 24 |
"busy_n": 412,
|
| 25 |
"busy_notable_n": 145,
|
| 26 |
"confusions": {
|
| 27 |
+
"finished->working": 5,
|
| 28 |
+
"working->finished": 2,
|
| 29 |
+
"idle->finished": 1,
|
| 30 |
+
"working->question": 1
|
|
|
|
| 31 |
},
|
| 32 |
"summary_n": 1142,
|
| 33 |
+
"summary_empty_agree": 0.9238178633975481,
|
| 34 |
+
"summary_top1_in_gold": 0.7219679633867276,
|
| 35 |
+
"summary_top1_exact": 0.43363844393592677,
|
| 36 |
+
"summary_any_hit": 0.8615560640732265,
|
| 37 |
+
"summary_precision": 0.6434599156118144,
|
| 38 |
+
"summary_recall": 0.6202338586680224,
|
| 39 |
+
"summary_f1": 0.6316334455086721,
|
| 40 |
+
"summary_lines_per_card": 1.660245183887916,
|
| 41 |
+
"options_exact": 0.9525547445255474,
|
| 42 |
+
"options_false_on_none": 0.005787037037037037,
|
| 43 |
+
"action_line_acc": 0.9754816112084063,
|
| 44 |
+
"task_line_acc": 0.9562171628721541,
|
| 45 |
+
"tests_acc": 0.9693520140105079,
|
| 46 |
+
"changes_acc": 0.9124343257443083,
|
| 47 |
+
"risk_acc": 0.9859894921190894,
|
| 48 |
"instruction_acc": 0.9903677758318739,
|
| 49 |
+
"progress_acc": 0.5893169877408057,
|
| 50 |
+
"failure_repeats_acc": 0.9115586690017513,
|
| 51 |
+
"score_acc": 0.8450087565674256,
|
| 52 |
+
"progress_within_10": 0.7241681260945709,
|
| 53 |
+
"score_within_1_anchor": 0.9667250437828371
|
| 54 |
},
|
| 55 |
"real": {
|
| 56 |
"gold": {
|
| 57 |
"n": 60,
|
| 58 |
"category_coarse": 57,
|
| 59 |
"category_exact": 56,
|
| 60 |
+
"gate": 56,
|
| 61 |
"hid": 1,
|
| 62 |
+
"surfaced": 3,
|
| 63 |
"misses": [
|
| 64 |
+
"c8f4b58a8e [working/claude] got error att=0.81 want working|idle",
|
| 65 |
+
"0ee96e895b [working/claude] got finished att=0.87 want working",
|
| 66 |
+
"f19b520cb2 [interrupted/codex] got working att=0.51 want working",
|
| 67 |
+
"03f3c93a3e [error/cursor] got idle att=0.40 want error"
|
| 68 |
],
|
| 69 |
"tests": 56,
|
| 70 |
+
"changes": 52,
|
| 71 |
+
"score": 51,
|
| 72 |
+
"score_within_1_anchor": 55,
|
| 73 |
"task_present_agree": 25
|
| 74 |
},
|
| 75 |
"agents": {
|
| 76 |
"n": 93,
|
| 77 |
+
"category_coarse": 88,
|
| 78 |
+
"category_exact": 88,
|
| 79 |
"gate": 90,
|
| 80 |
"hid": 3,
|
| 81 |
"surfaced": 0,
|
| 82 |
"misses": [
|
| 83 |
+
"aider#1 [approval/aider] got working att=0.16 want approval",
|
| 84 |
+
"crush#3 [question/crush] got idle att=0.19 want question",
|
|
|
|
|
|
|
| 85 |
"goose#3 [question/goose] got approval att=1.00 want question",
|
| 86 |
+
"kiro#3 [interrupted/kiro] got working att=0.41 want finished",
|
| 87 |
+
"mistral#1 [question/mistral] got approval att=1.00 want question"
|
|
|
|
| 88 |
]
|
| 89 |
},
|
| 90 |
"summary_vs_luna": {
|
| 91 |
"summary_n": 153,
|
| 92 |
+
"summary_empty_agree": 0.7450980392156863,
|
| 93 |
+
"summary_top1_in_gold": 0.6296296296296297,
|
| 94 |
+
"summary_top1_exact": 0.49074074074074076,
|
| 95 |
+
"summary_any_hit": 0.6666666666666666,
|
| 96 |
+
"summary_precision": 0.6369047619047619,
|
| 97 |
+
"summary_recall": 0.5572916666666666,
|
| 98 |
+
"summary_f1": 0.5944444444444444,
|
| 99 |
+
"summary_lines_per_card": 1.0980392156862746
|
| 100 |
},
|
| 101 |
"lines_vs_gold": {
|
| 102 |
"n": 153,
|
| 103 |
+
"headline_acc": 0.7581699346405228,
|
| 104 |
+
"task_acc": 0.954248366013072,
|
| 105 |
+
"options_exact": 0.782608695652174,
|
| 106 |
"options_false_on_none": 0.009345794392523364,
|
| 107 |
"summary_n": 153,
|
| 108 |
+
"summary_empty_agree": 0.7189542483660131,
|
| 109 |
+
"summary_top1_in_gold": 0.48360655737704916,
|
| 110 |
+
"summary_top1_exact": 0.3770491803278688,
|
| 111 |
"summary_any_hit": 0.5655737704918032,
|
| 112 |
+
"summary_precision": 0.5714285714285714,
|
| 113 |
+
"summary_recall": 0.3950617283950617,
|
| 114 |
+
"summary_f1": 0.46715328467153283,
|
| 115 |
+
"summary_lines_per_card": 1.0980392156862746,
|
| 116 |
+
"card_any_hit": 0.8620689655172413,
|
| 117 |
+
"card_precision": 0.7686832740213523,
|
| 118 |
+
"card_recall": 0.5567010309278351,
|
| 119 |
+
"card_f1": 0.6457399103139013
|
| 120 |
}
|
| 121 |
}
|
| 122 |
},
|
| 123 |
"int8": {
|
| 124 |
"mb": 17.3,
|
| 125 |
"load_s": 0.07,
|
| 126 |
+
"ms_median": 24.5,
|
| 127 |
+
"ms_p90": 50.8,
|
| 128 |
"split": {
|
| 129 |
"n": 1142,
|
| 130 |
+
"category_acc": 0.9938704028021016,
|
| 131 |
+
"category_macro_f1": 0.9947484950966307,
|
| 132 |
+
"gate_acc": 0.9929947460595446,
|
| 133 |
+
"gate_hid": 3,
|
| 134 |
+
"gate_surfaced": 5,
|
| 135 |
"ask_recall": 1.0,
|
| 136 |
"false_finished": 1,
|
| 137 |
+
"blocked_agree": 1.0,
|
| 138 |
+
"question_line_acc": 0.9982486865148862,
|
| 139 |
+
"status_line_acc": 0.9299474605954466,
|
| 140 |
+
"key_recall": 0.9529184523450233,
|
| 141 |
+
"key_precision": 0.8797436611869601,
|
| 142 |
+
"notable_agree": 0.978108581436077,
|
| 143 |
+
"busy_notable_precision": 0.9624060150375939,
|
| 144 |
+
"busy_notable_recall": 0.8827586206896552,
|
| 145 |
"busy_n": 412,
|
| 146 |
"busy_notable_n": 145,
|
| 147 |
"confusions": {
|
| 148 |
+
"finished->working": 5,
|
| 149 |
+
"idle->finished": 1,
|
| 150 |
+
"working->error": 1
|
|
|
|
|
|
|
| 151 |
},
|
| 152 |
"summary_n": 1142,
|
| 153 |
+
"summary_empty_agree": 0.9290718038528897,
|
| 154 |
+
"summary_top1_in_gold": 0.7276887871853547,
|
| 155 |
+
"summary_top1_exact": 0.42105263157894735,
|
| 156 |
+
"summary_any_hit": 0.8764302059496567,
|
| 157 |
+
"summary_precision": 0.6371951219512195,
|
| 158 |
+
"summary_recall": 0.6375190645653279,
|
| 159 |
+
"summary_f1": 0.6373570520965692,
|
| 160 |
+
"summary_lines_per_card": 1.723292469352014,
|
| 161 |
"options_exact": 0.948905109489051,
|
| 162 |
+
"options_false_on_none": 0.004629629629629629,
|
| 163 |
+
"action_line_acc": 0.9728546409807356,
|
| 164 |
+
"task_line_acc": 0.9535901926444834,
|
| 165 |
+
"tests_acc": 0.9728546409807356,
|
| 166 |
"changes_acc": 0.9141856392294221,
|
| 167 |
"risk_acc": 0.9842381786339754,
|
| 168 |
"instruction_acc": 0.9903677758318739,
|
| 169 |
+
"progress_acc": 0.5831873905429071,
|
| 170 |
"failure_repeats_acc": 0.9106830122591943,
|
| 171 |
+
"score_acc": 0.841506129597198,
|
| 172 |
+
"progress_within_10": 0.7189141856392294,
|
| 173 |
+
"score_within_1_anchor": 0.9667250437828371
|
| 174 |
},
|
| 175 |
"real": {
|
| 176 |
"gold": {
|
|
|
|
| 178 |
"category_coarse": 57,
|
| 179 |
"category_exact": 56,
|
| 180 |
"gate": 58,
|
| 181 |
+
"hid": 0,
|
| 182 |
+
"surfaced": 2,
|
| 183 |
"misses": [
|
| 184 |
+
"c8f4b58a8e [working/claude] got finished att=0.79 want working|idle",
|
| 185 |
+
"0ee96e895b [working/claude] got finished att=0.77 want working",
|
| 186 |
+
"03f3c93a3e [error/cursor] got idle att=0.54 want error"
|
| 187 |
],
|
| 188 |
+
"tests": 56,
|
| 189 |
+
"changes": 51,
|
| 190 |
"score": 53,
|
| 191 |
+
"score_within_1_anchor": 56,
|
| 192 |
"task_present_agree": 25
|
| 193 |
},
|
| 194 |
"agents": {
|
| 195 |
"n": 93,
|
| 196 |
"category_coarse": 86,
|
| 197 |
"category_exact": 86,
|
| 198 |
+
"gate": 89,
|
| 199 |
+
"hid": 4,
|
| 200 |
"surfaced": 0,
|
| 201 |
"misses": [
|
| 202 |
+
"aider#1 [approval/aider] got working att=0.16 want approval",
|
| 203 |
+
"crush#3 [question/crush] got idle att=0.21 want question",
|
|
|
|
|
|
|
|
|
|
| 204 |
"goose#3 [question/goose] got approval att=1.00 want question",
|
| 205 |
+
"interpreter#0 [approval/interpreter] got working att=0.41 want approval",
|
| 206 |
+
"kiro#3 [interrupted/kiro] got working att=0.51 want finished",
|
| 207 |
+
"mistral#1 [question/mistral] got approval att=1.00 want question",
|
| 208 |
+
"opencode#4 [error/opencode] got working att=0.34 want error"
|
| 209 |
]
|
| 210 |
},
|
| 211 |
"summary_vs_luna": {
|
| 212 |
"summary_n": 153,
|
| 213 |
+
"summary_empty_agree": 0.7581699346405228,
|
| 214 |
+
"summary_top1_in_gold": 0.6111111111111112,
|
| 215 |
+
"summary_top1_exact": 0.4351851851851852,
|
| 216 |
+
"summary_any_hit": 0.6759259259259259,
|
| 217 |
+
"summary_precision": 0.6566265060240963,
|
| 218 |
+
"summary_recall": 0.5677083333333334,
|
| 219 |
+
"summary_f1": 0.6089385474860336,
|
| 220 |
+
"summary_lines_per_card": 1.0849673202614378
|
| 221 |
},
|
| 222 |
"lines_vs_gold": {
|
| 223 |
"n": 153,
|
| 224 |
+
"headline_acc": 0.7581699346405228,
|
| 225 |
+
"task_acc": 0.9607843137254902,
|
| 226 |
+
"options_exact": 0.7608695652173914,
|
| 227 |
"options_false_on_none": 0.009345794392523364,
|
| 228 |
"summary_n": 153,
|
| 229 |
+
"summary_empty_agree": 0.7320261437908496,
|
| 230 |
+
"summary_top1_in_gold": 0.5163934426229508,
|
| 231 |
+
"summary_top1_exact": 0.36885245901639346,
|
| 232 |
"summary_any_hit": 0.5819672131147541,
|
| 233 |
+
"summary_precision": 0.5963855421686747,
|
| 234 |
+
"summary_recall": 0.4074074074074074,
|
| 235 |
+
"summary_f1": 0.4841075794621027,
|
| 236 |
+
"summary_lines_per_card": 1.0849673202614378,
|
| 237 |
+
"card_any_hit": 0.8758620689655172,
|
| 238 |
+
"card_precision": 0.7816901408450704,
|
| 239 |
+
"card_recall": 0.5721649484536082,
|
| 240 |
+
"card_f1": 0.6607142857142857
|
| 241 |
}
|
| 242 |
}
|
| 243 |
}
|
model.fp32.onnx
CHANGED
|
@@ -1,3 +1,3 @@
|
|
| 1 |
version https://git-lfs.github.com/spec/v1
|
| 2 |
-
oid sha256:
|
| 3 |
size 67726050
|
|
|
|
| 1 |
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:fe6fb4d76e68ae79bb25a833394912a420d0a0654390bdc5ca7fddc9ebd5d011
|
| 3 |
size 67726050
|
model.int8.onnx
CHANGED
|
@@ -1,3 +1,3 @@
|
|
| 1 |
version https://git-lfs.github.com/spec/v1
|
| 2 |
-
oid sha256:
|
| 3 |
size 17298433
|
|
|
|
| 1 |
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:21807b2c66b4e6e53e28e290538a53de93228eec2e760c127e19d469aa9b2136
|
| 3 |
size 17298433
|
reader.json
CHANGED
|
@@ -121,7 +121,7 @@
|
|
| 121 |
"summary_first": 0.5,
|
| 122 |
"summary_more": 0.6,
|
| 123 |
"option": 0.5,
|
| 124 |
-
"gate": 0.
|
| 125 |
"gate_categories": [
|
| 126 |
"approval",
|
| 127 |
"question",
|
|
|
|
| 121 |
"summary_first": 0.5,
|
| 122 |
"summary_more": 0.6,
|
| 123 |
"option": 0.5,
|
| 124 |
+
"gate": 0.4,
|
| 125 |
"gate_categories": [
|
| 126 |
"approval",
|
| 127 |
"question",
|