Model card: confirmed result
Browse files
README.md
CHANGED
|
@@ -26,7 +26,10 @@ mildly optimistic. Costs: billed tokens, Batch API prices (50% off); double them
|
|
| 26 |
| Claude Sonnet 5.5 + 16 retrieved examples | 857 | 862 | 0% | 4.80 |
|
| 27 |
| **Milo Pointer, defer to Sonnet 5.5** | **881** | **889** | **70.7%** | **1.41** |
|
| 28 |
|
| 29 |
-
|
|
|
|
|
|
|
|
|
|
| 30 |
its calls. On the steps the pointer keeps it is right 94.9% of the time, against 93.5% for Opus on the same steps.
|
| 31 |
|
| 32 |
Full AndroidControl test set (8,444 steps, strict metric): 88.5% step accuracy; IDD 89.8, unseen app 87.9, unseen task 87.2,
|
|
|
|
| 26 |
| Claude Sonnet 5.5 + 16 retrieved examples | 857 | 862 | 0% | 4.80 |
|
| 27 |
| **Milo Pointer, defer to Sonnet 5.5** | **881** | **889** | **70.7%** | **1.41** |
|
| 28 |
|
| 29 |
+
A further registered test on the last 650 unused test steps, with a learned hand-off rule frozen in advance, found routing to Opus
|
| 30 |
+
**more accurate than Opus alone: 599 vs 581 (+2.8 points, 95% CI [+1.0, +4.5], p = 0.003)**, with 73.7% of steps kept local.
|
| 31 |
+
|
| 32 |
+
On the 1,000 steps above, the cascade is non-inferior to Opus alone (+1.0 points, 95% CI [-0.2, +2.2]; not significantly better, p = 0.14) and uses 29% of
|
| 33 |
its calls. On the steps the pointer keeps it is right 94.9% of the time, against 93.5% for Opus on the same steps.
|
| 34 |
|
| 35 |
Full AndroidControl test set (8,444 steps, strict metric): 88.5% step accuracy; IDD 89.8, unseen app 87.9, unseen task 87.2,
|