coolnow commited on
Commit
29b5f0b
·
verified ·
1 Parent(s): 4b78807

Model card: confirmed result

Browse files
Files changed (1) hide show
  1. README.md +4 -1
README.md CHANGED
@@ -26,7 +26,10 @@ mildly optimistic. Costs: billed tokens, Batch API prices (50% off); double them
26
  | Claude Sonnet 5.5 + 16 retrieved examples | 857 | 862 | 0% | 4.80 |
27
  | **Milo Pointer, defer to Sonnet 5.5** | **881** | **889** | **70.7%** | **1.41** |
28
 
29
- The cascade is non-inferior to Opus alone (+1.0 points, 95% CI [-0.2, +2.2]; not significantly better, p = 0.14) and uses 29% of
 
 
 
30
  its calls. On the steps the pointer keeps it is right 94.9% of the time, against 93.5% for Opus on the same steps.
31
 
32
  Full AndroidControl test set (8,444 steps, strict metric): 88.5% step accuracy; IDD 89.8, unseen app 87.9, unseen task 87.2,
 
26
  | Claude Sonnet 5.5 + 16 retrieved examples | 857 | 862 | 0% | 4.80 |
27
  | **Milo Pointer, defer to Sonnet 5.5** | **881** | **889** | **70.7%** | **1.41** |
28
 
29
+ A further registered test on the last 650 unused test steps, with a learned hand-off rule frozen in advance, found routing to Opus
30
+ **more accurate than Opus alone: 599 vs 581 (+2.8 points, 95% CI [+1.0, +4.5], p = 0.003)**, with 73.7% of steps kept local.
31
+
32
+ On the 1,000 steps above, the cascade is non-inferior to Opus alone (+1.0 points, 95% CI [-0.2, +2.2]; not significantly better, p = 0.14) and uses 29% of
33
  its calls. On the steps the pointer keeps it is right 94.9% of the time, against 93.5% for Opus on the same steps.
34
 
35
  Full AndroidControl test set (8,444 steps, strict metric): 88.5% step accuracy; IDD 89.8, unseen app 87.9, unseen task 87.2,