primce commited on
Commit
2ad96fb
·
verified ·
1 Parent(s): 72b3867

State the oracle call count per split

Browse files
Files changed (1) hide show
  1. README.md +2 -1
README.md CHANGED
@@ -56,7 +56,8 @@ seed adapters are kept for the ablation and variance controls.
56
  round-2 adapter, **18 fixed, 2 broken, p = 0.0004**; against the round-1 adapter, 7 fixed,
57
  3 broken, p = 0.34 (directional, not significant).
58
  - Tool calls per task: **3.57** on hard-30, exactly the oracle's count (base: 7.03). On hard-90:
59
- 3.38 against a base 7.24.
 
60
  - Tool error rate **0.00%** on both splits, loop collapse **0.00%**, redundant calls **0.00%**,
61
  parse failures **0.00%** (base: 22.3% tool errors, 16 of 30 episodes never finishing,
62
  `overcall` 1.08).
 
56
  round-2 adapter, **18 fixed, 2 broken, p = 0.0004**; against the round-1 adapter, 7 fixed,
57
  3 broken, p = 0.34 (directional, not significant).
58
  - Tool calls per task: **3.57** on hard-30, exactly the oracle's count (base: 7.03). On hard-90:
59
+ 3.38 against an oracle 3.41 and a base 7.24 — 87 of 90 tasks take exactly the oracle number of
60
+ calls, 3 take one fewer, none take more.
61
  - Tool error rate **0.00%** on both splits, loop collapse **0.00%**, redundant calls **0.00%**,
62
  parse failures **0.00%** (base: 22.3% tool errors, 16 of 30 episodes never finishing,
63
  `overcall` 1.08).