File size: 12,594 Bytes
dfe1826
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
## Method notes (interpretations of `docs/analysis_plan.md` and data anomalies)

Hand-written; appended verbatim to `results.md` by `make_tables.py`. Each item is a point where the plan
had to be read literally or was silent, or a fact found in the data.

### Analysis code

1. **Analysis code.** Plan §7 places the statistics in `src/formal_mas/analysis/stats.py`. The first version of
   this analysis implemented them inside `docs/results/make_tables.py` (the file did not exist yet); on
   2026-11-07, by team decision, they were moved to `src/formal_mas/analysis/stats.py` (Wilson, exact McNemar,
   Bonferroni, bootstrap, quartiles), with offline tests against closed-form values in `tests/test_stats.py`.
   `make_tables.py` imports them, and the regenerated `results.md` and `tables.tex` were checked to be byte for
   byte identical to the earlier output (before Table 10 was switched to the post-test fidelity review). Exact binomial tests and Wilson intervals come from
   `scipy.stats.binomtest` (Wilson without continuity correction). The bootstrap uses
   `numpy.random.default_rng`. The κ classifier is imported read-only from `src/formal_mas/orchestrator/policy.py`.
   `scripts/compare.py` was not written: `make_tables.py` produces every table of §7.
2. **Dev before test (plan §7).** The script was first run in `--dev` mode on dev runs, using stand-ins on the
   same 15 dev problems, and checked there: Wilson 11/15 = [48.0, 89.1], exact McNemar p values computed by
   hand, and the C1 κ classification on `20261103T161957Z_C1_experiment_c1_cost`. Unit tests of the statistics
   were added later (item 1).
   Disclosure: before the script existed, the analyst had seen the first `test` row of C5 (`mathd_algebra_206`,
   solved) and the summaries stored in the `run_config.json` of the C5 resume runs and of C4, while checking the
   configurations. After the first `test` run, the script changed in three ways: (a) C2 wall time now stops at
   `claim_check_start` (the row's `wall_s` includes the claim check, and the 2026-11-03 Clarification excludes
   the check from C2's figures); (b) a note on responses cut at `max_tokens` was added; (c) LaTeX layout. No test,
   comparison, correction or metric of the plan was changed.

### Merges and integrity

3. **C5 merge** follows the C5 Deviations entry exactly. It keeps 23 rows of `20261103T174646Z_C5_experiment`,
   `imo_1987_p6` and `imo_1979_p1` of `..._resume`, and 5 rows of `..._resume2`. The `budget_stop` row of
   `imo_1984_p6` in resume 1 is discarded; the script asserts its status. The script fails if a row is kept twice
   or if a source row is neither kept nor discarded.
4. **C1** uses `runs/20261107T200233Z_C1_experiment_resume/C1_test_merged_results.jsonl`. Every merged row equals
   (as parsed JSON) exactly one source row: 23 from `20261105T131443Z_C1_experiment` and 7 from the resume.
5. All five configurations have exactly 30 rows with the same 30 `test` ids. There is no `provider_error`,
   `budget_stop`, `error`, `node_error`, `invalid_statement` or `lean_down` row, and no degraded run. The plan's
   re-run rule and sensitivity analysis (§5) therefore do not apply. Area and difficulty in each row match
   `data/eval.jsonl`.
6. All `run_config.json` files agree on profile, seed, split, Lean toolchain, Mathlib revision and model settings.
   `git diff ad70ec2 2c8eae5 -- src config` shows that only `config/pipeline.yaml: spend.run_usd` changed (the
   documented spend-cap deviations). The input sha256 of the resume runs match the Deviations entry.
   **Anomaly:** `run_config.json` records `dirty: true` for C5 resume 1, C5 resume 2, C4 and the C1 resume, with
   `changed_files = [docs/analysis_plan.md]` only. The Deviations entry calls these trees "clean". The only
   uncommitted file was the plan itself.
7. The first C5 run and the first C1 run have run status `running`: they were killed, as documented. Their
   results files hold only finished problems.

### Primary outcome and comparisons

8. **C2 in the SR table.** Following §7.1, C2 has no verified SR. It enters through its claimed rate
   |B⁺|/30 = 30/30 and FPR_B. The 19 claims that the check verified are reported as a count only, because the
   plan defines no SR for them. C2 is not in any McNemar test (§3). In the per-cell and margin tables, C2 shows
   **claimed** counts, labelled as such, because §4.1 says "for each configuration".
9. **McNemar:** b = first configuration solved and second not; c = the reverse; two-sided exact binomial on
   b | b + c with p = 1/2 (p = 1 if b + c = 0). Bonferroni-adjusted p = min(1, 3p); a test is significant if
   p < 0.05/3.
10. **Paired bootstrap** (plan: 10 000 resamples of problems, seed 2026, percentile). The generator is a fresh
    `numpy.random.default_rng(2026)` for each statistic, so all comparisons use the same 10 000 index vectors.
    The plan does not name a generator. For H2 the lower bound is exactly 0.0 pp.
11. Exploratory comparisons (C3 vs C4, C1 vs C3, C1 vs C4) get b, c, ΔSR and an uncorrected exact p only. The
    plan specifies no bootstrap CI for them.

### FPR_B (Eq. 16, §4.2, Clarification 2026-11-03)

12. B⁺ holds all 30 problems (every C2 row is `claimed_solved`). Check status: 19 `verified`, 11 `not_verified`.
    No check ended in `budget_exhausted`, `provider_error`, `budget_stop`, `error` or `lean_down`, so none is
    left out. FPR_B = 11/30.
13. **Manual-review sample.** 11 claims are not verified, which is more than 10, so 10 are drawn "with seed 2026".
    The plan does not give the drawing procedure; `random.Random(2026).sample(sorted(ids), 10)` was used, which
    leaves out `amc12a_2008_p15`. The manual review itself (informal proof wrong vs. formalization failure) is a
    task for the team and has not been done.
14. Factual cross-reference, not a conclusion: for 5 of the 11 not-verified claims (`amc12a_2008_p15`,
    `amc12b_2002_p3`, `imo_1964_p1_2`, `imo_1965_p2`, `putnam_1965_b2`), C5 has a kernel-accepted proof of the
    same formal statement.

### Efficiency (§4.3)

15. C(x) = `llm.llm_calls` of the row. It includes cache hits and JSON-repair calls, as the budget counts them.
    Tokens = prompt + completion (reasoning tokens are part of completion in the ledger). Nominal cost
    (`cost_usd_nominal`) is the compared figure (§1). Real cost is shown as well.
16. **Real cost.** The real cost of every kept row equals the ledger sum for (run_id, problem_id); no mismatch was
    found. The ledger totals include spend that is in no kept row:
    C1 +0.09845 USD (`imo_1987_p6` in the first C1 run: 10 calls, 5 paid and 5 cached JSON repairs), so
    C1 = 2.4695 + 0.0985 = 2.5679 USD.
    C5 +0.18153 USD (`imo_1987_p6` in the first run: 0.02309 USD, 8 paid calls; `imo_1984_p6` in resume 1:
    0.15844 USD, 15 paid calls), so C5 = 1.1770 + 0.1815 = 1.3585 USD.
    C2 = 0.0814 (C2 itself) + 0.6269 (claim check) = 0.7083 USD. C3 = 1.3500 USD. C4 = 0.0000 USD.
    Per-problem and per-solved figures come from kept rows only.
17. **C2 efficiency** leaves out the claim check: its calls, tokens and cost come from the row's `llm`, and its
    wall time runs from `problem_start` to `claim_check_start`. C̄_solved needs a "solved" count, and C2 has no
    verified one, so its ratio uses claimed solutions (30).
18. **C4:** all 132 LLM calls were cache hits (requests identical to C5's), so its real cost is 0 and its LLM
    latency is near zero. Lean verification was real (5 of 259 `lean_verify` events were Lean-cache hits). C4's
    wall time (about 0.1 min per problem) therefore does not measure LLM computation. Nominal cost is the
    comparable figure (§1).
19. Bootstrap CIs for the efficiency ratios: plan §4.3 gives no seed, so the same procedure as note 10 is used
    (seed 2026, percentile). For Σx/#solved, resamples with no solved problem would be dropped; none occurred on
    `test`.
20. The median/IQR table (4d) and wall time by difficulty (Table 5) were requested by the team. They are
    descriptive only and not in the plan.

### Refinement robustness (§4.4, Eq. 14)

21. **Accepted plan.** The trace of a solved C5 problem is split at each `plan_proposed` / `plan_fallback` that
    follows subgoal work (attempt counters restart after a replan). The accepted plan is the last segment.
    "Attempts on a subgoal" is the `attempts` field of `subgoal_solved`; pre-pass subgoals count as 0. Read
    literally ("its accepted proof used at most k attempts on every subgoal"), a replanned problem counts by its
    accepted plan only. This affects `algebra_absapbon1pabsapbleqsumabsaon1pabsa`: C5 replanned once, then
    solved every subgoal on its first attempt, so it counts at k = 1. It is the only problem that differs
    between SR_C5(1) = 15/30 and SR_C4 = 14/30. SR_C5(3) = SR_C5 = 24/30, as expected.
22. **SR(0).** Δ̂(1) = SR(1) − SR(0) needs SR(0), which the plan does not define. The same rule with k = 0 is
    used: the accepted plan has no LLM-attempted subgoal (pre-pass or automatic closing only). This gives 9/30.
23. **Independence curve.** The plan says p_i is "estimated from first-attempt subgoal success in C5". Each
    subgoal has one first attempt, so a per-subgoal p_i would be 0 or 1. A pooled p̂ = 18/44 is used instead,
    counting every LLM-attempted subgoal of every plan (including abandoned plans and failed problems). The curve
    reported is Eq. 14's per-subgoal marginal gain Δ(k) = p̂(1 − p̂)^(k−1). It sits at a different level
    (subgoal) from the problem-level Δ̂(k). The last two columns of Table 6 give the subgoal-level empirical
    counterpart and are supplementary. The figure the plan asks for (§7.5) is
    `docs/results/figures/srk_curve.pdf` (and `.png`), drawn from the same numbers: (a) SR_C5(k), k = 0..3, with
    Wilson CIs and C4 at k = 1; (b) Δ̂(k) against p̂(1 − p̂)^(k−1).

### Error taxonomy (§4.5)

24. For C3–C5 the classes are the rows' `error_classes`, one per Critic event (checked equal to the
    `critique` events), including the k_max escalation to `planning`. Only kept rows count; the discarded
    attempts of the merges are excluded.
25. **C1** attempts are classified offline with `policy.classify(Trace(reason, messages, None))`, using the
    stored `lean_errors` (at most 3 messages of 300 characters, versus 5 × 1500 in the graph). The statement end
    is unknown, so `misformalization` cannot fire, which fits since C1 never writes statements. `escalate` is
    not applied because C1 has no k_max or strategies. The 44 `sanitize:empty` attempts (classed as `syntax`)
    happen alongside 52 C1 responses that ended at `max_tokens`.
26. **C2** has no verified attempts of its own. Its taxonomy row is the claim-check loop (labelled). The plan
    does not say how C2 enters §4.5.
27. Final failure reasons group stages by prefix (`verify:sg2` becomes `verify`).

### Other data facts

28. Responses that ended at `max_tokens` (`finish_reason = length`): C1 52 (9 rows), C3 18 (6 rows),
    C5 14 (5 rows), C4 2 (2 rows), C2 0 (12 in the claim check).
29. C1 `budget_exhausted` rows (7) reached b_max = 18 calls after 9–17 recorded attempts, because JSON-repair
    calls count against the budget. The 3 `failed` C1 rows used all 18 attempts.
30. **Fidelity (§6).** Four solved test problems carry a fidelity note: `amc12b_2003_p6`, `mathd_algebra_234`,
    `mathd_algebra_455` and `putnam_1965_b2`. The team reviewed them on 2026-11-07 (`post_test_review` in
    `data/curation/fidelity_review.yaml`; Table 10): all four count as solved, `amc12b_2003_p6` with a note in the
    report. The verified SR is reported unchanged, as §6 requires.
31. **Supplementary, not in the plan:** Table 8 (pre-pass vs LLM `solved_by`) and Table 4d.

### Illustrative traces (for the appendix)

- `amc12a_2008_p15` (Critic loop). C5 solved it on the third attempt at its only subgoal (two `tactic_failure`
  retries with Critic feedback, 16 calls, first C5 run). C4 abandoned after the first failure. C3 failed after a
  replan (30 calls). C1 failed. The C2 claim was not verified.
- `numbertheory_sumkmulnckeqnmul2pownm1` (RAG). C5 solved it with 5 calls on the first attempt. C3 failed after
  4 `missing_lemma` and 2 `tactic_failure` attempts over two plans (30 calls).
- `imo_1965_p2` (FPR_B as an upper bound). C2 claimed it, and its check failed (`missing_lemma`,
  `tactic_failure`, `missing_lemma`). C5 found a kernel-accepted proof: the pre-pass closed one subgoal and the
  LLM closed the other two (one retry after `missing_lemma`).