formal-mas / docs /results /method_notes.md
NicolasT08's picture
Formal-MAS source code
dfe1826 verified
|
Raw History Blame Contribute Delete
12.6 kB

Method notes (interpretations of docs/analysis_plan.md and data anomalies)

Hand-written; appended verbatim to results.md by make_tables.py. Each item is a point where the plan had to be read literally or was silent, or a fact found in the data.

Analysis code

  1. Analysis code. Plan §7 places the statistics in src/formal_mas/analysis/stats.py. The first version of this analysis implemented them inside docs/results/make_tables.py (the file did not exist yet); on 2026-11-07, by team decision, they were moved to src/formal_mas/analysis/stats.py (Wilson, exact McNemar, Bonferroni, bootstrap, quartiles), with offline tests against closed-form values in tests/test_stats.py. make_tables.py imports them, and the regenerated results.md and tables.tex were checked to be byte for byte identical to the earlier output (before Table 10 was switched to the post-test fidelity review). Exact binomial tests and Wilson intervals come from scipy.stats.binomtest (Wilson without continuity correction). The bootstrap uses numpy.random.default_rng. The κ classifier is imported read-only from src/formal_mas/orchestrator/policy.py. scripts/compare.py was not written: make_tables.py produces every table of §7.
  2. Dev before test (plan §7). The script was first run in --dev mode on dev runs, using stand-ins on the same 15 dev problems, and checked there: Wilson 11/15 = [48.0, 89.1], exact McNemar p values computed by hand, and the C1 κ classification on 20261103T161957Z_C1_experiment_c1_cost. Unit tests of the statistics were added later (item 1). Disclosure: before the script existed, the analyst had seen the first test row of C5 (mathd_algebra_206, solved) and the summaries stored in the run_config.json of the C5 resume runs and of C4, while checking the configurations. After the first test run, the script changed in three ways: (a) C2 wall time now stops at claim_check_start (the row's wall_s includes the claim check, and the 2026-11-03 Clarification excludes the check from C2's figures); (b) a note on responses cut at max_tokens was added; (c) LaTeX layout. No test, comparison, correction or metric of the plan was changed.

Merges and integrity

  1. C5 merge follows the C5 Deviations entry exactly. It keeps 23 rows of 20261103T174646Z_C5_experiment, imo_1987_p6 and imo_1979_p1 of ..._resume, and 5 rows of ..._resume2. The budget_stop row of imo_1984_p6 in resume 1 is discarded; the script asserts its status. The script fails if a row is kept twice or if a source row is neither kept nor discarded.
  2. C1 uses runs/20261107T200233Z_C1_experiment_resume/C1_test_merged_results.jsonl. Every merged row equals (as parsed JSON) exactly one source row: 23 from 20261105T131443Z_C1_experiment and 7 from the resume.
  3. All five configurations have exactly 30 rows with the same 30 test ids. There is no provider_error, budget_stop, error, node_error, invalid_statement or lean_down row, and no degraded run. The plan's re-run rule and sensitivity analysis (§5) therefore do not apply. Area and difficulty in each row match data/eval.jsonl.
  4. All run_config.json files agree on profile, seed, split, Lean toolchain, Mathlib revision and model settings. git diff ad70ec2 2c8eae5 -- src config shows that only config/pipeline.yaml: spend.run_usd changed (the documented spend-cap deviations). The input sha256 of the resume runs match the Deviations entry. Anomaly: run_config.json records dirty: true for C5 resume 1, C5 resume 2, C4 and the C1 resume, with changed_files = [docs/analysis_plan.md] only. The Deviations entry calls these trees "clean". The only uncommitted file was the plan itself.
  5. The first C5 run and the first C1 run have run status running: they were killed, as documented. Their results files hold only finished problems.

Primary outcome and comparisons

  1. C2 in the SR table. Following §7.1, C2 has no verified SR. It enters through its claimed rate |B⁺|/30 = 30/30 and FPR_B. The 19 claims that the check verified are reported as a count only, because the plan defines no SR for them. C2 is not in any McNemar test (§3). In the per-cell and margin tables, C2 shows claimed counts, labelled as such, because §4.1 says "for each configuration".
  2. McNemar: b = first configuration solved and second not; c = the reverse; two-sided exact binomial on b | b + c with p = 1/2 (p = 1 if b + c = 0). Bonferroni-adjusted p = min(1, 3p); a test is significant if p < 0.05/3.
  3. Paired bootstrap (plan: 10 000 resamples of problems, seed 2026, percentile). The generator is a fresh numpy.random.default_rng(2026) for each statistic, so all comparisons use the same 10 000 index vectors. The plan does not name a generator. For H2 the lower bound is exactly 0.0 pp.
  4. Exploratory comparisons (C3 vs C4, C1 vs C3, C1 vs C4) get b, c, ΔSR and an uncorrected exact p only. The plan specifies no bootstrap CI for them.

FPR_B (Eq. 16, §4.2, Clarification 2026-11-03)

  1. B⁺ holds all 30 problems (every C2 row is claimed_solved). Check status: 19 verified, 11 not_verified. No check ended in budget_exhausted, provider_error, budget_stop, error or lean_down, so none is left out. FPR_B = 11/30.
  2. Manual-review sample. 11 claims are not verified, which is more than 10, so 10 are drawn "with seed 2026". The plan does not give the drawing procedure; random.Random(2026).sample(sorted(ids), 10) was used, which leaves out amc12a_2008_p15. The manual review itself (informal proof wrong vs. formalization failure) is a task for the team and has not been done.
  3. Factual cross-reference, not a conclusion: for 5 of the 11 not-verified claims (amc12a_2008_p15, amc12b_2002_p3, imo_1964_p1_2, imo_1965_p2, putnam_1965_b2), C5 has a kernel-accepted proof of the same formal statement.

Efficiency (§4.3)

  1. C(x) = llm.llm_calls of the row. It includes cache hits and JSON-repair calls, as the budget counts them. Tokens = prompt + completion (reasoning tokens are part of completion in the ledger). Nominal cost (cost_usd_nominal) is the compared figure (§1). Real cost is shown as well.
  2. Real cost. The real cost of every kept row equals the ledger sum for (run_id, problem_id); no mismatch was found. The ledger totals include spend that is in no kept row: C1 +0.09845 USD (imo_1987_p6 in the first C1 run: 10 calls, 5 paid and 5 cached JSON repairs), so C1 = 2.4695 + 0.0985 = 2.5679 USD. C5 +0.18153 USD (imo_1987_p6 in the first run: 0.02309 USD, 8 paid calls; imo_1984_p6 in resume 1: 0.15844 USD, 15 paid calls), so C5 = 1.1770 + 0.1815 = 1.3585 USD. C2 = 0.0814 (C2 itself) + 0.6269 (claim check) = 0.7083 USD. C3 = 1.3500 USD. C4 = 0.0000 USD. Per-problem and per-solved figures come from kept rows only.
  3. C2 efficiency leaves out the claim check: its calls, tokens and cost come from the row's llm, and its wall time runs from problem_start to claim_check_start. C̄_solved needs a "solved" count, and C2 has no verified one, so its ratio uses claimed solutions (30).
  4. C4: all 132 LLM calls were cache hits (requests identical to C5's), so its real cost is 0 and its LLM latency is near zero. Lean verification was real (5 of 259 lean_verify events were Lean-cache hits). C4's wall time (about 0.1 min per problem) therefore does not measure LLM computation. Nominal cost is the comparable figure (§1).
  5. Bootstrap CIs for the efficiency ratios: plan §4.3 gives no seed, so the same procedure as note 10 is used (seed 2026, percentile). For Σx/#solved, resamples with no solved problem would be dropped; none occurred on test.
  6. The median/IQR table (4d) and wall time by difficulty (Table 5) were requested by the team. They are descriptive only and not in the plan.

Refinement robustness (§4.4, Eq. 14)

  1. Accepted plan. The trace of a solved C5 problem is split at each plan_proposed / plan_fallback that follows subgoal work (attempt counters restart after a replan). The accepted plan is the last segment. "Attempts on a subgoal" is the attempts field of subgoal_solved; pre-pass subgoals count as 0. Read literally ("its accepted proof used at most k attempts on every subgoal"), a replanned problem counts by its accepted plan only. This affects algebra_absapbon1pabsapbleqsumabsaon1pabsa: C5 replanned once, then solved every subgoal on its first attempt, so it counts at k = 1. It is the only problem that differs between SR_C5(1) = 15/30 and SR_C4 = 14/30. SR_C5(3) = SR_C5 = 24/30, as expected.
  2. SR(0). Δ̂(1) = SR(1) − SR(0) needs SR(0), which the plan does not define. The same rule with k = 0 is used: the accepted plan has no LLM-attempted subgoal (pre-pass or automatic closing only). This gives 9/30.
  3. Independence curve. The plan says p_i is "estimated from first-attempt subgoal success in C5". Each subgoal has one first attempt, so a per-subgoal p_i would be 0 or 1. A pooled p̂ = 18/44 is used instead, counting every LLM-attempted subgoal of every plan (including abandoned plans and failed problems). The curve reported is Eq. 14's per-subgoal marginal gain Δ(k) = p̂(1 − p̂)^(k−1). It sits at a different level (subgoal) from the problem-level Δ̂(k). The last two columns of Table 6 give the subgoal-level empirical counterpart and are supplementary. The figure the plan asks for (§7.5) is docs/results/figures/srk_curve.pdf (and .png), drawn from the same numbers: (a) SR_C5(k), k = 0..3, with Wilson CIs and C4 at k = 1; (b) Δ̂(k) against p̂(1 − p̂)^(k−1).

Error taxonomy (§4.5)

  1. For C3–C5 the classes are the rows' error_classes, one per Critic event (checked equal to the critique events), including the k_max escalation to planning. Only kept rows count; the discarded attempts of the merges are excluded.
  2. C1 attempts are classified offline with policy.classify(Trace(reason, messages, None)), using the stored lean_errors (at most 3 messages of 300 characters, versus 5 × 1500 in the graph). The statement end is unknown, so misformalization cannot fire, which fits since C1 never writes statements. escalate is not applied because C1 has no k_max or strategies. The 44 sanitize:empty attempts (classed as syntax) happen alongside 52 C1 responses that ended at max_tokens.
  3. C2 has no verified attempts of its own. Its taxonomy row is the claim-check loop (labelled). The plan does not say how C2 enters §4.5.
  4. Final failure reasons group stages by prefix (verify:sg2 becomes verify).

Other data facts

  1. Responses that ended at max_tokens (finish_reason = length): C1 52 (9 rows), C3 18 (6 rows), C5 14 (5 rows), C4 2 (2 rows), C2 0 (12 in the claim check).
  2. C1 budget_exhausted rows (7) reached b_max = 18 calls after 9–17 recorded attempts, because JSON-repair calls count against the budget. The 3 failed C1 rows used all 18 attempts.
  3. Fidelity (§6). Four solved test problems carry a fidelity note: amc12b_2003_p6, mathd_algebra_234, mathd_algebra_455 and putnam_1965_b2. The team reviewed them on 2026-11-07 (post_test_review in data/curation/fidelity_review.yaml; Table 10): all four count as solved, amc12b_2003_p6 with a note in the report. The verified SR is reported unchanged, as §6 requires.
  4. Supplementary, not in the plan: Table 8 (pre-pass vs LLM solved_by) and Table 4d.

Illustrative traces (for the appendix)

  • amc12a_2008_p15 (Critic loop). C5 solved it on the third attempt at its only subgoal (two tactic_failure retries with Critic feedback, 16 calls, first C5 run). C4 abandoned after the first failure. C3 failed after a replan (30 calls). C1 failed. The C2 claim was not verified.
  • numbertheory_sumkmulnckeqnmul2pownm1 (RAG). C5 solved it with 5 calls on the first attempt. C3 failed after 4 missing_lemma and 2 tactic_failure attempts over two plans (30 calls).
  • imo_1965_p2 (FPR_B as an upper bound). C2 claimed it, and its check failed (missing_lemma, tactic_failure, missing_lemma). C5 found a kernel-accepted proof: the pre-pass closed one subgoal and the LLM closed the other two (one retry after missing_lemma).