|
Download docs/results/method_notes.md from NicolasT08/formal-mas: direct link, hf CLI and curl.
- Browser
- Download file 12.6 kB
-
https://huggingface.co/NicolasT08/formal-mas/resolve/main/docs/results/method_notes.md
- Command line
-
hf download hf://NicolasT08/formal-mas/docs/results/method_notes.md
-
curl -L -o method_notes.md https://huggingface.co/NicolasT08/formal-mas/resolve/main/docs/results/method_notes.md
12.6 kB
Method notes (interpretations of docs/analysis_plan.md and data anomalies)
Hand-written; appended verbatim to results.md by make_tables.py. Each item is a point where the plan
had to be read literally or was silent, or a fact found in the data.
Analysis code
- Analysis code. Plan §7 places the statistics in
src/formal_mas/analysis/stats.py. The first version of this analysis implemented them insidedocs/results/make_tables.py(the file did not exist yet); on 2026-11-07, by team decision, they were moved tosrc/formal_mas/analysis/stats.py(Wilson, exact McNemar, Bonferroni, bootstrap, quartiles), with offline tests against closed-form values intests/test_stats.py.make_tables.pyimports them, and the regeneratedresults.mdandtables.texwere checked to be byte for byte identical to the earlier output (before Table 10 was switched to the post-test fidelity review). Exact binomial tests and Wilson intervals come fromscipy.stats.binomtest(Wilson without continuity correction). The bootstrap usesnumpy.random.default_rng. The κ classifier is imported read-only fromsrc/formal_mas/orchestrator/policy.py.scripts/compare.pywas not written:make_tables.pyproduces every table of §7. - Dev before test (plan §7). The script was first run in
--devmode on dev runs, using stand-ins on the same 15 dev problems, and checked there: Wilson 11/15 = [48.0, 89.1], exact McNemar p values computed by hand, and the C1 κ classification on20261103T161957Z_C1_experiment_c1_cost. Unit tests of the statistics were added later (item 1). Disclosure: before the script existed, the analyst had seen the firsttestrow of C5 (mathd_algebra_206, solved) and the summaries stored in therun_config.jsonof the C5 resume runs and of C4, while checking the configurations. After the firsttestrun, the script changed in three ways: (a) C2 wall time now stops atclaim_check_start(the row'swall_sincludes the claim check, and the 2026-11-03 Clarification excludes the check from C2's figures); (b) a note on responses cut atmax_tokenswas added; (c) LaTeX layout. No test, comparison, correction or metric of the plan was changed.
Merges and integrity
- C5 merge follows the C5 Deviations entry exactly. It keeps 23 rows of
20261103T174646Z_C5_experiment,imo_1987_p6andimo_1979_p1of..._resume, and 5 rows of..._resume2. Thebudget_stoprow ofimo_1984_p6in resume 1 is discarded; the script asserts its status. The script fails if a row is kept twice or if a source row is neither kept nor discarded. - C1 uses
runs/20261107T200233Z_C1_experiment_resume/C1_test_merged_results.jsonl. Every merged row equals (as parsed JSON) exactly one source row: 23 from20261105T131443Z_C1_experimentand 7 from the resume. - All five configurations have exactly 30 rows with the same 30
testids. There is noprovider_error,budget_stop,error,node_error,invalid_statementorlean_downrow, and no degraded run. The plan's re-run rule and sensitivity analysis (§5) therefore do not apply. Area and difficulty in each row matchdata/eval.jsonl. - All
run_config.jsonfiles agree on profile, seed, split, Lean toolchain, Mathlib revision and model settings.git diff ad70ec2 2c8eae5 -- src configshows that onlyconfig/pipeline.yaml: spend.run_usdchanged (the documented spend-cap deviations). The input sha256 of the resume runs match the Deviations entry. Anomaly:run_config.jsonrecordsdirty: truefor C5 resume 1, C5 resume 2, C4 and the C1 resume, withchanged_files = [docs/analysis_plan.md]only. The Deviations entry calls these trees "clean". The only uncommitted file was the plan itself. - The first C5 run and the first C1 run have run status
running: they were killed, as documented. Their results files hold only finished problems.
Primary outcome and comparisons
- C2 in the SR table. Following §7.1, C2 has no verified SR. It enters through its claimed rate |B⁺|/30 = 30/30 and FPR_B. The 19 claims that the check verified are reported as a count only, because the plan defines no SR for them. C2 is not in any McNemar test (§3). In the per-cell and margin tables, C2 shows claimed counts, labelled as such, because §4.1 says "for each configuration".
- McNemar: b = first configuration solved and second not; c = the reverse; two-sided exact binomial on b | b + c with p = 1/2 (p = 1 if b + c = 0). Bonferroni-adjusted p = min(1, 3p); a test is significant if p < 0.05/3.
- Paired bootstrap (plan: 10 000 resamples of problems, seed 2026, percentile). The generator is a fresh
numpy.random.default_rng(2026)for each statistic, so all comparisons use the same 10 000 index vectors. The plan does not name a generator. For H2 the lower bound is exactly 0.0 pp. - Exploratory comparisons (C3 vs C4, C1 vs C3, C1 vs C4) get b, c, ΔSR and an uncorrected exact p only. The plan specifies no bootstrap CI for them.
FPR_B (Eq. 16, §4.2, Clarification 2026-11-03)
- B⁺ holds all 30 problems (every C2 row is
claimed_solved). Check status: 19verified, 11not_verified. No check ended inbudget_exhausted,provider_error,budget_stop,errororlean_down, so none is left out. FPR_B = 11/30. - Manual-review sample. 11 claims are not verified, which is more than 10, so 10 are drawn "with seed 2026".
The plan does not give the drawing procedure;
random.Random(2026).sample(sorted(ids), 10)was used, which leaves outamc12a_2008_p15. The manual review itself (informal proof wrong vs. formalization failure) is a task for the team and has not been done. - Factual cross-reference, not a conclusion: for 5 of the 11 not-verified claims (
amc12a_2008_p15,amc12b_2002_p3,imo_1964_p1_2,imo_1965_p2,putnam_1965_b2), C5 has a kernel-accepted proof of the same formal statement.
Efficiency (§4.3)
- C(x) =
llm.llm_callsof the row. It includes cache hits and JSON-repair calls, as the budget counts them. Tokens = prompt + completion (reasoning tokens are part of completion in the ledger). Nominal cost (cost_usd_nominal) is the compared figure (§1). Real cost is shown as well. - Real cost. The real cost of every kept row equals the ledger sum for (run_id, problem_id); no mismatch was
found. The ledger totals include spend that is in no kept row:
C1 +0.09845 USD (
imo_1987_p6in the first C1 run: 10 calls, 5 paid and 5 cached JSON repairs), so C1 = 2.4695 + 0.0985 = 2.5679 USD. C5 +0.18153 USD (imo_1987_p6in the first run: 0.02309 USD, 8 paid calls;imo_1984_p6in resume 1: 0.15844 USD, 15 paid calls), so C5 = 1.1770 + 0.1815 = 1.3585 USD. C2 = 0.0814 (C2 itself) + 0.6269 (claim check) = 0.7083 USD. C3 = 1.3500 USD. C4 = 0.0000 USD. Per-problem and per-solved figures come from kept rows only. - C2 efficiency leaves out the claim check: its calls, tokens and cost come from the row's
llm, and its wall time runs fromproblem_starttoclaim_check_start. C̄_solved needs a "solved" count, and C2 has no verified one, so its ratio uses claimed solutions (30). - C4: all 132 LLM calls were cache hits (requests identical to C5's), so its real cost is 0 and its LLM
latency is near zero. Lean verification was real (5 of 259
lean_verifyevents were Lean-cache hits). C4's wall time (about 0.1 min per problem) therefore does not measure LLM computation. Nominal cost is the comparable figure (§1). - Bootstrap CIs for the efficiency ratios: plan §4.3 gives no seed, so the same procedure as note 10 is used
(seed 2026, percentile). For Σx/#solved, resamples with no solved problem would be dropped; none occurred on
test. - The median/IQR table (4d) and wall time by difficulty (Table 5) were requested by the team. They are descriptive only and not in the plan.
Refinement robustness (§4.4, Eq. 14)
- Accepted plan. The trace of a solved C5 problem is split at each
plan_proposed/plan_fallbackthat follows subgoal work (attempt counters restart after a replan). The accepted plan is the last segment. "Attempts on a subgoal" is theattemptsfield ofsubgoal_solved; pre-pass subgoals count as 0. Read literally ("its accepted proof used at most k attempts on every subgoal"), a replanned problem counts by its accepted plan only. This affectsalgebra_absapbon1pabsapbleqsumabsaon1pabsa: C5 replanned once, then solved every subgoal on its first attempt, so it counts at k = 1. It is the only problem that differs between SR_C5(1) = 15/30 and SR_C4 = 14/30. SR_C5(3) = SR_C5 = 24/30, as expected. - SR(0). Δ̂(1) = SR(1) − SR(0) needs SR(0), which the plan does not define. The same rule with k = 0 is used: the accepted plan has no LLM-attempted subgoal (pre-pass or automatic closing only). This gives 9/30.
- Independence curve. The plan says p_i is "estimated from first-attempt subgoal success in C5". Each
subgoal has one first attempt, so a per-subgoal p_i would be 0 or 1. A pooled p̂ = 18/44 is used instead,
counting every LLM-attempted subgoal of every plan (including abandoned plans and failed problems). The curve
reported is Eq. 14's per-subgoal marginal gain Δ(k) = p̂(1 − p̂)^(k−1). It sits at a different level
(subgoal) from the problem-level Δ̂(k). The last two columns of Table 6 give the subgoal-level empirical
counterpart and are supplementary. The figure the plan asks for (§7.5) is
docs/results/figures/srk_curve.pdf(and.png), drawn from the same numbers: (a) SR_C5(k), k = 0..3, with Wilson CIs and C4 at k = 1; (b) Δ̂(k) against p̂(1 − p̂)^(k−1).
Error taxonomy (§4.5)
- For C3–C5 the classes are the rows'
error_classes, one per Critic event (checked equal to thecritiqueevents), including the k_max escalation toplanning. Only kept rows count; the discarded attempts of the merges are excluded. - C1 attempts are classified offline with
policy.classify(Trace(reason, messages, None)), using the storedlean_errors(at most 3 messages of 300 characters, versus 5 × 1500 in the graph). The statement end is unknown, somisformalizationcannot fire, which fits since C1 never writes statements.escalateis not applied because C1 has no k_max or strategies. The 44sanitize:emptyattempts (classed assyntax) happen alongside 52 C1 responses that ended atmax_tokens. - C2 has no verified attempts of its own. Its taxonomy row is the claim-check loop (labelled). The plan does not say how C2 enters §4.5.
- Final failure reasons group stages by prefix (
verify:sg2becomesverify).
Other data facts
- Responses that ended at
max_tokens(finish_reason = length): C1 52 (9 rows), C3 18 (6 rows), C5 14 (5 rows), C4 2 (2 rows), C2 0 (12 in the claim check). - C1
budget_exhaustedrows (7) reached b_max = 18 calls after 9–17 recorded attempts, because JSON-repair calls count against the budget. The 3failedC1 rows used all 18 attempts. - Fidelity (§6). Four solved test problems carry a fidelity note:
amc12b_2003_p6,mathd_algebra_234,mathd_algebra_455andputnam_1965_b2. The team reviewed them on 2026-11-07 (post_test_reviewindata/curation/fidelity_review.yaml; Table 10): all four count as solved,amc12b_2003_p6with a note in the report. The verified SR is reported unchanged, as §6 requires. - Supplementary, not in the plan: Table 8 (pre-pass vs LLM
solved_by) and Table 4d.
Illustrative traces (for the appendix)
amc12a_2008_p15(Critic loop). C5 solved it on the third attempt at its only subgoal (twotactic_failureretries with Critic feedback, 16 calls, first C5 run). C4 abandoned after the first failure. C3 failed after a replan (30 calls). C1 failed. The C2 claim was not verified.numbertheory_sumkmulnckeqnmul2pownm1(RAG). C5 solved it with 5 calls on the first attempt. C3 failed after 4missing_lemmaand 2tactic_failureattempts over two plans (30 calls).imo_1965_p2(FPR_B as an upper bound). C2 claimed it, and its check failed (missing_lemma,tactic_failure,missing_lemma). C5 found a kernel-accepted proof: the pre-pass closed one subgoal and the LLM closed the other two (one retry aftermissing_lemma).