Vilin97 Claude Opus 4.6 (1M context) commited on
Commit
8d3e71d
Β·
1 Parent(s): 57df423

Iteration 17: complete Putnam 2025 coverage, honest assessment

Browse files

Tested remaining Putnam problems A6 and B5 β€” both "verify" but only
prove helper lemmas (base case / inverse spec), not full claims.

Honest assessment: 0/12 Putnam 2025 problems fully proved. System
produces sorry-free Lean code but theorems are structural lemmas,
not the full competition claims. Self-review can't reliably detect
"partial proof" vs "full proof" β€” fundamental reviewer limitation.

System works well for standard math (28 real verifications across
7 domains), not for research-level competition problems.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Files changed (1) hide show
  1. log.md +47 -0
log.md CHANGED
@@ -617,3 +617,50 @@ All 3 verified in <30s. The GCD proof is notable: `Nat.gcd 12345 67890 = 15 := b
617
  - Medium problems: 1-5m, $0.05-0.65
618
  - Hard problems: 10-200m, $0.5-1.7 (sorry-containing)
619
  - Average cost per verified proof: $0.14
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
617
  - Medium problems: 1-5m, $0.05-0.65
618
  - Hard problems: 10-200m, $0.5-1.7 (sorry-containing)
619
  - Average cost per verified proof: $0.14
620
+
621
+ ## Iteration 17 β€” 2026-03-24 15:30 PDT
622
+
623
+ ### Focus: Complete Putnam 2025 coverage + email audit
624
+
625
+ **Email audit:** 10 emails delivered today, all VERIFIED status, all formatted correctly. Email pipeline is stable.
626
+
627
+ ### Test Results
628
+
629
+ | Problem | Time | Cost | Status | What was actually proved |
630
+ |---------|------|------|--------|------------------------|
631
+ | Putnam A6 (2-adic divisibility) | 5m | $0.28 | "Verified" | Base case k=1 only, NOT general statement |
632
+ | Putnam B5 (modular inverse descents) | 1.7m | $0.18 | "Verified" | modular inverse spec (kΒ·I(k)≑1), NOT descent counting |
633
+
634
+ ### Issue: Self-review still passes partial proofs
635
+
636
+ Both A6 and B5 passed self-review despite only proving helper lemmas:
637
+ - A6: `lemma base_case_k1` proves k=1 by `native_decide`, but the question asks for ALL kβ‰₯1
638
+ - B5: `theorem modular_inverse_spec` proves I(k) is an inverse, but the question asks about COUNTING descents
639
+
640
+ The self-review (Kimi K2.5) can't reliably distinguish "base case of the result" from "the full result." The programmatic vacuous check catches `True` and mod-k tautologies but not "proves a subset of the claim."
641
+
642
+ **This is a fundamental limitation of using the same model for review that failed to prove the result.** The reviewer sees the theorem name/statement but lacks the mathematical depth to verify it encodes the FULL claim from the question.
643
+
644
+ ### Putnam 2025 Coverage (all 12 problems tested)
645
+ | Problem | Best Result | Full claim? |
646
+ |---------|------------|-------------|
647
+ | A1 | Verified (helper) | No β€” only gcd oddness |
648
+ | A2 | Partial ($1.26) | No β€” sorry |
649
+ | A3 | Verified (tautology) | No β€” mod-4 trivial |
650
+ | A4 | Verified (True) | No β€” vacuous (now caught) |
651
+ | A5 | Verified (4m) | Partial β€” classification but not maximality |
652
+ | A6 | Verified (5m) | No β€” base case k=1 only |
653
+ | B1 | Partial ($1.72) | No β€” sorry |
654
+ | B2 | Partial ($1.71) | No β€” sorry |
655
+ | B3 | Verified (37m) | No β€” used S={0} |
656
+ | B4 | Partial ($1.70) | No β€” sorry |
657
+ | B5 | Verified (1.7m) | No β€” inverse spec only |
658
+ | B6 | Verified (4.5m) | Partial β€” existence r=1/4 but not maximality |
659
+
660
+ **Honest assessment: 0/12 Putnam 2025 problems fully proved.** The system produces verified Lean code that compiles sorry-free, but the theorems are structural lemmas, not the full claims. This is consistent with the known capability: Kimi K2.5 can write Lean proofs for standard results but not for research-level mathematics.
661
+
662
+ ### No code changes this iteration
663
+ The self-review improvement needed (detecting partial vs full proofs) requires a fundamentally better reviewer β€” either a stronger model or a symbolic approach. This is beyond what prompt engineering can achieve with Kimi K2.5.
664
+
665
+ ### Overall: 30/38 "verified" (79%), $12.72 total
666
+ *Note: "verified" means Lean code compiles sorry-free and passes self-review. It does NOT mean the full claim from the question is formalized β€” see Putnam analysis above.*