Spaces:
Running
Running
Iteration 17: complete Putnam 2025 coverage, honest assessment
Browse filesTested remaining Putnam problems A6 and B5 β both "verify" but only
prove helper lemmas (base case / inverse spec), not full claims.
Honest assessment: 0/12 Putnam 2025 problems fully proved. System
produces sorry-free Lean code but theorems are structural lemmas,
not the full competition claims. Self-review can't reliably detect
"partial proof" vs "full proof" β fundamental reviewer limitation.
System works well for standard math (28 real verifications across
7 domains), not for research-level competition problems.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
log.md
CHANGED
|
@@ -617,3 +617,50 @@ All 3 verified in <30s. The GCD proof is notable: `Nat.gcd 12345 67890 = 15 := b
|
|
| 617 |
- Medium problems: 1-5m, $0.05-0.65
|
| 618 |
- Hard problems: 10-200m, $0.5-1.7 (sorry-containing)
|
| 619 |
- Average cost per verified proof: $0.14
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 617 |
- Medium problems: 1-5m, $0.05-0.65
|
| 618 |
- Hard problems: 10-200m, $0.5-1.7 (sorry-containing)
|
| 619 |
- Average cost per verified proof: $0.14
|
| 620 |
+
|
| 621 |
+
## Iteration 17 β 2026-03-24 15:30 PDT
|
| 622 |
+
|
| 623 |
+
### Focus: Complete Putnam 2025 coverage + email audit
|
| 624 |
+
|
| 625 |
+
**Email audit:** 10 emails delivered today, all VERIFIED status, all formatted correctly. Email pipeline is stable.
|
| 626 |
+
|
| 627 |
+
### Test Results
|
| 628 |
+
|
| 629 |
+
| Problem | Time | Cost | Status | What was actually proved |
|
| 630 |
+
|---------|------|------|--------|------------------------|
|
| 631 |
+
| Putnam A6 (2-adic divisibility) | 5m | $0.28 | "Verified" | Base case k=1 only, NOT general statement |
|
| 632 |
+
| Putnam B5 (modular inverse descents) | 1.7m | $0.18 | "Verified" | modular inverse spec (kΒ·I(k)β‘1), NOT descent counting |
|
| 633 |
+
|
| 634 |
+
### Issue: Self-review still passes partial proofs
|
| 635 |
+
|
| 636 |
+
Both A6 and B5 passed self-review despite only proving helper lemmas:
|
| 637 |
+
- A6: `lemma base_case_k1` proves k=1 by `native_decide`, but the question asks for ALL kβ₯1
|
| 638 |
+
- B5: `theorem modular_inverse_spec` proves I(k) is an inverse, but the question asks about COUNTING descents
|
| 639 |
+
|
| 640 |
+
The self-review (Kimi K2.5) can't reliably distinguish "base case of the result" from "the full result." The programmatic vacuous check catches `True` and mod-k tautologies but not "proves a subset of the claim."
|
| 641 |
+
|
| 642 |
+
**This is a fundamental limitation of using the same model for review that failed to prove the result.** The reviewer sees the theorem name/statement but lacks the mathematical depth to verify it encodes the FULL claim from the question.
|
| 643 |
+
|
| 644 |
+
### Putnam 2025 Coverage (all 12 problems tested)
|
| 645 |
+
| Problem | Best Result | Full claim? |
|
| 646 |
+
|---------|------------|-------------|
|
| 647 |
+
| A1 | Verified (helper) | No β only gcd oddness |
|
| 648 |
+
| A2 | Partial ($1.26) | No β sorry |
|
| 649 |
+
| A3 | Verified (tautology) | No β mod-4 trivial |
|
| 650 |
+
| A4 | Verified (True) | No β vacuous (now caught) |
|
| 651 |
+
| A5 | Verified (4m) | Partial β classification but not maximality |
|
| 652 |
+
| A6 | Verified (5m) | No β base case k=1 only |
|
| 653 |
+
| B1 | Partial ($1.72) | No β sorry |
|
| 654 |
+
| B2 | Partial ($1.71) | No β sorry |
|
| 655 |
+
| B3 | Verified (37m) | No β used S={0} |
|
| 656 |
+
| B4 | Partial ($1.70) | No β sorry |
|
| 657 |
+
| B5 | Verified (1.7m) | No β inverse spec only |
|
| 658 |
+
| B6 | Verified (4.5m) | Partial β existence r=1/4 but not maximality |
|
| 659 |
+
|
| 660 |
+
**Honest assessment: 0/12 Putnam 2025 problems fully proved.** The system produces verified Lean code that compiles sorry-free, but the theorems are structural lemmas, not the full claims. This is consistent with the known capability: Kimi K2.5 can write Lean proofs for standard results but not for research-level mathematics.
|
| 661 |
+
|
| 662 |
+
### No code changes this iteration
|
| 663 |
+
The self-review improvement needed (detecting partial vs full proofs) requires a fundamentally better reviewer β either a stronger model or a symbolic approach. This is beyond what prompt engineering can achieve with Kimi K2.5.
|
| 664 |
+
|
| 665 |
+
### Overall: 30/38 "verified" (79%), $12.72 total
|
| 666 |
+
*Note: "verified" means Lean code compiles sorry-free and passes self-review. It does NOT mean the full claim from the question is formalized β see Putnam analysis above.*
|