Joe-Davis commited on
Commit
b465d4b
·
verified ·
1 Parent(s): a6b262a

Factual recall-vs-frontier framing; consistent 150-proof sample

Browse files
Files changed (1) hide show
  1. README.md +13 -6
README.md CHANGED
@@ -26,20 +26,27 @@ job is not to *generate* science but to **check** it.
26
 
27
  *By [MorphMind](https://morphmind.ai). Research preview.*
28
 
29
- ## Benchmark — matches GPT-5.4, beats Opus 4.8 on catch-rate
30
 
31
- ![CFM-Proof-3B leaderboard](leaderboard.png)
32
 
33
- On a frozen, by-paper held-out test of arXiv proofs — **every model given JSON output + an adequate token budget**, so the comparison is fair:
 
34
 
35
  | Model | Recall (errors caught) | Size |
36
  |---|---|---|
37
  | base Qwen2.5-3B (zero-shot) | 0.04 | 3B |
38
  | Claude Opus 4.8 | 0.61 | frontier |
39
  | GPT-5.4 | 0.84 | frontier |
40
- | **CFM-Proof-3B (ours)** | **0.83** | **3B · on-prem** |
41
-
42
- On the full test and an **entirely held-out domain it never trained on**, CFM-Proof-3B scores **recall 0.83 / 0.82** (localization 0.30 / 0.28), even across fields (cs.CC 0.87 · cs.IT 0.84 · cs.LG 0.84 · math.OC 0.84 · stat 0.80). It **matches GPT-5.4's catch-rate and exceeds Claude Opus 4.8 — at ~1/100 the size, on a single GPU.** (The models sit at different precision/recall trade-offs: Opus is more conservative — higher precision, lower recall; CFM and GPT-5.4 favor recall, the right bias for a screen that must not miss errors.)
 
 
 
 
 
 
43
 
44
  ## When & how to use it
45
  Use CFM-Proof-3B as a **fast first-pass reviewer** — to catch slips before a human deep-read, to
 
26
 
27
  *By [MorphMind](https://morphmind.ai). Research preview.*
28
 
29
+ ## Benchmark — proof-error recall vs. frontier models
30
 
31
+ ![CFM-Proof-3B proof-error recall](leaderboard.png)
32
 
33
+ Recall (share of injected proof errors caught) on the **same 150-proof held-out sample** — every model
34
+ given JSON output and an adequate token budget, so the comparison is like-for-like:
35
 
36
  | Model | Recall (errors caught) | Size |
37
  |---|---|---|
38
  | base Qwen2.5-3B (zero-shot) | 0.04 | 3B |
39
  | Claude Opus 4.8 | 0.61 | frontier |
40
  | GPT-5.4 | 0.84 | frontier |
41
+ | **CFM-Proof-3B (ours)** | **0.88** | **3B · on-prem** |
42
+
43
+ On this held-out sample CFM-Proof-3B is **competitive with frontier models on error catch-rate at roughly
44
+ 1/100 the size, running on a single GPU.** On the **full** 1,977-proof test and an **entirely held-out
45
+ domain**, its robust recall is **0.83 / 0.82** (localization 0.30 / 0.28), consistent across fields
46
+ (cs.CC 0.87 · cs.IT 0.84 · cs.LG 0.84 · math.OC 0.84 · stat 0.80). Read the table as a recall screen,
47
+ not a verdict on overall capability: the models sit at different precision/recall trade-offs — Opus is
48
+ more conservative (higher precision, lower recall), while CFM-Proof-3B and GPT-5.4 favor recall, the
49
+ right bias for a first-pass screen that must not miss errors.
50
 
51
  ## When & how to use it
52
  Use CFM-Proof-3B as a **fast first-pass reviewer** — to catch slips before a human deep-read, to