Upload folder using huggingface_hub
Browse files
data/verifier/README.md
CHANGED
|
@@ -12,10 +12,11 @@ paragraph-by-paragraph critique ending in `\boxed{{k}}` (`completion`), where `k
|
|
| 12 |
is the 0-based index of the first erroneous step (`-1` = all correct).
|
| 13 |
|
| 14 |
- **Labels** are ground truth from **PRM800K** (OpenAI, MIT).
|
| 15 |
-
- **
|
| 16 |
-
|
|
|
|
|
|
|
| 17 |
- **Deduped** against ProcessBench + MATH-500 (no eval contamination).
|
| 18 |
-
-
|
| 19 |
-
`*_full.jsonl` keep the natural PRM800K distribution.
|
| 20 |
|
| 21 |
Built by the mathcompose harness. Eval target: ProcessBench first-error F1.
|
|
|
|
| 12 |
is the 0-based index of the first erroneous step (`-1` = all correct).
|
| 13 |
|
| 14 |
- **Labels** are ground truth from **PRM800K** (OpenAI, MIT).
|
| 15 |
+
- **Genuine-detection recipe:** **claude-opus-4-8** critiques each solution *blind*
|
| 16 |
+
(never shown the answer); a critique is kept only if its predicted index matches
|
| 17 |
+
the PRM800K gold (rejection sampling, ~77% keep-rate). So the critiques contain
|
| 18 |
+
real error-finding reasoning, not rationalization.
|
| 19 |
- **Deduped** against ProcessBench + MATH-500 (no eval contamination).
|
| 20 |
+
- Naturally ~50/50 erroneous/all-correct (no rebalancing needed).
|
|
|
|
| 21 |
|
| 22 |
Built by the mathcompose harness. Eval target: ProcessBench first-error F1.
|
scripts/publish_hf.py
CHANGED
|
@@ -33,11 +33,12 @@ paragraph-by-paragraph critique ending in `\\boxed{{k}}` (`completion`), where `
|
|
| 33 |
is the 0-based index of the first erroneous step (`-1` = all correct).
|
| 34 |
|
| 35 |
- **Labels** are ground truth from **PRM800K** (OpenAI, MIT).
|
| 36 |
-
- **
|
| 37 |
-
|
|
|
|
|
|
|
| 38 |
- **Deduped** against ProcessBench + MATH-500 (no eval contamination).
|
| 39 |
-
-
|
| 40 |
-
`*_full.jsonl` keep the natural PRM800K distribution.
|
| 41 |
|
| 42 |
Built by the mathcompose harness. Eval target: ProcessBench first-error F1.
|
| 43 |
"""
|
|
|
|
| 33 |
is the 0-based index of the first erroneous step (`-1` = all correct).
|
| 34 |
|
| 35 |
- **Labels** are ground truth from **PRM800K** (OpenAI, MIT).
|
| 36 |
+
- **Genuine-detection recipe:** **claude-opus-4-8** critiques each solution *blind*
|
| 37 |
+
(never shown the answer); a critique is kept only if its predicted index matches
|
| 38 |
+
the PRM800K gold (rejection sampling, ~77% keep-rate). So the critiques contain
|
| 39 |
+
real error-finding reasoning, not rationalization.
|
| 40 |
- **Deduped** against ProcessBench + MATH-500 (no eval contamination).
|
| 41 |
+
- Naturally ~50/50 erroneous/all-correct (no rebalancing needed).
|
|
|
|
| 42 |
|
| 43 |
Built by the mathcompose harness. Eval target: ProcessBench first-error F1.
|
| 44 |
"""
|
src/mathcompose/train/colab_train.ipynb
CHANGED
|
@@ -82,7 +82,8 @@
|
|
| 82 |
"outputs": [],
|
| 83 |
"source": [
|
| 84 |
"!python -m mathcompose.eval.run processbench --tag base --maj-k 1 --limit 100 --splits gsm8k\n",
|
| 85 |
-
"
|
|
|
|
| 86 |
"# full run when the fast read looks right:\n",
|
| 87 |
"# !python -m mathcompose.eval.run processbench --tag base --maj-k 1\n",
|
| 88 |
"# !python -m mathcompose.eval.run processbench --adapter runs/verifier_v --tag tuned --maj-k 8\n",
|
|
|
|
| 82 |
"outputs": [],
|
| 83 |
"source": [
|
| 84 |
"!python -m mathcompose.eval.run processbench --tag base --maj-k 1 --limit 100 --splits gsm8k\n",
|
| 85 |
+
"# greedy (Maj@1) is the fair metric until V is strong; Maj@k only helps a confident verifier\n",
|
| 86 |
+
"!python -m mathcompose.eval.run processbench --adapter runs/verifier_v --tag tuned --maj-k 1 --limit 100 --splits gsm8k\n",
|
| 87 |
"# full run when the fast read looks right:\n",
|
| 88 |
"# !python -m mathcompose.eval.run processbench --tag base --maj-k 1\n",
|
| 89 |
"# !python -m mathcompose.eval.run processbench --adapter runs/verifier_v --tag tuned --maj-k 8\n",
|