42e commited on
Commit
74b39bc
·
verified ·
1 Parent(s): 81fee4a

Upload folder using huggingface_hub

Browse files
data/verifier/README.md CHANGED
@@ -12,10 +12,11 @@ paragraph-by-paragraph critique ending in `\boxed{{k}}` (`completion`), where `k
12
  is the 0-based index of the first erroneous step (`-1` = all correct).
13
 
14
  - **Labels** are ground truth from **PRM800K** (OpenAI, MIT).
15
- - **Critiques** distilled from **claude-opus-4-8** (rationalization conditioned on
16
- the known label; the boxed index is stamped authoritatively, not trusted to the teacher).
 
 
17
  - **Deduped** against ProcessBench + MATH-500 (no eval contamination).
18
- - `train.jsonl`/`val.jsonl` are class-balanced (~50/50 erroneous/all-correct);
19
- `*_full.jsonl` keep the natural PRM800K distribution.
20
 
21
  Built by the mathcompose harness. Eval target: ProcessBench first-error F1.
 
12
  is the 0-based index of the first erroneous step (`-1` = all correct).
13
 
14
  - **Labels** are ground truth from **PRM800K** (OpenAI, MIT).
15
+ - **Genuine-detection recipe:** **claude-opus-4-8** critiques each solution *blind*
16
+ (never shown the answer); a critique is kept only if its predicted index matches
17
+ the PRM800K gold (rejection sampling, ~77% keep-rate). So the critiques contain
18
+ real error-finding reasoning, not rationalization.
19
  - **Deduped** against ProcessBench + MATH-500 (no eval contamination).
20
+ - Naturally ~50/50 erroneous/all-correct (no rebalancing needed).
 
21
 
22
  Built by the mathcompose harness. Eval target: ProcessBench first-error F1.
scripts/publish_hf.py CHANGED
@@ -33,11 +33,12 @@ paragraph-by-paragraph critique ending in `\\boxed{{k}}` (`completion`), where `
33
  is the 0-based index of the first erroneous step (`-1` = all correct).
34
 
35
  - **Labels** are ground truth from **PRM800K** (OpenAI, MIT).
36
- - **Critiques** distilled from **claude-opus-4-8** (rationalization conditioned on
37
- the known label; the boxed index is stamped authoritatively, not trusted to the teacher).
 
 
38
  - **Deduped** against ProcessBench + MATH-500 (no eval contamination).
39
- - `train.jsonl`/`val.jsonl` are class-balanced (~50/50 erroneous/all-correct);
40
- `*_full.jsonl` keep the natural PRM800K distribution.
41
 
42
  Built by the mathcompose harness. Eval target: ProcessBench first-error F1.
43
  """
 
33
  is the 0-based index of the first erroneous step (`-1` = all correct).
34
 
35
  - **Labels** are ground truth from **PRM800K** (OpenAI, MIT).
36
+ - **Genuine-detection recipe:** **claude-opus-4-8** critiques each solution *blind*
37
+ (never shown the answer); a critique is kept only if its predicted index matches
38
+ the PRM800K gold (rejection sampling, ~77% keep-rate). So the critiques contain
39
+ real error-finding reasoning, not rationalization.
40
  - **Deduped** against ProcessBench + MATH-500 (no eval contamination).
41
+ - Naturally ~50/50 erroneous/all-correct (no rebalancing needed).
 
42
 
43
  Built by the mathcompose harness. Eval target: ProcessBench first-error F1.
44
  """
src/mathcompose/train/colab_train.ipynb CHANGED
@@ -82,7 +82,8 @@
82
  "outputs": [],
83
  "source": [
84
  "!python -m mathcompose.eval.run processbench --tag base --maj-k 1 --limit 100 --splits gsm8k\n",
85
- "!python -m mathcompose.eval.run processbench --adapter runs/verifier_v --tag tuned --maj-k 4 --limit 100 --splits gsm8k\n",
 
86
  "# full run when the fast read looks right:\n",
87
  "# !python -m mathcompose.eval.run processbench --tag base --maj-k 1\n",
88
  "# !python -m mathcompose.eval.run processbench --adapter runs/verifier_v --tag tuned --maj-k 8\n",
 
82
  "outputs": [],
83
  "source": [
84
  "!python -m mathcompose.eval.run processbench --tag base --maj-k 1 --limit 100 --splits gsm8k\n",
85
+ "# greedy (Maj@1) is the fair metric until V is strong; Maj@k only helps a confident verifier\n",
86
+ "!python -m mathcompose.eval.run processbench --adapter runs/verifier_v --tag tuned --maj-k 1 --limit 100 --splits gsm8k\n",
87
  "# full run when the fast read looks right:\n",
88
  "# !python -m mathcompose.eval.run processbench --tag base --maj-k 1\n",
89
  "# !python -m mathcompose.eval.run processbench --adapter runs/verifier_v --tag tuned --maj-k 8\n",