gpu-sft/EVAL_RESULTS.md

# GPU eval results (PARTIAL) — captured 2026-09-13 before a machine reorg

Raw per-task JSONs live on HDFS at `/mnt/hdfs/fangzhao_writable/marin-eval/results`
(durable across the reorg). This file is the accessible-anywhere (GitHub) copy of the
headline numbers so they're not gated behind HDFS access.

**Eval stack:** driver 535, cu124 **vLLM 0.8.5 + evalchemy@7f24168**, `gpu_eval_driver.py`,
tp=8, temp 0.7 / top_p 1.0 / max_gen 32768, seeds 42..51 (AIME*/AMC23/HMMT n=10; MATH500 &
OlympiadBench n=1; LiveCodeBench* n=6). Per the port's design, GPU numbers are **not
bit-comparable** to the TPU tables (different kernels/seeding) — deltas are computed vs the
`qwen3_8b_base` row below.

## qwen3_8b_base — baseline (the (+Δ) denominator)

### math
| task | mean | std | seeds |
|---|---|---|---|
| MATH500 | 0.9080 | – | 1 |
| OlympiadBench | 0.6795 | – | 1 |
| AIME24 | 0.7500 | 0.0478 | 10 |
| AIME25 | 0.6467 | 0.0358 | 10 |
| AIME26 | 0.6833 | 0.0572 | 10 |
| AMC23 | 0.9425 | 0.0290 | 10 |
| HMMT | **FAILED** (exited 1 @ 11s — bug, see `_raw/HMMT.evalchemy.log`) | – | – |

### code
| task | mean | std | seeds |
|---|---|---|---|
| LiveCodeBench | 0.6181 | 0.0230 | 6 |
| LiveCodeBenchv5_official | not finished (killed mid-run) | – | – |
| LiveCodeBenchv6_official | pending | – | – |

## Gate 2 (eval parity) — PASS
GPU AIME24 **0.750 [±0.048]** vs TPU reference **0.740 [±0.049]** → within ±1σ. GPU baseline
runs slightly higher than the TPU reference across the board (MATH500 90.8 vs 89.2,
OlympiadBench 68.0 vs 62.5, AIME25 64.7 vs 60.0, AIME26 68.3 vs 62.0) — a consistent, small
upward shift consistent with the known kernel/seeding differences (not bit-comparable, as
designed). The eval stack is validated.

## Still pending (resume on the next machine)
- **baseline:** HMMT (fix the quick failure), LiveCodeBench v5/v6.
- **experiments (needed for the deltas):** `hs_competition_nofilter` (math),
  `coding_depth4v3_nofilter` (code). A last-hour salvage of hs AIME24 may have banked —
  append here if it did.
- then `claude/compile_results_local.py` over the HDFS results → `results.md` with `(+Δ)`.

## Trained models (durable on HDFS, final step-1999)
- `/mnt/hdfs/fangzhao_writable/marin-sft/coding_depth4v3_nofilter/hf/step-1999` — DONE (2000 steps)
- `/mnt/hdfs/fangzhao_writable/marin-sft/hs_competition_nofilter/hf/step-1999` — DONE (2000 steps)
- `science_depth4v3_nofilter` — re-running fresh (`_v3`) on a separate machine