File size: 1,427 Bytes
7c85c7e | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 | # Reproduce Pivot's public results
Run the standalone `jevbench_public.py` script from this repository. It checks out the official JevBench v1.4.1 source at commit `24b9b5c1609a7a9e8fa14f49e5985a836c9dc842`, verifies a clean checkout, and uses its public tasks and official per-task scorer.
Install Python, Git, and a PyTorch build for your hardware. The requirements file records the versions used for the reported run. On Windows PowerShell:
```powershell
py -m pip install -r benchmarks/requirements.txt
py benchmarks/jevbench_public.py --device cpu --threads 4 --batch-size 4 --output pivot_cpu_results.json
```
On a CUDA machine use `--device cuda --batch-size 32`. If you only need timings, add `--mode speed`. For public task accuracy only, add `--mode accuracy`. The model is pinned to commit `14bf8c26bf344ebdf88e22a4b6152dc5f75f3578` unless you deliberately provide another full SHA via `--revision`. Allow time and disk space for the approximately 1.43 GB checkpoint.
The script uses the same candidate wording, input rendering, token limits, FP32 scoring, warmup count, and timing protocol as the published run. Timings still vary with hardware, software and background load. Only public JevBench tasks are available here; `official_jevbench_score` is always null. The full official score requires organizer-held sealed and judge tasks and cost measurement. See [the benchmark protocol](../docs/BENCHMARK.md).
|