jdi-ptr14-0.8b / engine /VALIDATION.md
AnamK's picture
Upload jdi-ptr14-0.8b (Qwen3.5-0.8B, 14 layers + pointer head)
5bb55cb verified
|
Raw History Blame Contribute Delete
2.5 kB
# Validation: jdi_ptr14 engine vs. the evaluator behind the submitted run
Date: 2026-10-18. GPU: RTX 3090 (shared), `CUDA_VISIBLE_DEVICES=0`, `mem_gb=4.5`. torch 2.9.1+cu128, transformers 5.8.0.
**Rows:** 200 suite rows from the public 0.3 suite, stratified over all 43 in-edition benchmarks with a fixed seed
(`work/make_val200.py`).
- 1,091 questions in total.
- 47 rows are multi-question and take the shared-prefix path.
- 40 questions have more than 9 options.
**Three result sets:**
1. `engine`: `python -m decision_index run --engine jdi_ptr14.engine:Ptr14Engine --option model=<ckpt> --option mem_gb=4.5 --rows val200_rows.jsonl`, i.e. the kit runner, which validates every response with `decision_index.engines.base.validate`.
2. `e2e_fresh`: a fresh `e2e_prefix.py eval <ckpt> <payloads> --layer 14 --batch-tokens 8192 --mem-gb 4.5` on the same rows.
3. `submitted`: the same rows taken from `runs/ptr14nosyn-soup3/results.jsonl.gz`.
| Pair | Questions | Argmax agreement | Max abs prob diff |
|---|---|---|---|
| engine vs e2e_fresh | 1,091 | 1,091 (100%) | 0.0 |
| engine vs submitted | 1,091 | 1,091 (100%) | 0.0 |
| e2e_fresh vs submitted | 1,091 | 1,091 (100%) | 0.0 |
**Kit answer validator:**
- All 200 engine rows have status `ok` (0 unsupported, 0 errors). The runner's `validate` passed on every one, and
`compare.py` re-validated all three sets with 0 failures.
- All 140,620 rows of the full submitted results file also pass `validate` (`work/assemble.py`).
**Equivalence:** the outputs are bit-identical. This is expected, because the engine performs exactly the same tensor
operations as `e2e_prefix.py`: same truncation, same prompt, same split, same batch packing and same head. It also
confirms that the submitted run was produced by `bigind_ptr14nosyn_soup3`.
**Per-request latency (kit runner, this shared 3090, the stratified 200 rows; not the board's sample or hardware):**
median 48 ms, mean 107 ms, p80 116 ms, max 2.2 s. The maximum is a long multi-question row; the mean is pulled up by
ToolRet, BRIGHT and API-Bank rows.
**bf16 note:** different bf16 code paths can move probabilities in near-ties. For example, flash-linear-attention
kernels instead of the torch fallback, or another GPU. Earlier checks found that bf16 vs fp32 changes about 1% of
answers. Agreement on the maintainers' hardware should therefore be measured as an argmax-agreement rate, not as
exact equality.
Raw outputs: `work/val_engine/`, `work/val_e2e/`, `engine/val_compare.json`.