# Validation: jdi_ptr14 engine vs. the evaluator behind the submitted run Date: 2026-10-18. GPU: RTX 3090 (shared), `CUDA_VISIBLE_DEVICES=0`, `mem_gb=4.5`. torch 2.9.1+cu128, transformers 5.8.0. **Rows:** 200 suite rows from the public 0.3 suite, stratified over all 43 in-edition benchmarks with a fixed seed (`work/make_val200.py`). - 1,091 questions in total. - 47 rows are multi-question and take the shared-prefix path. - 40 questions have more than 9 options. **Three result sets:** 1. `engine`: `python -m decision_index run --engine jdi_ptr14.engine:Ptr14Engine --option model= --option mem_gb=4.5 --rows val200_rows.jsonl`, i.e. the kit runner, which validates every response with `decision_index.engines.base.validate`. 2. `e2e_fresh`: a fresh `e2e_prefix.py eval --layer 14 --batch-tokens 8192 --mem-gb 4.5` on the same rows. 3. `submitted`: the same rows taken from `runs/ptr14nosyn-soup3/results.jsonl.gz`. | Pair | Questions | Argmax agreement | Max abs prob diff | |---|---|---|---| | engine vs e2e_fresh | 1,091 | 1,091 (100%) | 0.0 | | engine vs submitted | 1,091 | 1,091 (100%) | 0.0 | | e2e_fresh vs submitted | 1,091 | 1,091 (100%) | 0.0 | **Kit answer validator:** - All 200 engine rows have status `ok` (0 unsupported, 0 errors). The runner's `validate` passed on every one, and `compare.py` re-validated all three sets with 0 failures. - All 140,620 rows of the full submitted results file also pass `validate` (`work/assemble.py`). **Equivalence:** the outputs are bit-identical. This is expected, because the engine performs exactly the same tensor operations as `e2e_prefix.py`: same truncation, same prompt, same split, same batch packing and same head. It also confirms that the submitted run was produced by `bigind_ptr14nosyn_soup3`. **Per-request latency (kit runner, this shared 3090, the stratified 200 rows; not the board's sample or hardware):** median 48 ms, mean 107 ms, p80 116 ms, max 2.2 s. The maximum is a long multi-question row; the mean is pulled up by ToolRet, BRIGHT and API-Bank rows. **bf16 note:** different bf16 code paths can move probabilities in near-ties. For example, flash-linear-attention kernels instead of the torch fallback, or another GPU. Earlier checks found that bf16 vs fp32 changes about 1% of answers. Agreement on the maintainers' hardware should therefore be measured as an argmax-agreement rate, not as exact equality. Raw outputs: `work/val_engine/`, `work/val_e2e/`, `engine/val_compare.json`.