jdi-ptr14-0.8b / engine /VALIDATION.md
AnamK's picture
Upload jdi-ptr14-0.8b (Qwen3.5-0.8B, 14 layers + pointer head)
5bb55cb verified
|
Raw History Blame Contribute Delete
2.5 kB

Validation: jdi_ptr14 engine vs. the evaluator behind the submitted run

Date: 2026-10-18. GPU: RTX 3090 (shared), CUDA_VISIBLE_DEVICES=0, mem_gb=4.5. torch 2.9.1+cu128, transformers 5.8.0.

Rows: 200 suite rows from the public 0.3 suite, stratified over all 43 in-edition benchmarks with a fixed seed (work/make_val200.py).

  • 1,091 questions in total.
  • 47 rows are multi-question and take the shared-prefix path.
  • 40 questions have more than 9 options.

Three result sets:

  1. engine: python -m decision_index run --engine jdi_ptr14.engine:Ptr14Engine --option model=<ckpt> --option mem_gb=4.5 --rows val200_rows.jsonl, i.e. the kit runner, which validates every response with decision_index.engines.base.validate.
  2. e2e_fresh: a fresh e2e_prefix.py eval <ckpt> <payloads> --layer 14 --batch-tokens 8192 --mem-gb 4.5 on the same rows.
  3. submitted: the same rows taken from runs/ptr14nosyn-soup3/results.jsonl.gz.
Pair Questions Argmax agreement Max abs prob diff
engine vs e2e_fresh 1,091 1,091 (100%) 0.0
engine vs submitted 1,091 1,091 (100%) 0.0
e2e_fresh vs submitted 1,091 1,091 (100%) 0.0

Kit answer validator:

  • All 200 engine rows have status ok (0 unsupported, 0 errors). The runner's validate passed on every one, and compare.py re-validated all three sets with 0 failures.
  • All 140,620 rows of the full submitted results file also pass validate (work/assemble.py).

Equivalence: the outputs are bit-identical. This is expected, because the engine performs exactly the same tensor operations as e2e_prefix.py: same truncation, same prompt, same split, same batch packing and same head. It also confirms that the submitted run was produced by bigind_ptr14nosyn_soup3.

Per-request latency (kit runner, this shared 3090, the stratified 200 rows; not the board's sample or hardware): median 48 ms, mean 107 ms, p80 116 ms, max 2.2 s. The maximum is a long multi-question row; the mean is pulled up by ToolRet, BRIGHT and API-Bank rows.

bf16 note: different bf16 code paths can move probabilities in near-ties. For example, flash-linear-attention kernels instead of the torch fallback, or another GPU. Earlier checks found that bf16 vs fp32 changes about 1% of answers. Agreement on the maintainers' hardware should therefore be measured as an argmax-agreement rate, not as exact equality.

Raw outputs: work/val_engine/, work/val_e2e/, engine/val_compare.json.