Multiple Choice
Transformers
Safetensors
qwen3_5_text
text-generation
decision-index
classification
early-exit
pointer-head
Instructions to use AnamK/jdi-ptr14-0.8b with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use AnamK/jdi-ptr14-0.8b with Transformers:
# pip install -U transformers accelerate # Load model directly from transformers import AutoTokenizer, AutoModelForCausalLM tokenizer = AutoTokenizer.from_pretrained("AnamK/jdi-ptr14-0.8b") model = AutoModelForCausalLM.from_pretrained("AnamK/jdi-ptr14-0.8b", device_map="auto") - Notebooks
- Google Colab
- Kaggle
|
Download engine/VALIDATION.md from AnamK/jdi-ptr14-0.8b: direct link, hf CLI and curl.
- Browser
- Download file 2.5 kB
-
https://huggingface.co/AnamK/jdi-ptr14-0.8b/resolve/main/engine/VALIDATION.md
- Command line
-
hf download hf://AnamK/jdi-ptr14-0.8b/engine/VALIDATION.md
-
curl -L -o VALIDATION.md https://huggingface.co/AnamK/jdi-ptr14-0.8b/resolve/main/engine/VALIDATION.md
2.5 kB
| # Validation: jdi_ptr14 engine vs. the evaluator behind the submitted run | |
| Date: 2026-10-18. GPU: RTX 3090 (shared), `CUDA_VISIBLE_DEVICES=0`, `mem_gb=4.5`. torch 2.9.1+cu128, transformers 5.8.0. | |
| **Rows:** 200 suite rows from the public 0.3 suite, stratified over all 43 in-edition benchmarks with a fixed seed | |
| (`work/make_val200.py`). | |
| - 1,091 questions in total. | |
| - 47 rows are multi-question and take the shared-prefix path. | |
| - 40 questions have more than 9 options. | |
| **Three result sets:** | |
| 1. `engine`: `python -m decision_index run --engine jdi_ptr14.engine:Ptr14Engine --option model=<ckpt> --option mem_gb=4.5 --rows val200_rows.jsonl`, i.e. the kit runner, which validates every response with `decision_index.engines.base.validate`. | |
| 2. `e2e_fresh`: a fresh `e2e_prefix.py eval <ckpt> <payloads> --layer 14 --batch-tokens 8192 --mem-gb 4.5` on the same rows. | |
| 3. `submitted`: the same rows taken from `runs/ptr14nosyn-soup3/results.jsonl.gz`. | |
| | Pair | Questions | Argmax agreement | Max abs prob diff | | |
| |---|---|---|---| | |
| | engine vs e2e_fresh | 1,091 | 1,091 (100%) | 0.0 | | |
| | engine vs submitted | 1,091 | 1,091 (100%) | 0.0 | | |
| | e2e_fresh vs submitted | 1,091 | 1,091 (100%) | 0.0 | | |
| **Kit answer validator:** | |
| - All 200 engine rows have status `ok` (0 unsupported, 0 errors). The runner's `validate` passed on every one, and | |
| `compare.py` re-validated all three sets with 0 failures. | |
| - All 140,620 rows of the full submitted results file also pass `validate` (`work/assemble.py`). | |
| **Equivalence:** the outputs are bit-identical. This is expected, because the engine performs exactly the same tensor | |
| operations as `e2e_prefix.py`: same truncation, same prompt, same split, same batch packing and same head. It also | |
| confirms that the submitted run was produced by `bigind_ptr14nosyn_soup3`. | |
| **Per-request latency (kit runner, this shared 3090, the stratified 200 rows; not the board's sample or hardware):** | |
| median 48 ms, mean 107 ms, p80 116 ms, max 2.2 s. The maximum is a long multi-question row; the mean is pulled up by | |
| ToolRet, BRIGHT and API-Bank rows. | |
| **bf16 note:** different bf16 code paths can move probabilities in near-ties. For example, flash-linear-attention | |
| kernels instead of the torch fallback, or another GPU. Earlier checks found that bf16 vs fp32 changes about 1% of | |
| answers. Agreement on the maintainers' hardware should therefore be measured as an argmax-agreement rate, not as | |
| exact equality. | |
| Raw outputs: `work/val_engine/`, `work/val_e2e/`, `engine/val_compare.json`. | |