source/docs/research/profiling-protocol.md

Training wrap-up and accuracy/speed protocol

The user requested training to finish and accuracy/speed to be profiled on 2026-09-17 at approximately 07:49 UTC. This explicitly advances the training stop and final selection; it does not extend the original 18:16:10 UTC deadline. All further model work is evaluation/inference, with no optimizer updates.

Checkpoint selection

Stop the three active supervisors gracefully and retain both current/resumable and validation-selected checkpoints. Record final steps and exit codes. Evaluate the latest saved weights of each stopped run on the same frozen 512 validation decisions, including the expanded-data run whose next periodic validation had not yet occurred. Keep the fixed four-fold temperature-crossfit macro-family NLL policy and require improvement strictly greater than 0.001 to replace that run's previous best. Preserve the original checkpoint bytes, source/model/data pins, optimizer/RNG state and matching validation evidence in separate snapshots.

scripts/final_validation.py creates a durable reconstruction before inference, verifies original source and all data hashes, reads only validation decisions, and writes a separate fleet-compatible candidate directory. The fixed reference SHA256 is e671e1508185765552b0f933ba03f356be62143c531d8ef534457d34b1645c9b. Final validation runs on the three independent GPUs, with a 09:00 UTC watchdog. The original stopped GX10 and completed 2B candidates remain eligible. Freeze the winner before reading any held-out prediction results.

Accuracy

Use the existing finalizer for the winner, preserving the calibrated checkpoint before test predictions. The 4B evaluation contains 510 calibration decisions, 2,042 original test decisions and 768 Social IQA decisions from an untrained task family. Compare the tuned scorer with the unchanged pinned pretrained readout, both raw and with independently fitted global temperatures. Report accuracy, NLL, multiclass Brier, top-label ECE and per-family results. Existing confidence intervals use 400 paired, stratified source-group bootstrap resamples.

Separately compare inference methods on a predeclared balanced sample of 320 held-out decisions (64 each from ARC, Banking77, BoolQ, SNLI and Social IQA), plus the 383 retained HellaSwag/PIQA/CommonsenseQA diagnostic decisions that never entered training or checkpoint selection. Sample IDs are chosen deterministically before scoring; apply a common 1,024-token inference eligibility limit, report every exclusion, and use matched retained rows across methods. These additional diagnostics cannot change the frozen winner. If the expanded latest checkpoint does not win, evaluate it on the same diagnostic rows on the spare Spark to measure the observed effect of the expanded-data run without further selection.

Speed

Benchmark the actual current inference implementation. Do not introduce shared prefix caching or change numerical precision during this profile. Compare on one otherwise idle Spark, with the same pinned 4B base, FP32, SDPA and isolated software environment:

1. Trained adapters and scalar head, full context separately for each option.
2. Physically adapter-free pretrained yes-minus-no scoring, with the identical per-option verifier prompt.
3. Physically adapter-free pretrained model reading all options together and returning one constrained answer-label token. Its label probabilities use logits restricted to verified single-token label IDs. Computing only those output projections is an explicit optimization equivalent to constrained next-token scoring; there is no generated explanation or JSON output.

The first two methods treat candidates independently; the third uses one joint prompt. Record that semantic difference and each method's processed token counts. The third method is a strong, efficient baseline, not a claim about arbitrary general-purpose serving systems or the hosted Jev API.

Use 12 workload cells: approximately 128/768 state tokens crossed with (questions, choices) = (1,2), (1,4), (1,16), (4,2), (4,4), (16,2). Use distinct fixed invented questions and serialize the exact requests. These are timing workloads, not semantic generalization tests. Each cell has two warmups and ten measured repetitions, with synchronized wall-clock timings including tokenization and probability construction. Save all observations, median, an exploratory empirical p95, throughput, token/branch counts and peak allocation. Ten observations provide only a rough tail estimate. Record model-load time separately; warm timings must not be labeled cold-start timings.

Keep the 16 GiB CUDA allocation cap, at least 24 GiB available host memory before loading, GPU process inspection and OOM adjustment 0. No training shares the benchmark GPU. The GX10 GUI remains available on 7466 while accuracy evaluation runs; speed measurements therefore use a Spark. Hardware/software/source and checkpoint hashes, exclusions, exit status and exact raw measurements accompany the report. Source, final checkpoints and evidence are backed up to the existing private Hugging Face repository with its current licenses and visibility.

Execution and findings

All three training stops and final validation passes exited 0. The latest expanded159 / A4765 / B2000 checkpoints scored validation crossfit NLL 0.172775 / 0.189890 / 0.170304, retaining best0 / best4000 / best1500 respectively. All five candidates were eligible. The selection froze at 08:09:51 UTC: expanded branch0 ties Spark B1500 at 0.170150 and 94.7266% validation accuracy, and wins the deterministic name tie-break. A live CPU proof confirms all 506 trainable tensors are identical across selected branch0, parent1500 and the calibrated artifact.

Calibration is complete with temperature 1.745822072, checkpoint SHA256 e270e3da905604d97bf5a8f380ea308133403d1c4790a5c012cb1c12e9b6f348 saved before test prediction. Full accuracy and matched speed evaluations completed from source 07f10e7; see the handover for run paths and controls. The fixed profile contains 320 held-out decisions and all 383 diagnostics, preserving the original 512-token per-choice eligibility before the common 1,024-token inference check. Six original test rows and one diagnostic row remain excluded; no further rows are dropped.

The selected calibrated model is available in the port-7466 GUI. All matched profiling jobs have now exited 0; the independent audit checked all 2,812 prediction rows, 36 timing cells and 360 measured samples. Both Spark GPUs were idle at 09:09:56 UTC. Full GX10 test/holdout evaluation and local API checks completed at 09:20:23 UTC, with evaluation, harness and coordinator exit 0.

Method	Matched held-out accuracy (320)	Expansion diagnostics (383)
Selected 4B scorer	89.06% (285)	79.11% (303)
Pretrained per-option verifier	80.94% (259)	77.81% (298)
Pretrained joint-label method	86.25% (276)	79.11% (303)
Expanded checkpoint 159	88.75% (284)	81.46% (312)

The selected scorer beats the joint-label method by 9 correct answers on the matched 320: 19 selected-only successes versus 10 label-only successes. Its aggregate diagnostic score equals the joint-label method, with different errors. These point estimates do not establish a universal accuracy ranking. The expanded checkpoint gains 14 answers and loses 5 on the diagnostics; it gains 2 and loses 3 on the matched held-out sample. Neither result changes the frozen selection.

For one four-choice question with a 768-token state, median local warm latency is 3.710 s selected / 3.177 s per-option base / 0.818 s joint-label base. The current trained implementation is therefore about 4.53 times slower than the efficient joint-label baseline on this workload. Across all 12 cells it is 11–17% slower than the matched unadapted per-option verifier. Repeated context processing and unmerged adapter work are targets for future measurement; no 4B prefix-cache or merged-adapter speedup has been established. See next-steps.md.

Raw profiles, exact requests and independent audit live under 20260917T075209Z-training-wrapup/profile-deployment in the run store; profile source is 07f10e7. The completed report contains tables, CSV/JSON data and standalone latency/accuracy plots. Full test accuracy is 92.90% selected versus 84.48% base verifier; Social IQA is 72.92% versus 70.31%. The 95% paired source-group bootstrap intervals on the gains are [6.85, 9.89] and [0.13, 5.34] percentage points respectively. These results cover the declared public datasets; pretraining exposure is unknown. Proper scores and independent temperature calibration are reported separately from accuracy.

The final calibrated model has been uploaded and remotely verified at andyshu/opensysone. FINAL_MODEL.json identifies the calibrated release; PROFILE_RESULTS.json identifies the separately verified profiling/report/checkpoint archive. Publication receipts and live service controls are recorded in the handover.