source/docs/research/profiling-protocol.md

# Training wrap-up and accuracy/speed protocol

The user requested training to finish and accuracy/speed to be profiled on
2026-09-17 at approximately 07:49 UTC. This explicitly advances the training
stop and final selection; it does not extend the original 18:16:10 UTC deadline.
All further model work is evaluation/inference, with no optimizer updates.

## Checkpoint selection

Stop the three active supervisors gracefully and retain both current/resumable
and validation-selected checkpoints. Record final steps and exit codes. Evaluate
the latest saved weights of each stopped run on the same frozen 512 validation
decisions, including the expanded-data run whose next periodic validation had
not yet occurred. Keep the fixed four-fold temperature-crossfit macro-family NLL
policy and require improvement strictly greater than 0.001 to replace that run's
previous best. Preserve the original checkpoint bytes, source/model/data pins,
optimizer/RNG state and matching validation evidence in separate snapshots.

`scripts/final_validation.py` creates a durable reconstruction before inference,
verifies original source and all data hashes, reads only validation decisions,
and writes a separate fleet-compatible candidate directory. The fixed reference
SHA256 is `e671e1508185765552b0f933ba03f356be62143c531d8ef534457d34b1645c9b`.
Final validation runs on the three independent GPUs, with a 09:00 UTC watchdog.
The original stopped GX10 and completed 2B candidates remain eligible. Freeze
the winner before reading any held-out prediction results.

## Accuracy

Use the existing finalizer for the winner, preserving the calibrated checkpoint
before test predictions. The 4B evaluation contains 510 calibration decisions,
2,042 original test decisions and 768 Social IQA decisions from an untrained task
family. Compare the tuned scorer with the unchanged pinned pretrained readout,
both raw and with independently fitted global temperatures. Report accuracy,
NLL, multiclass Brier, top-label ECE and per-family results. Existing confidence
intervals use 400 paired, stratified source-group bootstrap resamples.

Separately compare inference methods on a predeclared balanced sample of 320
held-out decisions (64 each from ARC, Banking77, BoolQ, SNLI and Social IQA), plus
the 383 retained HellaSwag/PIQA/CommonsenseQA diagnostic decisions that never
entered training or checkpoint selection. Sample IDs are chosen deterministically
before scoring; apply a common 1,024-token inference eligibility limit, report
every exclusion, and use matched retained rows across methods. These additional
diagnostics cannot change the frozen winner. If the expanded latest checkpoint
does not win, evaluate it on the same diagnostic rows on the spare Spark to
measure the observed effect of the expanded-data run without further selection.

## Speed

Benchmark the actual current inference implementation. Do not introduce shared
prefix caching or change numerical precision during this profile. Compare on
one otherwise idle Spark, with the same pinned 4B base, FP32, SDPA and isolated
software environment:

1. Trained adapters and scalar head, full context separately for each option.
2. Physically adapter-free pretrained yes-minus-no scoring, with the identical
   per-option verifier prompt.
3. Physically adapter-free pretrained model reading all options together and
   returning one constrained answer-label token. Its label probabilities use
   logits restricted to verified single-token label IDs. Computing only those
   output projections is an explicit optimization equivalent to constrained
   next-token scoring; there is no generated explanation or JSON output.

The first two methods treat candidates independently; the third uses one joint
prompt. Record that semantic difference and each method's processed token counts.
The third method is a strong, efficient baseline, not a claim about arbitrary
general-purpose serving systems or the hosted Jev API.

Use 12 workload cells: approximately 128/768 state tokens crossed with
(questions, choices) = (1,2), (1,4), (1,16), (4,2), (4,4), (16,2).
Use distinct fixed invented questions and serialize the exact requests. These are
timing workloads, not semantic generalization tests. Each cell has two warmups
and ten measured repetitions, with synchronized wall-clock timings including
tokenization and probability construction. Save all observations, median, an
exploratory empirical p95, throughput, token/branch counts and peak allocation.
Ten observations provide only a rough tail estimate. Record model-load time
separately; warm timings must not be labeled cold-start timings.

Keep the 16 GiB CUDA allocation cap, at least 24 GiB available host memory before
loading, GPU process inspection and OOM adjustment 0. No training shares the
benchmark GPU. The GX10 GUI remains available on 7466 while accuracy evaluation
runs; speed measurements therefore use a Spark. Hardware/software/source and
checkpoint hashes, exclusions, exit status and exact raw measurements accompany
the report. Source, final checkpoints and evidence are backed up to the existing
private Hugging Face repository with its current licenses and visibility.

## Execution and findings

All three training stops and final validation passes exited 0. The latest
expanded159 / A4765 / B2000 checkpoints scored validation crossfit NLL
0.172775 / 0.189890 / 0.170304, retaining best0 / best4000 / best1500 respectively.
All five candidates were eligible. The selection froze at 08:09:51 UTC: expanded
branch0 ties Spark B1500 at 0.170150 and 94.7266% validation accuracy, and wins the
deterministic name tie-break. A live CPU proof confirms all 506 trainable tensors
are identical across selected branch0, parent1500 and the calibrated artifact.

Calibration is complete with temperature 1.745822072, checkpoint SHA256
`e270e3da905604d97bf5a8f380ea308133403d1c4790a5c012cb1c12e9b6f348` saved before test
prediction. Full accuracy and matched speed evaluations completed from source
`07f10e7`; see the [handover](../operations/handover.md) for run paths and controls. The fixed profile contains
320 held-out decisions and all 383 diagnostics, preserving the original 512-token
per-choice eligibility before the common 1,024-token inference check. Six original
test rows and one diagnostic row remain excluded; no further rows are dropped.

The selected calibrated model is available in the port-7466 GUI. All matched
profiling jobs have now exited 0; the independent audit checked all 2,812
prediction rows, 36 timing cells and 360 measured samples. Both Spark GPUs were
idle at 09:09:56 UTC. Full GX10 test/holdout evaluation and local API checks completed at 09:20:23 UTC,
with evaluation, harness and coordinator exit 0.

| Method | Matched held-out accuracy (320) | Expansion diagnostics (383) |
| --- | ---: | ---: |
| Selected 4B scorer | 89.06% (285) | 79.11% (303) |
| Pretrained per-option verifier | 80.94% (259) | 77.81% (298) |
| Pretrained joint-label method | 86.25% (276) | 79.11% (303) |
| Expanded checkpoint 159 | 88.75% (284) | 81.46% (312) |

The selected scorer beats the joint-label method by 9 correct answers on the
matched 320: 19 selected-only successes versus 10 label-only successes. Its
aggregate diagnostic score equals the joint-label method, with different errors.
These point estimates do not establish a universal accuracy ranking. The expanded
checkpoint gains 14 answers and loses 5 on the diagnostics; it gains 2 and loses
3 on the matched held-out sample. Neither result changes the frozen selection.

For one four-choice question with a 768-token state, median local warm latency is
3.710 s selected / 3.177 s per-option base / 0.818 s joint-label base. The current
trained implementation is therefore about 4.53 times slower than the efficient
joint-label baseline on this workload. Across all 12 cells it is 11–17% slower
than the matched unadapted per-option verifier. Repeated context processing and
unmerged adapter work are targets for future measurement; no 4B prefix-cache or
merged-adapter speedup has been established. See [next-steps.md](next-steps.md).

Raw profiles, exact requests and independent audit live under
`20260917T075209Z-training-wrapup/profile-deployment` in the run store; profile
source is `07f10e7`. The [completed report](../../results/20260917-wrapup/profile-report/report.md)
contains tables, CSV/JSON data and standalone latency/accuracy plots. Full test
accuracy is 92.90% selected versus 84.48% base verifier; Social IQA is 72.92%
versus 70.31%. The 95% paired source-group bootstrap intervals on the gains are
[6.85, 9.89] and [0.13, 5.34] percentage points respectively. These results cover
the declared public datasets; pretraining exposure is unknown. Proper scores
and independent temperature calibration are reported separately from accuracy.

The final calibrated model has been uploaded and remotely verified at
[andyshu/opensysone](https://huggingface.co/andyshu/opensysone). `FINAL_MODEL.json`
identifies the calibrated release; `PROFILE_RESULTS.json` identifies the separately
verified profiling/report/checkpoint archive. Publication receipts and live
service controls are recorded in the [handover](../operations/handover.md).