| # Training wrap-up and accuracy/speed protocol |
|
|
| The user requested training to finish and accuracy/speed to be profiled on |
| 2026-09-17 at approximately 07:49 UTC. This explicitly advances the training |
| stop and final selection; it does not extend the original 18:16:10 UTC deadline. |
| All further model work is evaluation/inference, with no optimizer updates. |
|
|
| ## Checkpoint selection |
|
|
| Stop the three active supervisors gracefully and retain both current/resumable |
| and validation-selected checkpoints. Record final steps and exit codes. Evaluate |
| the latest saved weights of each stopped run on the same frozen 512 validation |
| decisions, including the expanded-data run whose next periodic validation had |
| not yet occurred. Keep the fixed four-fold temperature-crossfit macro-family NLL |
| policy and require improvement strictly greater than 0.001 to replace that run's |
| previous best. Preserve the original checkpoint bytes, source/model/data pins, |
| optimizer/RNG state and matching validation evidence in separate snapshots. |
|
|
| `scripts/final_validation.py` creates a durable reconstruction before inference, |
| verifies original source and all data hashes, reads only validation decisions, |
| and writes a separate fleet-compatible candidate directory. The fixed reference |
| SHA256 is `e671e1508185765552b0f933ba03f356be62143c531d8ef534457d34b1645c9b`. |
| Final validation runs on the three independent GPUs, with a 09:00 UTC watchdog. |
| The original stopped GX10 and completed 2B candidates remain eligible. Freeze |
| the winner before reading any held-out prediction results. |
|
|
| ## Accuracy |
|
|
| Use the existing finalizer for the winner, preserving the calibrated checkpoint |
| before test predictions. The 4B evaluation contains 510 calibration decisions, |
| 2,042 original test decisions and 768 Social IQA decisions from an untrained task |
| family. Compare the tuned scorer with the unchanged pinned pretrained readout, |
| both raw and with independently fitted global temperatures. Report accuracy, |
| NLL, multiclass Brier, top-label ECE and per-family results. Existing confidence |
| intervals use 400 paired, stratified source-group bootstrap resamples. |
|
|
| Separately compare inference methods on a predeclared balanced sample of 320 |
| held-out decisions (64 each from ARC, Banking77, BoolQ, SNLI and Social IQA), plus |
| the 383 retained HellaSwag/PIQA/CommonsenseQA diagnostic decisions that never |
| entered training or checkpoint selection. Sample IDs are chosen deterministically |
| before scoring; apply a common 1,024-token inference eligibility limit, report |
| every exclusion, and use matched retained rows across methods. These additional |
| diagnostics cannot change the frozen winner. If the expanded latest checkpoint |
| does not win, evaluate it on the same diagnostic rows on the spare Spark to |
| measure the observed effect of the expanded-data run without further selection. |
|
|
| ## Speed |
|
|
| Benchmark the actual current inference implementation. Do not introduce shared |
| prefix caching or change numerical precision during this profile. Compare on |
| one otherwise idle Spark, with the same pinned 4B base, FP32, SDPA and isolated |
| software environment: |
|
|
| 1. Trained adapters and scalar head, full context separately for each option. |
| 2. Physically adapter-free pretrained yes-minus-no scoring, with the identical |
| per-option verifier prompt. |
| 3. Physically adapter-free pretrained model reading all options together and |
| returning one constrained answer-label token. Its label probabilities use |
| logits restricted to verified single-token label IDs. Computing only those |
| output projections is an explicit optimization equivalent to constrained |
| next-token scoring; there is no generated explanation or JSON output. |
|
|
| The first two methods treat candidates independently; the third uses one joint |
| prompt. Record that semantic difference and each method's processed token counts. |
| The third method is a strong, efficient baseline, not a claim about arbitrary |
| general-purpose serving systems or the hosted Jev API. |
|
|
| Use 12 workload cells: approximately 128/768 state tokens crossed with |
| (questions, choices) = (1,2), (1,4), (1,16), (4,2), (4,4), (16,2). |
| Use distinct fixed invented questions and serialize the exact requests. These are |
| timing workloads, not semantic generalization tests. Each cell has two warmups |
| and ten measured repetitions, with synchronized wall-clock timings including |
| tokenization and probability construction. Save all observations, median, an |
| exploratory empirical p95, throughput, token/branch counts and peak allocation. |
| Ten observations provide only a rough tail estimate. Record model-load time |
| separately; warm timings must not be labeled cold-start timings. |
|
|
| Keep the 16 GiB CUDA allocation cap, at least 24 GiB available host memory before |
| loading, GPU process inspection and OOM adjustment 0. No training shares the |
| benchmark GPU. The GX10 GUI remains available on 7466 while accuracy evaluation |
| runs; speed measurements therefore use a Spark. Hardware/software/source and |
| checkpoint hashes, exclusions, exit status and exact raw measurements accompany |
| the report. Source, final checkpoints and evidence are backed up to the existing |
| private Hugging Face repository with its current licenses and visibility. |
|
|
| ## Execution and findings |
|
|
| All three training stops and final validation passes exited 0. The latest |
| expanded159 / A4765 / B2000 checkpoints scored validation crossfit NLL |
| 0.172775 / 0.189890 / 0.170304, retaining best0 / best4000 / best1500 respectively. |
| All five candidates were eligible. The selection froze at 08:09:51 UTC: expanded |
| branch0 ties Spark B1500 at 0.170150 and 94.7266% validation accuracy, and wins the |
| deterministic name tie-break. A live CPU proof confirms all 506 trainable tensors |
| are identical across selected branch0, parent1500 and the calibrated artifact. |
|
|
| Calibration is complete with temperature 1.745822072, checkpoint SHA256 |
| `e270e3da905604d97bf5a8f380ea308133403d1c4790a5c012cb1c12e9b6f348` saved before test |
| prediction. Full accuracy and matched speed evaluations completed from source |
| `07f10e7`; see the [handover](../operations/handover.md) for run paths and controls. The fixed profile contains |
| 320 held-out decisions and all 383 diagnostics, preserving the original 512-token |
| per-choice eligibility before the common 1,024-token inference check. Six original |
| test rows and one diagnostic row remain excluded; no further rows are dropped. |
|
|
| The selected calibrated model is available in the port-7466 GUI. All matched |
| profiling jobs have now exited 0; the independent audit checked all 2,812 |
| prediction rows, 36 timing cells and 360 measured samples. Both Spark GPUs were |
| idle at 09:09:56 UTC. Full GX10 test/holdout evaluation and local API checks completed at 09:20:23 UTC, |
| with evaluation, harness and coordinator exit 0. |
|
|
| | Method | Matched held-out accuracy (320) | Expansion diagnostics (383) | |
| | --- | ---: | ---: | |
| | Selected 4B scorer | 89.06% (285) | 79.11% (303) | |
| | Pretrained per-option verifier | 80.94% (259) | 77.81% (298) | |
| | Pretrained joint-label method | 86.25% (276) | 79.11% (303) | |
| | Expanded checkpoint 159 | 88.75% (284) | 81.46% (312) | |
|
|
| The selected scorer beats the joint-label method by 9 correct answers on the |
| matched 320: 19 selected-only successes versus 10 label-only successes. Its |
| aggregate diagnostic score equals the joint-label method, with different errors. |
| These point estimates do not establish a universal accuracy ranking. The expanded |
| checkpoint gains 14 answers and loses 5 on the diagnostics; it gains 2 and loses |
| 3 on the matched held-out sample. Neither result changes the frozen selection. |
|
|
| For one four-choice question with a 768-token state, median local warm latency is |
| 3.710 s selected / 3.177 s per-option base / 0.818 s joint-label base. The current |
| trained implementation is therefore about 4.53 times slower than the efficient |
| joint-label baseline on this workload. Across all 12 cells it is 11–17% slower |
| than the matched unadapted per-option verifier. Repeated context processing and |
| unmerged adapter work are targets for future measurement; no 4B prefix-cache or |
| merged-adapter speedup has been established. See [next-steps.md](next-steps.md). |
|
|
| Raw profiles, exact requests and independent audit live under |
| `20260917T075209Z-training-wrapup/profile-deployment` in the run store; profile |
| source is `07f10e7`. The [completed report](../../results/20260917-wrapup/profile-report/report.md) |
| contains tables, CSV/JSON data and standalone latency/accuracy plots. Full test |
| accuracy is 92.90% selected versus 84.48% base verifier; Social IQA is 72.92% |
| versus 70.31%. The 95% paired source-group bootstrap intervals on the gains are |
| [6.85, 9.89] and [0.13, 5.34] percentage points respectively. These results cover |
| the declared public datasets; pretraining exposure is unknown. Proper scores |
| and independent temperature calibration are reported separately from accuracy. |
|
|
| The final calibrated model has been uploaded and remotely verified at |
| [andyshu/opensysone](https://huggingface.co/andyshu/opensysone). `FINAL_MODEL.json` |
| identifies the calibrated release; `PROFILE_RESULTS.json` identifies the separately |
| verified profiling/report/checkpoint archive. Publication receipts and live |
| service controls are recorded in the [handover](../operations/handover.md). |
|
|
